August 3, 2022
Omdia report: Best practices for measuring the success of AI projects
As enterprise AI projects gain momentum, a critical component is developing the right key performance indicators (KPI) to act as guardrails for success.
But while 43% of enterprises have at least one live AI project, only 18% have AI KPIs in place for all of their AI initiatives, according to the latest Omdia report, “AI KPIs Remain the Exception, Not the Rule.”
“With so much investment at stake … KPIs for AI are the most important guardrails for senior management to use to guide their AI strategies,” said report author Mark Beccue, Omdia principal analyst of AI and NLP.
Meanwhile, 53% of respondents say they just have ad hoc metrics in place and only for some AI use cases. But this is not ideal. “Measuring outcomes using ad hoc metrics could distort the return on investment (ROI) for these projects,” the analyst said.
Generally, 57% said to start with business goals when developing AI KPIs; another 22% preferred operational goals. However, there is no agreement on who should lead the AI KPI programs, with preferences fairly evenly split among CDO, CTO and CIO teams.
Beccue believes that AI KPI leadership will increasingly reside with a company's line of business leaders rather than its data science teams.
Related story: Global enterprise AI adoption reaching 'critical mass'
Developing KPIs face another issue: There is “little consensus” in the approach to take, including what kinds of business metrics AI should be tied to and who should define and own AI KPIs, Beccue wrote. Without a standard, 55% of early adopters said they created their own AI KPIs from scratch.
“Choosing KPIs for AI is a challenge, as the industry is still in the early stages of AI market adoption. AI readiness, knowledge, experience, and expertise across enterprises are low. “ The report lists the following best practices for developing AI KPIs in the enterprise: 1. Work backwards to determine your KPI path. To build AI KPIs, start with the broad-based organizational outcomes and associated KPIs and map AI outcomes to them. AI outcomes will determine which AI model metrics to use that will align AI outcomes and KPIs to organizational KPIs. 2. Ensure AI KPIs make sense within your specific circumstances as a business or organization. AI KPIs vary significantly by horizontal application or industry vertical. 3. Cost reduction, engagement and time reduction are good starting points for AI KPIs. While AI KPIs vary widely, Omdia found these three measurements are commonly used across different use cases and industries. 4. Understand the difference between AI model metrics and AI KPIs. AI model metrics, such as classification metrics, regression metrics, General Language Understanding Evaluation (GLUE) and so on, measure specific AI outcomes but they are not by themselves good AI KPIs. AI model metrics are best used to inform and contribute to broad-based organizational AI KPIs (cost reduction, engagement, etc.). 5. Realize that mature use cases and applications will feature more developed, proven AI model metrics and KPIs — and the reverse is also true. Expect AI model metrics and KPIs to evolve over time. KPIs for less mature AI use cases will evolve more rapidly than KPIs for more mature use cases. The less mature the AI use case is, the more risk there is of getting associated KPIs wrong.