A new study from Metric Sherpa, produced with Sprinklr, puts a hard number on a familiar feeling: marketers believe AI is working, but few can back that belief with evidence.
Among organizations using customer-facing AI regularly, 92% said it met or exceeded expectations. Only 27% said they could show proof of that. Even on efficiency, 67% of those who said AI made work faster also said it reduced costs, but only 32% of that group had tracked costs before launch.
Measurement has to start before the AI does
Most teams know what they want AI to improve. Far fewer define the metric, record the starting point, or set a clear threshold for success.
The study found 99% could name an intended outcome, but only 26% defined success metrics and just 10% recorded a baseline. Without those two pieces, proving ROI is nearly impossible.
If the goal is lower acquisition cost, the team needs a cost baseline before AI starts making campaign decisions. If the goal is better customer service, containment alone won’t tell the story; teams need data on resolution, repeat contacts, escalations and handoffs.
Don’t let the platform grade its own homework
When baselines and success metrics are missing, teams end up relying on whatever numbers the platform provides. A media platform can optimize conversion, a service platform can optimize containment, and a content system can optimize engagement. All three can show improvement in their own dashboards while the business outcome stays flat.
That may explain this finding: 85% of respondents reported improvement in at least one metric category they hadn’t tracked before launch. Researchers couldn’t tell whether teams added measurement later or relied on another source, but the disconnect is hard to miss.
Marketing ops should connect platform activity to CRM, finance, customer or experiment data. Otherwise, the tool doing the work also gets to decide whether the work succeeded.
Someone still has to own the result
Only 17% of organizations name an owner for AI business outcomes, making it the least common of eight practices measured in the study.
The report distinguishes three roles:
- Platform owner keeps the tool working.
- Workflow owner keeps the process running.
- Outcome owner answers whether the AI produced the desired business result.
That last role matters. Organizations with a named outcome owner quantified results 46% of the time, compared with 23% for those without one.
Better measurement can make AI look worse
Teams that measure more carefully may find more problems. Organizations using three or more of the study’s measurement practices reported more AI-to-employee handoff issues, adoption problems and data-quality friction than those using zero or one.
That doesn’t mean measurement is bad. A mature AI program can look messier because someone is finally checking what happens after the automation fires. Reviews should go beyond output and speed to find where handoffs break, where employees ignore recommendations, and where customers repeat themselves.
Decide what would make you stop
Only 28% of organizations use agreed criteria to decide whether to scale, change or stop an AI application. Just 37% review AI results on a regular schedule.
That leaves plenty of AI running without a clear rule for what happens if the direct metric improves but the business result doesn’t. For any single application, define the outcome, baseline, success threshold and comparison before the next review. Then write down what result would justify scaling, what would require a change, and what would cause the team to stop.
Source: MarTech
