Measuring Agency KPIs Better with AI Agents

Agency KPIs for AI Agents are the metrics with which an agency demonstrates the value and economic viability of its agent-based services: client-side output, quality, conversion and time-to-value; internally utilisation, margin, token costs, plus error and HITL rates. The key is separating fast-growing adoption figures from the few auditable value metrics that a CFO can follow.

• Measure the client side, not just usage: success rate, quality, conversion/cost-out and time-to-value demonstrate value to the client – licence utilisation alone does not.
• Look at the margin internally: token cost per task, utilisation and above all the HITL escalation rate determine whether an agent project is profitable.
• Calculate conservatively: the credible reference point is a productivity increase of around 14 percent, not the tenfold figure often promised.

Why most agency dashboards celebrate the wrong number

The most common KPI mistake in DACH projects in 2026 is to pin success on adoption alone. A programme with full licence utilisation that has never moved a P&L line is a programme without impact. The empirical picture is unambiguous: the number of pilot projects launched, the number of use cases identified and the number of AI tools deployed do not correlate with value creation. BCG provides the clearest counterexample in its AI Radar – AI leaders concentrate on an average of 3.5 use cases, laggards spread across 6.1, and the leaders nonetheless expect 2.1x the ROI. Concentration beats proliferation. On top of this: around 60 percent of the companies surveyed do not define or monitor any financial KPI for their AI value at all.

For an agency this means: the reporting must cleanly separate two layers – adoption metrics (necessary, not sufficient) and outcome metrics (the ones that count).

Client-side KPIs: output, quality, conversion, time-to-value

For the client, what matters is what the agent achieves in the process. Four dimensions are leading:

• Output / success rate: the share of tasks the agent correctly brings to an end state – the single most important headline figure. Complemented by task completion (does the agent even reach a terminal state?) to distinguish "gave up" from "answered incorrectly".
• Quality: faithfulness or hallucination rate (the share of statements covered by retrieved context or world knowledge) and – especially for law, medicine, finance – citation accuracy at the statement level. Consistency across repeated runs belongs here too: an agent that solves 90 percent of cases but unpredictably tips over on 10 percent is often worse than one with 80 percent that fails in a predictable and remediable way.
• Conversion / impact: depending on the process, lead-to-quote rate, deflection rate in service, CSAT/NPS, cycle-time reduction (case-to-close), defect or error rate.
• Time-to-value: how quickly does the first measurable ROI emerge? Realistic DACH expectations are 3 to 6 months for service tier-1 augmentation, 6 to 9 for CRM-embedded sales/marketing copilots, 6 to 12 for internal knowledge/search agents, and 9 to 15 for document-heavy back-office processes. Three-month promises are not credible.

Internal KPIs: utilisation, margin, token costs, error/HITL rate

The second KPI family protects the economic viability of the agency itself:

• Utilisation of the scarce roles, above all the AI product managers who are accountable for use cases and outcomes.
• Project margin as a lagging metric, validated by Finance.
• Token/inference cost per task: input tokens, output tokens and – decisive since 2025 – reasoning tokens, which with reasoning models can dominate the cost. Reporting units: euros per task and euros per 1,000 tasks, plus latency P50/P95/P99.
• Error and HITL rate: the human-in-the-loop escalation rate is the direct lever on the margin. Every escalation ties up review time, which often eats up 30 to 60 percent of the gross deflection saving again – the largest hidden cost block in service and document agents. Important for interpretation: a high escalation rate is not automatically bad; well-calibrated escalation in high-risk processes is a feature. What matters is the downward trend at stable quality.

Leading vs. lagging – and the compensation rule

Early indicators move quickly and are controllable; late indicators show the business outcome with a delay, but credibly.

The discipline must be made explicit: report both, but compensate on lagging. Every study cited shows the same pattern – teams rewarded for adoption deliver high adoption and no value; teams rewarded for outcomes deliver measurable value.

Calculate ROI conservatively: 14 percent, not 10x

The most credible documented productivity figure for AI in the enterprise comes from the study by Brynjolfsson, Li and Raymond (NBER WP 31161, 2023; QJE 2025): 14 percent more cases resolved per hour in customer service on average, 34 percent for newcomers and low-skilled workers, with barely any effect on experienced professionals. That is the upper end of what is credible for service – and a strategic signal: AI distributes value down the skill curve.

Two corrections belong in every agency calculation:

• The GitHub Copilot classic "55 percent faster" applies only to tightly specified, isolated coding tasks – not to end-to-end delivery. There, Bain (Technology Report 2025) finds typical 10 to 15 percent, often not recovered into higher-value work because review, test and deployment bottlenecks remain downstream.
• Self-reports are unreliable. In the METR field trial (arXiv 2507.09089, 2025), 16 experienced open-source developers were 19 percent slower with AI, yet forecast plus 24 percent beforehand and afterwards believed in plus 20 percent; ML and economics experts even predicted 38 to 39 percent speed-up. Boardroom translation: steer on telemetry and outcome metrics, not on self-reports.

From this follows the clean bottom-up formula that a CFO can check:

```
Gross saving = time saving% x annual volume x fully-loaded cost per case
Net ROI = gross saving

• • • • licence/platform
      • deployment/integration
      • observability/eval
      • HITL review (30-60% of the gross saving)
        ```

One must stay honest about the "ROI-is-not-measurable" problem: when LLM costs are small relative to total OpEx, the gain shows up as faster work, not as a measurable cost reduction at line-item level. With broad horizontal copilots the ROI is often not detectable as a line item – that is not a failure if the bet was deliberately declared as a capability investment.

Example dashboard: service tier-1 agent of a mid-sized company

Assumption: 120,000 service cases/year, fully-loaded cost €6 per case, deflection 40 percent, per-conversation inference cost €0.30 (corridor per research: €0.10–1.00).

The dashboard makes two things visible: the seemingly expensive item (token costs) is not the cost driver – HITL and operations are. And the HITL rate is the lever on which the net value is decided over the following quarters.

OKR form that the CFO can check

Objective: build credible agent capability in the core revenue function.

• KR1: 70 %+ active weekly adoption of the agent in the function within 9 months.
• KR2: 25 %+ reduction in the median cycle time of the target process against baseline within 12 months.
• KR3: NPS/CSAT with no deterioration (or +5 %) over the period.
• KR4: HITL escalation rate <20 % within 12 months with a measurable downward trend.
• KR5: net P&L contribution validated by Finance within 18 months.

This form forces outcomes that are auditable – and kill discipline. Hard gates are more valuable than soft governance: at 6 months without a clear ROI path (adoption flat below 30 percent, no measurable improvement) and at 12 months without a quantitative ROI signal, end the project, reclaim the budget, no zombie. Every agent programme needs an explicit kill criterion in its founding charter.

For agencies and B2B decision-makers

Those who sell agent services will in future sell outcomes, not tool access. An agency that provides its clients with a two-layer KPI model – leading early indicators for steering, lagging value metrics for compensation – and calculates ROI conservatively with 14-percent logic instead of 10x promises wins trust in the boardroom. Blck Alpaca builds exactly such measurement and eval setups: from the client-side outcome dashboard (success rate, quality, conversion, time-to-value) to the internal margin and HITL cost accounting. If you want to set up an auditable KPI framework for AI agent projects for your agency or your company, talk to us – we define the metrics before deployment, with a baseline and Finance sign-off.