Agency17 June 20263 min read

Measuring AI's impact in production, honestly (no vanity metrics)

Lines of AI-written code and acceptance rates measure activity, not impact. The honest question is whether your team ships more of the right work at the same quality. How to read that, and the one number that's actually real.

The short answer

Measure the outcome, not the activity. Lines of AI-written code and acceptance rates are vanity metrics. The honest question is whether your team ships more of the right work, at the same or better quality, with the same people, read against your own baseline. There's no clean universal productivity number; anyone quoting one is selling.

tsukumo

Short version: most AI impact numbers measure activity, not value. Lines of AI-written code, suggestion-acceptance rate, "percent of code written by AI", commit counts, all of them are easy to grow and tell you almost nothing. The honest question is narrower and harder: does your team now ship more of the right work, at the same or better quality, with the same people? You read that against your own baseline, with a few real signals, not a single invented productivity percentage.

The metrics to ignore#

These look like measurement and aren't:

Lines of AI-written code. More code is a cost, not a result. This rewards volume and quietly punishes the cleanest solution, which is often less code.

Suggestion-acceptance rate. Measures how agreeable the autocomplete is, not whether the accepted code was right or survived review.

Percent of code written by AI. Trivially gamed, and a team can raise it while shipping more low-quality code that costs more to review and fix.

Raw commit and PR counts. Activity, not outcomes. A busier graph isn't a better quarter.

Each of these can go up while nothing valuable ships and quality slips. That's the tell of a vanity metric: it can improve while the business doesn't.

The honest question#

Reframe impact as one sentence: are we shipping more of the right work, at the same or better quality, with the same team? Three parts, and all three matter.

More of the right work, not more output. Throughput on the things that move the product, not a higher line count. This is the difference between 10x and just cheaper.
At held quality. If defects or escaped bugs rise, you didn't gain speed, you borrowed it. Quality is a gate on every other number.
With the same team. If the only "ROI" is headcount cut, you've measured the wrong thing and usually broken the capability. We make that case in augment, never replace.

Where does your team actually stand on this? A short agent-ops assessment is the low-risk way to find out.

Signals worth watching#

No single metric captures this, so watch a few and read them together, always against your own starting point:

Cycle time on real units of work. How long a meaningful change takes from start to shipped, not keystrokes saved. Track the trend, not a headline figure.
Reviewer confidence. Do your seniors trust agent-assisted changes enough to approve them without rescuing each one? That's a real, if qualitative, read on whether the work is sound.
Defect and escape rate. Held or improved means the speed is real. Rising means you're trading quality for the appearance of progress.
What the team can now take on. The clearest signal is often work the team couldn't have attempted before, now in reach. Capability, not just speed.
Cost per unit of work. Are agents doing this affordably, or burning tokens to look busy?

The one number that's actually real#

There is a hard number you can measure honestly, and it isn't a productivity percentage: what your agents cost to do real work. Context is the big lever. An agent that rereads the whole repo every session is expensive and often confidently wrong; one served the current canonical answer is cheaper and more reliable. That's measurable, not asserted, it's why we built trovex, which cuts the tokens per lookup by roughly 60%. Efficiency you can put a meter on beats a productivity number someone made up.

Why there's no single ROI figure#

If a vendor hands you a clean universal "AI makes teams N% more productive," treat it as marketing. An honest read depends on your codebase, your baseline, and your definition of the right work. The number that matters is the change from where you started, measured on outcomes, and it takes a few weeks of real work to see, not a benchmark to quote. The absence of a tidy headline figure is the honest part, not a weakness.

How we measure it#

We don't promise a productivity percentage, because an honest one doesn't exist before we've seen your repo. We start from your baseline, get agents doing real work on your stack, and watch the signals above (throughput on the right work, held quality, cost per unit) so the impact is something you can see rather than something we claim. If you want a grounded read on what agents are actually worth on your codebase, that's the conversation to have. Talk to us about your team.

Short answer: How do you measure the ROI of AI agents?.

Common questions

How do I measure the ROI of AI coding agents?

Against your own baseline, on outcomes. Ask whether the team ships more of the work that matters, at held quality, with the same people, and whether your cost per unit of work went down. Avoid single productivity percentages; they're usually invented. A handful of honest signals beats one impressive-looking number.

What AI productivity metrics should I ignore?

Lines of AI-written code, suggestion-acceptance rate, percent of code written by AI, and raw commit or PR counts. All measure activity, all are easy to inflate, and none tells you whether anything valuable shipped or whether quality held.

Is 'percent of code written by AI' a good metric?

No. It rewards volume, not value, and it's trivially gamed. A team can raise it while shipping more low-quality code that costs more to review and fix. Measure outcomes shipped at held quality instead.

Why can't a vendor just give me a single ROI number for AI?

Because an honest one comes from your codebase, your baseline, and your definition of the right work, not a benchmark. Anyone quoting a clean universal percentage is selling, not measuring. The real read is a few signals tracked against where you started.

Want this running on your team?

Get your assessment