Every engineering org using AI coding agents is climbing the same ladder. Most don’t know they’re on it. And at almost every rung, they measure the wrong thing.
We know because we watched it happen. Not just on our own teams, but on customer teams running Wingman inside their coding flow. Same ladder, same traps, over and over. We climbed it ourselves too, measuring the wrong things the whole way up. Here is the ladder, the trap on each rung, and the number that would have saved everyone the trouble.
The ladder
Adoption is not a switch you flip. It is a ladder you climb, one rung of trust at a time. Each rung hands the agent a little more of the loop.
Rung one: autocomplete
The agent suggests, you accept or ignore. The mode I like to call Clippy for code. “It looks like you are trying to write a while loop. Would you like help with that?” It finishes your line, fills in the obvious boilerplate, guesses the next function. You are still doing all the driving. The agent is a very good tab key.
The wrong metric here is code accuracy. Teams grade the quality of individual completions, as if a better suggestion were the point. But at this rung the agent is barely involved. Grading its completions is like grading a spellchecker on your novel.
It is also nearly impossible to grade at scale. Who wrote which line, the engineer or the agent? Nobody can untangle that across a team, so the signal that actually reaches leadership is not a metric at all. It is a vibe. Engineers report back “it doesn’t really help me that much,” or worse, “it slows me down because I have to fix what it writes.” That feeling is the real data at this rung, and it is telling you the agent does not have enough of the problem to be useful yet.
Rung two: constrained-context prompting
You scope tightly. One or two files the agent is allowed to touch, a task you can describe in a sentence, a fence around the work. The agent implements inside the fence while you watch. Higher leverage than autocomplete, still fully supervised.
The wrong metric here is agent calls. People start counting invocations, treating activity as progress. But a hundred small prompts is not more work done than ten good ones. It is often less.
There is a deeper cost at this rung, and it is the one that caps everything below it. When you fence the agent to two files and a one-sentence task, you have told it what to type. You have not told it what problem you are solving. It does not know the customer pain behind the ticket, where the flow actually breaks, which paths are dead ends, why the code is shaped the way it is. So it cannot do the expensive, valuable part: understanding the system well enough to find the fix you did not already know. You have hired a reasoning engine and handed it a stack of sticky notes.
This is also the rung people camp on. It feels productive, and it is safe, so engineers settle in and stop climbing. Tightly scoped prompting gives you a steady drip of small wins with no scary surprises, which is exactly why it becomes a local maximum. The work gets done, the agent never does anything alarming, and the team quietly concludes this is what AI coding is. They mistake a comfortable rung for the top of the ladder. The tell is a team that has used agents for months and still describes them as “helpful for boilerplate,” because boilerplate is all they have ever asked for.
Rung three: approve every action
Now the task is bigger, but you gate everything. Every file edit, every tool call, every command. You hit enter a thousand times. It feels responsible. It feels like control.
The wrong metric here is token burn. The spend climbs, so you gate harder to hold it down. But this is the sneakiest rung of all. You are paying frontier prices while hand-approving every step, so you get neither the safety of tight scoping nor the speed of real autonomy. You have built a bottleneck out of yourself.
And the safety you think you are buying is mostly theater. By the fortieth approval prompt you are not reviewing, you are rubber-stamping. The mind glazes, the pattern becomes “read the first line, hit enter,” and the one dangerous edit in the batch slides through on the same reflex as the forty harmless ones. Gating every action does not make you more careful. It spends your attention so fast that you have none left for the step that mattered.
There is a second trap waiting here, and it is the one that stalls people. This is the rung where the bill first gets big enough to notice. You are running a capable model over a real task with a human in the loop the whole time, and the spend is visible and climbing. So the instinct is to clamp down, gate harder, shrink the task, maybe drop to a cheaper model. Every one of those moves is a step back down the ladder, taken because a number on a dashboard got scary. The cost was not the problem. The cost was the receipt for work that, measured properly, was paying for itself.
Rung four: full auto
You point the agent at the codebase, describe the problem, and let it run. Maximum leverage. Maximum nerve.
The nerve is manageable, though, and this is the part people skip. Full auto does not mean unsupervised chaos. It means you move the safety from approving every keystroke to building an environment where the agent cannot do real damage in the first place. Let it work on a branch, not main. Give it a sandbox, not your prod database. Point it at the CI/CD pipeline you already built, the tests, the checks, the review gates, and let those catch what a human hitting enter forty times never would. The guardrails move from your fingers to your infrastructure, which is where they should have been all along. Done right, you can let the agent run hard precisely because the blast radius is contained.
Our CEO made this case concretely for security fixes: when our agent applies a fix, it never merges or touches main. It patches locally, then the change gets in line behind the same code review, tests, and CI every other change goes through. If you already trust that pipeline to catch a regression from a human, it catches one from an agent the same way. Letting the agent run is not a leap of faith. It is a bet on the guardrails you already built.
The wrong metric here is token burn again. Same trap as rung three, higher stakes. Now the bill is big enough to be scary, and fear of the number stops people from letting a capable agent actually finish. You climbed all the way up the ladder just to flinch at the top.
The pattern nobody names
Look at the four rungs together and the same mistake repeats. At every level, teams grab the metric that is easiest to see and treat it as the metric that matters.
Code accuracy. Agent calls. Token burn. Token burn. Every one of them is a local proxy. Every one of them measures the machinery instead of the output. And every one of them can go green while the thing you actually care about goes nowhere.
We did this. For a stretch we watched agent-call volume like it meant something. Later we watched token spend like it was the score. It was not. We were tuning the dials and ignoring the road.
What the ladder is really measuring
Here is the thing the rungs have in common, the reason climbing helps at all. It is not mainly about how much you approve. It is about how much of the problem you are willing to share.
Low on the ladder, you hand the agent keystrokes. A function to write, a file to edit, a symptom to patch. You keep the actual problem, the customer pain, the flow, the reason any of this exists, in your own head. High on the ladder, you hand the agent the problem itself and let it do the understanding: read the code, trace the path, find where it dead-ends, propose the fix you had not thought of.
Control and context rise together as you climb, but context is the one that pays. The higher rungs are not better because you supervise less. They are better because you finally tell the agent what you are actually trying to do. And a model that cannot reason across that context will fail at the top no matter how much rope you give it, which is why the model tier underneath all this matters. We wrote about that separately, in why cheap AI coding models cost more than they save. Context and capability are the two things the lower rungs starve.
We are not the only ones drawing ladders like this. Boris Cherny, who runs Claude Code at Anthropic, published his own “Steps of AI Adoption” that maps the same climb by a different axis: how many agents one person runs, from zero to a thousand and up.
His is the view from the frontier. Ours is the view from the field, where the question is not how many agents you can run but whether any of it reached a customer. His ladder counts agents. Some of his readers noticed those counts also track token spend and subscription tiers, which is exactly the trap this post is about. Count agents, tokens, or calls and you are still counting the machinery. We would rather count the fix that landed in a customer’s hands. Same ladder, opposite thing to measure.
The metric that survives every rung
There is exactly one thing worth measuring from autocomplete to full auto. Not a number you can copy from this post, though. The thing itself: how much product reaches customers and solves their problem.
Here is the honest part nobody puts on a slide. There is no universal delivery metric, and anyone who sells you one is wrong. Delivery looks different depending on what you ship. A SaaS product might watch feature adoption or task-completion rate. An infrastructure team might watch lead time to production, how fast a change gets from a ticket to something running. A consumer app might watch clicks to outcome, whether the user solved their problem in fewer steps than last month. The right metric is business-specific, and that is not a cop-out. It is the reason the proxies win by default: token burn and agent calls are the same for everybody, so they are easy to reach for. Delivery makes you do the work of knowing your own business.
So here is the rule for picking yours. Find the thing that sits closest to the customer actually getting their problem solved, and measure that. Not the closest thing that is easy to count. The closest thing that is true. If your candidate metric can go up while customers are no better off, it is still a proxy, and you have not found it yet.
Here is ours, so the rule is not just theory. The number we hold as delivery is fixes our loop produces for customers, actual vulnerabilities found, fixed, and verified in their applications. That is the thing closest to a customer’s problem getting solved. We watch two leading indicators underneath it, because they are what make fixes possible: time to a successful scan, and number of apps under test. You cannot produce a fix without a successful scan, and you cannot scan an app that is not under test. So those two feed the delivery number rather than competing with it. All three moved significantly after we rolled out Wingman. The point is not the numbers. The point is that we kept the easy-to-count metrics in their place, as feeders for the one that actually measures whether we helped anyone.
Once you have your delivery metric, the wrong ones fall away on their own. Token spend high but delivery climbing? Fine, you are buying output. Agent calls low but delivery strong? Also fine. The proxies only ever mattered as guesses about delivery. Measure the real thing and you can stop guessing.
Better still, delivery makes the ladder self-directing. You do not climb to the next rung because a blog told you to. You climb when delivery stalls and the current rung is the reason. The bottleneck tells you where to go. If approving every action is what is slowing your shipping, that is your signal to loosen the gates. If autocomplete is capping how much one engineer can move, that is your signal to scope real tasks. Delivery names your next bottleneck for you.
The edges we skipped
The ladder has two ends we have not talked about. Below the bottom rung are the teams who refuse to climb at all, still doing everything by hand. Past the top rung are the teams pushing beyond full auto into something closer to autonomous software factories. We have opinions about both, but we have not watched either closely enough to say anything useful yet, so we will leave them for another day. This post is about the rungs almost everyone is actually standing on.
Where to start
You are already on the ladder. The only question is whether you are measuring the rung or the outcome.
So find out what you are optimizing right now. If it is completions, calls, or cost, you are watching a proxy. Swap it for the one number that was ever real: are you shipping more, faster, that customers actually use. Then let that number tell you which rung to climb next.
We spend a lot of time thinking about what happens to security when teams move up this ladder, especially at the top, where code ships faster than any human can review it. That is a separate conversation, and one we are glad to have. But it starts here, with measuring the right thing.
