Token volume measures spend, not output. Leverage is what your agents turn into verified, valuable change per unit of human attention, compute, and rework, and no single number captures it. You have to triangulate from several points of view.
Author: James Phoenix | Date: July 2026
19.3 Billion Tokens Is Not a Result
My Codex usage stats read like a flex: 19.3 billion lifetime tokens, a 1.64 billion token peak month, a longest autonomous task of 4 hours 45 minutes, and a contribution grid that is saturated nearly every day for a year. For a while I treated numbers like these as evidence that the system was working.
They are evidence of exactly one thing: that I spent the tokens. A billion tokens can represent productive autonomous execution, or an agent solving the wrong problem very thoroughly, and the token counter cannot tell the difference. Repeated failed loops, excessive context replay, and confidently wrong refactors all burn tokens at the same rate as good work.
So the question I actually care about is leverage, and I define it as a ratio:
verified valuable changes
agent leverage = ─────────────────────────────────────
human attention + compute cost + rework
The numerator is not “code produced.” It is changes that shipped, survived review, and did what they claimed. The denominator is everything the change cost me: minutes of my attention, dollars of compute, and every hour spent reverting or repairing what the agent left behind.
The problem is that this ratio is not directly observable. You cannot query “verified valuable changes” from any dashboard. What you can do is measure the same underlying leverage from multiple vantage points and check whether the views agree. That is the point of the tier structure below: each tier is a different lens with a different failure mode, and no tier is trustworthy on its own.
Tier 0: Volume, Disqualified on Arrival
There is a tier below the ladder, and it exists only to be excluded. Lines of code, commit counts, and token volume are all the same metric wearing different clothes: they measure how much material moved, not whether any of it mattered. Lines of code was already a discredited productivity measure when humans wrote every line. Agents make it actively inverted, because an agent will happily generate 40,000 lines you did not need, and some of the highest-leverage agent sessions I run end with fewer lines than they started with.
Any metric an agent can maximise by simply running longer is disqualified as a leverage measure by construction. Tokens, LoC, and commits all fail that test instantly, which is why Tier 0 is not the bottom rung of the ladder. It is the floor the ladder stands on, and nothing on it counts.
Tier 1: The Reliability Ratio
The fastest lens is engineering hygiene, and I compress it into one headline ratio:
successful PRs / (bugs + outages + reverts)
A successful PR is one that merged and did not come back. The denominator is everything that proves a merge was not actually successful: post-merge defects, incidents traced to agent changes, and reverted commits. Supporting signals that feed the same tier:
- Accepted versus reverted agent changes
- First-pass test success rate (did the agent’s first submission go green?)
- Human review minutes per merged change
- Repeated tool calls and context-cache efficiency per completed task
This tier is the control loop for the harness itself. It updates daily, it is cheap to instrument from git history and CI logs, and when it degrades I know within a day that something in the environment broke: a flaky lane, a rotted lint rule, a prompt that stopped matching the codebase.
It is also the most gameable tier. An agent fleet can pump this ratio to infinity by shipping a hundred trivial, safe, worthless PRs. Renames, comment fixes, and dependency bumps all merge cleanly and break nothing. If Tier 1 is your only lens, you have built the metric-gaming trap described in Goodharting Prevention in Agent Systems, except the entity gaming the metric is your own operation. Tier 1 measures reliability of execution, not value. It needs a lens above it.
Tier 2: Feature Releases
The second vantage point asks a harder question: did all those reliable merges accumulate into capability a user can touch? The unit here is the shipped feature, not the PR. Signals:
- Features released per month, against the roadmap that existed at the start of the month
- Cost per completed spec (tokens plus human minutes, from ticket to release)
- Browser-flow success rate: does the end-to-end user journey pass, not just the unit tests?
This is where the Playwright loop earns its keep. A merged PR with green tests proves the code compiles and satisfies its own assertions. A passing browser flow proves the product works, which is a different claim. I treat browser-flow success as the boundary between Tier 1 and Tier 2: it is the cheapest available proxy for “a user would experience this as working.”
Tier 2 corrects Tier 1’s blind spot. A hundred trivial PRs produce zero feature releases, so the divergence is immediately visible: Tier 1 looks great, Tier 2 is flat, and the diagnosis is that the agents are busy rather than productive. But Tier 2 has its own failure mode. You can ship features nobody wanted at an impressive cadence. Feature count measures throughput of intent, and if the intent was wrong, the tier is silent about it.
Tier 3: Revenue and Subscriptions
The top lens is the only one that cannot be gamed from inside the engineering system: did the work move revenue, subscriptions, retention, or whatever the business actually runs on? For a SaaS this is new subscriptions attributable to shipped capability, expansion revenue, and churn that a fixed bug prevented.
Tier 3 is ground truth, and it is nearly useless as a daily instrument. It lags by weeks or months, the attribution is noisy, and a marketing change can swamp the signal from any engineering improvement. You cannot debug your agent harness with quarterly revenue, but you also cannot declare the harness successful without it. Tier 3 exists to audit the tiers below on a slow cadence, not to steer them.
Reading the Tiers Against Each Other
The layering is the actual method. Each tier is fast but gameable, or true but slow, and the disagreements are more informative than any single reading:
| Divergence | Diagnosis |
|---|---|
| Tier 1 high, Tier 2 flat | Agents are busy, not productive. Work selection is broken, not execution. |
| Tier 2 high, Tier 3 flat | Execution is fine, planning is wrong. You are shipping the wrong roadmap. |
| Tier 1 degrading, Tiers 2-3 fine (for now) | The harness is rotting. Rework is about to eat the upper tiers. |
The second row is worth sitting with, because it relocates the problem entirely outside the agent system. When reliable agents ship features that do not convert, the failure is at the planning boundary, which is exactly where human leverage is supposed to concentrate. The agents did their job. I picked the wrong job.
The denominator deserves its own instrumentation too. Compute cost under agent throughput behaves nothing like it did at human commit rates, which is the whole argument of Agents Broke the Economics of Your CI, and the attention-plus-rework side is the asymmetry at the heart of The Harness Is Cheaper Now: debugging agent output costs more per minute than writing your own code, so rework silently dominates the ratio if you do not count it.
What I Actually Track
The minimal dashboard, in order of cadence:
- Daily: accepted vs reverted agent changes, first-pass test success, review minutes per merge (Tier 1)
- Weekly: features released, cost per completed spec, browser-flow pass rate (Tier 2)
- Monthly or quarterly: subscriptions and revenue movement mapped against the release log (Tier 3)
None of these numbers is impressive on its own, and that is the point. The 19.3 billion tokens were never the achievement. The achievement is an environment where billions of tokens reliably collapse into reviewed, tested, revenue-bearing work, and the only way to know that is happening is to measure it from more than one altitude.
