Everyone has an opinion about coding agents: they're too expensive, or they're magic, or the code they write is garbage. I lead the frontend of a ~120k-line TypeScript app where agents write most of the code, so I wanted numbers instead of opinions.
It turns out the data was already on my machine.
The data Claude Code already keeps
Claude Code writes every session to ~/.claude/projects/**/*.jsonl. Each line is an event, and two kinds of event matter here:
-
Assistant responses include a
usageblock with token counts: uncached input, cache writes (split by 5-minute and 1-hour TTL), cache reads and output. -
Tool calls include every
Edit,WriteandBashcommand the agent ran, with its output.
That's enough to answer two questions:
- Is prompt caching actually working, and what does it do to cost?
- When the agent edits code, how often does the first typecheck or test run pass?
I wrote a small CLI, claude-code-metrics, that reads those files and answers both. Nothing leaves your machine.
npx tsx src/cli.ts # everything
npx tsx src/cli.ts --since 2026-10-01 # a period
npx tsx src/cli.ts --project my-app # sessions whose cwd contains "my-app"
Result 1: prompt caching
Over 8 weeks and 7.5B input tokens:
| Value | |
|---|---|
| Input tokens read from cache | 98.4% |
| Input cost vs. no caching | 87.4% lower |
| Responses | 30,439 |
How the savings are computed
Prompt caching changes the price of each input token, depending on where it comes from:
| Token type | Price vs. base input |
|---|---|
| Uncached input | 1× |
| Cache write, 5-minute TTL | 1.25× |
| Cache write, 1-hour TTL | 2× |
| Cache read | 0.1× |
So the savings are:
savings = 1 − actual / uncached
actual = input×1 + write5m×1.25 + write1h×2 + read×0.1
uncached = (input + write5m + write1h + read) × 1
The base price cancels out, so the ratio holds for any model. In code it's a few lines:
export const MULTIPLIERS = { input: 1, write5m: 1.25, write1h: 2, read: 0.1 };
export function ratios(t: TokenTotals): CacheRatios {
const uncached = t.input + t.write5m + t.write1h + t.read;
if (!uncached) return { readShare: 0, savings: 0 };
const actual =
t.input * MULTIPLIERS.input +
t.write5m * MULTIPLIERS.write5m +
t.write1h * MULTIPLIERS.write1h +
t.read * MULTIPLIERS.read;
return { readShare: t.read / uncached, savings: 1 - actual / uncached };
}
One gotcha: Claude Code logs one line per content block, all with the same message id. If you don't deduplicate by id, you count the same response several times.
Why the number is so high
An agent session re-sends a long, mostly stable prefix on every turn: system prompt, tools, project instructions, the conversation so far. Only the tail changes. That's the ideal case for caching: you pay to write the prefix once, then read it at 10% of the price for the rest of the session.
Update (Nov 3): a reader pointed out that edits made from Bash (sed -i, cat >, tee) didn’t arm a check. They do now, and attempts went from 503 to 1,942. Updated rates: 84.8% typecheck and 76.2% tests on the first run; 86.5% of tests pass on unfiltered runs only (n=89). Thanks for the catch!
Result 2: first-try pass rate
This is the number I care about most, because it measures how good the agent's first attempt is, not how good it gets after five retries.
| Check | Attempts | Passed on first run |
|---|---|---|
| Typecheck | 205 | 86.3% |
| Tests | 298 | 75.2% |
How "first try" is defined
Each session is replayed in order:
- An edit (
Write,Edit,MultiEdit) to a matching file arms a check:.ts/.tsx/.vuearms typecheck, code files arm tests. - The next Bash command that runs that check (
tsc,jest,vitest,npm test, …) is the first try. Its output decides pass or fail (error TS…,N failed,not ok). - Reruns before the next edit are ignored. They measure iteration, not the first attempt.
The caveat that matters most
Agents love to pipe long output through head, grep or tail, and a filtered output can hide the failing line. So the CLI also reports the rate on unfiltered runs only:
| Check | Unfiltered runs | Pass rate |
|---|---|---|
| Typecheck | 10 | 100% |
| Tests | 21 | 52.4% |
The samples are small, but the test number is a warning: treat 75% as an upper bound. This is also why, in my workflow, the agent never commits without a gate that re-runs typecheck and tests with full output.
Other caveats
- Subscription plans aren't billed per token. There, the 87% means "what these tokens would have cost on the API": a measure of efficiency, not money saved.
- This is observation, not an eval. Tasks aren't controlled. To compare prompts or context, you need a harness that runs the same tasks under each variant. That's what I built agent-context-evals for.
- Pre-existing failures make every attempt look like a failure. Measure on a clean baseline.
- Transcripts get cleaned up over time, so the period is whatever is still on disk.
What I took away
- Caching is not something to tune. It's something not to break. Keep the prefix stable (instructions, tool definitions, project context) and the savings come on their own.
- Agents are good, not reliable. 86% on the first typecheck is impressive. The other 14% is why verification gates aren't optional.
- Measure your own workflow. Your numbers will be different from mine, and the data is already sitting on your disk.
The code is MIT-licensed: github.com/diego-prieri/claude-code-metrics. I'd love to see other people's numbers. If you run it, share yours in the comments.
Top comments (6)
Your Nov 3 update is a good example of why the metric definition matters: counting Bash edits moved attempts from 503 to 1,942, and filtered test output can still make the first-run rate optimistic. In
agent-context-evals, would you pair each transcript-derived attempt with an independent, unfiltered repository gate on a clean baseline? That would distinguish the agent’s reported pass from a verified green task.Thanks, that’s the right distinction. Short answer: in agent-context-evals the gate is already independent. The harder part is the pairing.
What the harness already does:
Every run starts in a fresh git worktree of the base commit.
The agent runs with Bash disabled. It can’t run tsc or tests, so there’s no agent-reported pass to trust. All it produces is a diff.
After the agent exits, the harness runs typecheck and tests itself. It reads the full output and the exit code, with nothing piped through head or grep.
Typecheck is diffed against a baseline computed once per base commit. Errors match by file, code and message, so pre-existing errors don’t count against the agent.
One gap: tests don’t have a baseline yet, only an exit code. A test that already fails on the base commit would fail every run, so for now you have to start from a green suite. A per-commit test baseline is the obvious next fix.
What it doesn’t do is pair those results with transcript-derived attempts. The two tools answer different questions. claude-code-metrics watches real sessions, where the agent runs its own checks. The harness controls the task and verifies it from outside. For a real session I can’t replay the gate later, because the transcript records the edits and the command output but not the full repository state at each attempt.
The way to get the pairing is prospectively: a hook that runs the full gate, unfiltered, whenever the agent runs a check, and logs it next to the transcript. Then every first try has two verdicts, what the agent saw and what the repo says, and the gap between them is exactly the optimism you’re describing. That’s a good addition. Thanks for pushing on it.
98.4% cache reads is the number that hides the next one. Caching discounts input, but it can't touch which model you're paying for or how much output effort generates. In my own month, 89% of spend sat on Opus, including lookups that passed on Haiku for 1/50th of the price. You built the meter; rightsize is the steering, with its own transcript audit, also local. github.com/spar-x-fr/spar-x-skills...
Fair point, and my own data agrees. Caching only tells you how cheaply you paid for input. It says nothing about whether you should have used that model in the first place.
The CLI already breaks usage down by model, and it’s lopsided: about 75% of my responses ran on Opus, 21% on Sonnet and barely 1% on Haiku. I haven’t priced that split or checked which of those Opus turns a cheaper model could have handled. That’s the next question, and it’s harder, because “passed on Haiku” needs a definition of passed.
Adding cost per model and output tokens to the report is now on my list. Thanks for pushing on it.
Eight weeks of one person's sessions is a sample of one. The honest part is the pass rate beside the 87%. Lower cost only counts for the tasks that finished.
Agreed on both counts. It’s one person, one codebase and one way of working, so I’d read it as a measurement of my workflow, not a benchmark of Claude Code. That’s also why the tool is open: the interesting data is what other people get.
And yes, cost per finished task is the number that actually matters. The CLI measures cost per token and first-try quality separately, but never joins them. Tying spend to tasks that actually landed would need a definition of “done” that transcripts don’t carry, like a merged commit or a closed ticket. That’s a good next step.