Chapter 5: Caching: Why Order Matters

Harness Engineering 101, Part I — The Wire. Series index · Prev · Next: Context Is a Budget

The failure: the agent loop works, but it costs money in a way that’s easy to miss. Every round resends the entire array, and the array grows every round. Let me put numbers on it. Suppose a coding task runs 40 rounds and each round adds about 2,000 tokens (a tool call plus its result). The array starts at 10,000 tokens (system prompt, tool schemas, the user’s request, some file context). Round 1 sends 10,000 input tokens. Round 2 sends 12,000. Round 40 sends 88,000. Total input tokens across the turn:

10,000 + 12,000 + 14,000 + … + 88,000 = 1.96 million tokens, for one user request.

The work grows with the square of the conversation length, because you pay for the whole history on every step. In the GPT-3.5 days this didn’t hurt, because conversations were short and nothing looped. The agent loop is what made it hurt.

The patch: the provider caches the part of your array it has already processed, and charges you a tenth of the price for it. But the cache only works if your harness keeps the array byte-stable. This chapter is about what that means and how it changes the way you build.

How prefix caching works

When a model reads your array, it processes tokens front to back, building up internal state as it goes. That state is expensive to compute. Prompt caching means the provider stores it, keyed on the exact bytes processed so far. On your next request, it compares your new array against the stored one, front to back, finds the longest matching prefix, and skips recomputing it. Only the new tail gets processed at full price.

Cached tokens cost roughly 10% of normal input price on Anthropic (cached input is similarly discounted on OpenAI, which caches automatically). So the economics of the agent loop become:

• Round 1: process 10,000 tokens, full price. Cache them.
• Round 2: the first 10,000 tokens match the cache. Pay 10% for those, full price only for the new 2,000.
• Round 40: pay 10% on 86,000 cached tokens, full price on 2,000 new.

The conversation, the model and the replies don’t change, but the cost drops by about 80 to 90 percent. Latency drops a lot too, because the provider actually skips recomputing those tokens instead of only giving a billing discount.

Generated by diagrams/gen_cache_economics.py.

This works because the agent loop is append-only. Each round adds messages at the end and touches nothing earlier. An append-only array has a perfectly stable prefix, which is a perfectly cacheable prefix. The loop we built in chapter 4 already has this shape, and your job now is to keep it that way.

The one rule: never edit the top

The cache matches bytes from the front. The moment byte 1,000 differs from the cached version, everything after byte 1,000 is recomputed at full price, even if the remaining 80,000 tokens are identical. So the rule is:

Appending is cheap. Editing anything above the append point costs you everything below it.

This sounds easy to follow, but it’s also easy to break. I have broken it many times, including in One Code, and I still do now and then. Here are the classic accidental cache-busters:

• A timestamp in the system prompt. Current time: 14:32:07 at the top of the array means no request ever hits the cache. If the model needs the date, put it somewhere stable (the date, not the second), or inject it low in the array.
• Reordering tools. Tool schemas are part of the prefix. If you build the tool list from an unordered dict, so it serializes in a different order per process, the cache is gone. Sort your tools.
• Changing the tool list between turns. I found this one recently in WSO2 BI Copilot. Users can turn MCP servers (chapter 11) on and off, and every toggle edited the tool array, so the next request missed the cache. Fetching MCP tools fresh before every request can do the same, because any change in what the servers return changes the prefix.
• “Improving” the system prompt mid-session. Any conditional text up top (“the user seems frustrated, add a tone note”) rewrites byte one.
• Rotating content in place, like keeping a live “current status” section near the top of the array and updating it each round.
• Removing old messages from the middle to save space. This one hurts the most. Trimming the array to make it smaller can actually make it more expensive, because the trim invalidates the prefix. Context reduction has to be done in deliberate, occasional jumps (chapter 6), not in small continuous trims.

This rule also shapes the design, beyond avoiding bugs. New information should enter the array at the bottom. When the harness must tell the model something mid-session (a file changed on disk, the current todo list), append it as a new message near the end. Don’t update some canonical block near the top. Chapter 8 builds a whole steering mechanism on this principle, and it exists because of this chapter.

Production harnesses treat prefix stability as an invariant with tests. In One Code, the system prompt stays byte-stable across turns unless something genuinely changed. Payload-capture tests (chapter 12) check this, because one careless byte near the top causes a 10x price increase with no error or visible failure. You pay full price on every request and only notice if you track your cache-hit rate. API responses report cache_read_input_tokens, and that number should account for most of your input tokens on every round after the first.

In a complex harness, the final array is assembled fresh for every request from many sources: the system prompt, the tool definitions, MCP servers, memory files and reminders. Any one of them can change a byte near the top, so a cache break is easy to cause and hard to see. Because I keep breaking it, I don’t trust myself here anymore. Before every release and pull request, I run a cache probe. It sends a short multi-turn conversation to the real API, captures every request, and checks that each request reads what the one before it cached. Unit tests can’t catch a broken cache, because the code still works. It only shows up in the bill.

Cache breakpoints (Anthropic) vs automatic (OpenAI)

OpenAI caches automatically. Send a request whose prefix matches a recent one and the discount appears. There is nothing to configure.

Anthropic makes it explicit. You mark up to four cache breakpoints in the array with cache_control markers:

{"type": "text", "text": "...end of system prompt...",
 "cache_control": {"type": "ephemeral"}}

A breakpoint says “cache everything up to and including this block.” The natural layout for an agent has breakpoints at the stable frontiers:

flowchart TD
    subgraph array [the array, front to back]
        A["tool schemas<br/>(never change mid-session)"] --- B["system prompt<br/>(never changes mid-session)"]
        B --- C["conversation history<br/>(grows every round)"]
        C --- D["newest messages<br/>(this round's additions)"]
    end
    A -.breakpoint 1.-> X1[cached once, hit forever]
    B -.breakpoint 2.-> X2[cached once, hit forever]
    C -.breakpoint 3, moved each round.-> X3[re-cached incrementally]

The part worth remembering is that the third breakpoint moves. Each round you place it on the newest message. The provider then caches through that point, so next round’s prefix match covers everything you have sent so far. Explicit control costs a little code (a 25% write surcharge on newly cached tokens, on Anthropic). In return, you know exactly what is cached and can design the array around it.

Either way, the discipline is the same, because the provider only rewards a harness that keeps its prefix stable. Caching is not a setting you flip on; it only works if your harness keeps the prefix stable.

Cache lifetime, briefly

Caches expire. Anthropic’s default entries live about 5 minutes (refreshed on every hit; a paid 1-hour option exists), OpenAI’s several minutes to an hour depending on load. For an active agent loop this does not matter, because rounds are seconds apart and the cache stays hot. It hurts during the human pause. A user who reads your agent’s answer for ten minutes and then replies pays a full-price re-read of the whole array. The harness can’t fix this, because you don’t control the provider’s cache TTL, so the first request after a long pause will be expensive. When a turn costs 10x as much as the previous one, an expired cache is usually the reason.

What this changes about your thinking

Chapter 1 said the harness “builds and maintains the array.” This chapter adds the constraint that makes that job interesting. The array is also a physical layout with a price gradient. The top of the array is frozen and cheap, and you pay if you touch it. The bottom is fluid, and all new information lands there. Every later chapter respects this gradient:

• Context management (chapter 6) trims in rare, deliberate jumps because every trim is a cache reset.
• Steering (chapter 8) injects at the bottom, never edits the top.
• Memory and instructions ride in the first user message in Claude Code rather than the system prompt partly so the system prompt can stay identical across sessions and features.
• Debugging (chapter 12) watches cache_read_input_tokens as a vital sign.

What you now know

• Prefix caching matches your array byte for byte from the front and discounts the matched part by about 90%.
• Appending keeps the cache. Changing anything above the append point loses it for everything below, and nothing tells you.
• Watch cache_read_input_tokens. A low cache-hit rate is an expensive bug.

Part I is complete: the wire, the brain, tools, the loop, and the economics. Part II is about what happens when the loop runs long: the array approaches the context window, and everything from here on is about spending that space well.

Next: Chapter 6 — Context Is a Budget, Not a Bag