CONCEPT Caching is a prefix match
Prompt caching stores the leading tokens of a request so later requests that share that exact prefix skip re-processing them. The critical property: it's a prefix match. Any byte change anywhere in the prefix invalidates everything after it. In an agent loop — where you replay the whole conversation every turn — a stable prefix is what makes the loop affordable.
| Change | What it invalidates | Why |
|---|---|---|
| Edit the tools | The whole cache | Tools render first — every downstream byte shifts. |
| Switch the model mid-loop | The whole cache | Caches are model-scoped; a different model has no shared cache. |
| Change the system prompt | System + all messages after it | System renders before messages. |
| Append a new message (normal loop) | Nothing before it | Growth at the tail — the whole prefix still matches. |
DIAGRAM Prefix, breakpoint & the volatile tail
render first"] --> S["🟢 system prompt"] S --> H["🟢 old conversation turns"] end H --> B{{"cache_control
ephemeral breakpoint"}} B --> V["🟠 new user turn / tool_result
(volatile, last)"] T -. "any byte change here" .-> X["❌ invalidates everything after"] classDef stable fill:#d4edda,stroke:#28a745,color:#155724; classDef vol fill:#ffe8cc,stroke:#d9480f,color:#7a2e08; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class T stable; class S stable; class H stable; class V vol; class X bad;
CODE Placing a cache breakpoint & verifying
Mark the end of the stable prefix with cache_control {"type":"ephemeral"}. Then
verify the cache is actually working by reading
usage.cache_read_input_tokens — if it's 0 across identical-prefix requests, something is
silently invalidating it.
# Stable content first (tools → system), breakpoint at the end of the prefix. resp = client.messages.create( model="claude-opus-4-8", max_tokens=2048, tools=TOOLS, # renders first — keep it STABLE system=[{ "type": "text", "text": SYSTEM_PROMPT, # stable; NO datetime.now() in here! "cache_control": {"type": "ephemeral"}, # cache the prefix up to here }], messages=messages, # volatile tail grows each turn ) # Verify the cache is live across identical-prefix requests: print(resp.usage.cache_read_input_tokens) # > 0 ✅ prefix reused print(resp.usage.cache_creation_input_tokens) # write cost (~1.25x) if resp.usage.cache_read_input_tokens == 0: # silent invalidator: model switch, tool edit, or volatile system prompt log.warning("cache miss — check for a silent invalidator")
TRADE-OFFS What to cache
✅ Cache pays off when…
- A big stable prefix (tools + system + long history) is replayed every turn
- Reads are ~0.1× — huge savings on long agent loops
- Model & tools are fixed for the whole run
⚠️ Caching backfires when…
- The prefix churns — you pay ~1.25× writes with no reuse
- You switch models or edit tools mid-loop (full invalidation)
- A volatile value (timestamp) sits in the prefix
EXAM TRAPS
datetime.now() (or any per-request value) in the system
prompt. → It changes the prefix every call, so cache_read_input_tokens stays 0 — a
silent invalidator. Keep volatile data in the last message.usage.cache_read_input_tokens; a 0 across identical prefixes means an invalidator is
hiding in the prefix.PRACTICE
Q1 · Your loop's cache_read_input_tokens is 0 every turn even though the system prompt "looks" identical. Prime suspect?
A per-request value baked into the prefix — classically datetime.now() in the
system prompt. It changes one byte, so the whole downstream cache misses. Move it to the volatile last
message.
Q2 · Mid-run you down-tier from Opus to Haiku to cut cost. What happens to the cache?
The entire cache is invalidated — caches are model-scoped. You re-pay the full write on the first Haiku call. Weigh that one-time cost against the per-token savings before switching.
Q3 · Where do you place the cache_control breakpoint in a tools+system+history request?
At the end of the stable region — after tools and system (and any fixed history), before the volatile new turn. Render order is tools → system → messages, so everything up to the breakpoint is the reusable prefix.