D1 · Agentic Architecture › Cluster 7 · Performance

7.3 · Prompt Caching in Agent Loops

Stable-first, or you pay to re-read every turn

CONCEPT Caching is a prefix match

Prompt caching stores the leading tokens of a request so later requests that share that exact prefix skip re-processing them. The critical property: it's a prefix match. Any byte change anywhere in the prefix invalidates everything after it. In an agent loop — where you replay the whole conversation every turn — a stable prefix is what makes the loop affordable.

🔑 Render order is tools → system → messages. Cache reads cost ~0.1× and writes ~1.25× of normal input tokens. Put stable content first, volatile content last so the expensive, unchanging parts stay cached across turns.
ChangeWhat it invalidatesWhy
Edit the toolsThe whole cache Tools render first — every downstream byte shifts.
Switch the model mid-loopThe whole cache Caches are model-scoped; a different model has no shared cache.
Change the system promptSystem + all messages after it System renders before messages.
Append a new message (normal loop)Nothing before it Growth at the tail — the whole prefix still matches.

DIAGRAM Prefix, breakpoint & the volatile tail

flowchart TD subgraph P["Cacheable prefix (stable, first)"] T["🟢 tools
render first"] --> S["🟢 system prompt"] S --> H["🟢 old conversation turns"] end H --> B{{"cache_control
ephemeral breakpoint"}} B --> V["🟠 new user turn / tool_result
(volatile, last)"] T -. "any byte change here" .-> X["❌ invalidates everything after"] classDef stable fill:#d4edda,stroke:#28a745,color:#155724; classDef vol fill:#ffe8cc,stroke:#d9480f,color:#7a2e08; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class T stable; class S stable; class H stable; class V vol; class X bad;

CODE Placing a cache breakpoint & verifying

Mark the end of the stable prefix with cache_control {"type":"ephemeral"}. Then verify the cache is actually working by reading usage.cache_read_input_tokens — if it's 0 across identical-prefix requests, something is silently invalidating it.

# Stable content first (tools → system), breakpoint at the end of the prefix.
resp = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=2048,
    tools=TOOLS,                          # renders first — keep it STABLE
    system=[{
        "type": "text",
        "text": SYSTEM_PROMPT,           # stable; NO datetime.now() in here!
        "cache_control": {"type": "ephemeral"},   # cache the prefix up to here
    }],
    messages=messages,                    # volatile tail grows each turn
)

# Verify the cache is live across identical-prefix requests:
print(resp.usage.cache_read_input_tokens)      # > 0  ✅ prefix reused
print(resp.usage.cache_creation_input_tokens)  # write cost (~1.25x)
if resp.usage.cache_read_input_tokens == 0:
    # silent invalidator: model switch, tool edit, or volatile system prompt
    log.warning("cache miss — check for a silent invalidator")

TRADE-OFFS What to cache

✅ Cache pays off when…

  • A big stable prefix (tools + system + long history) is replayed every turn
  • Reads are ~0.1× — huge savings on long agent loops
  • Model & tools are fixed for the whole run

⚠️ Caching backfires when…

  • The prefix churns — you pay ~1.25× writes with no reuse
  • You switch models or edit tools mid-loop (full invalidation)
  • A volatile value (timestamp) sits in the prefix

EXAM TRAPS

Trap: Putting datetime.now() (or any per-request value) in the system prompt. → It changes the prefix every call, so cache_read_input_tokens stays 0 — a silent invalidator. Keep volatile data in the last message.
Trap: Switching model mid-conversation to "save money." → Caches are model-scoped; a switch invalidates the whole cache and you re-pay the full write.
Trap: Editing/appending tools between turns. → Tools render first, so any tool change invalidates everything after it.
Trap: Assuming caching "just works." → Always verify via usage.cache_read_input_tokens; a 0 across identical prefixes means an invalidator is hiding in the prefix.

PRACTICE

Q1 · Your loop's cache_read_input_tokens is 0 every turn even though the system prompt "looks" identical. Prime suspect?

A per-request value baked into the prefix — classically datetime.now() in the system prompt. It changes one byte, so the whole downstream cache misses. Move it to the volatile last message.

Q2 · Mid-run you down-tier from Opus to Haiku to cut cost. What happens to the cache?

The entire cache is invalidated — caches are model-scoped. You re-pay the full write on the first Haiku call. Weigh that one-time cost against the per-token savings before switching.

Q3 · Where do you place the cache_control breakpoint in a tools+system+history request?

At the end of the stable region — after tools and system (and any fixed history), before the volatile new turn. Render order is tools → system → messages, so everything up to the breakpoint is the reusable prefix.