CONCEPT Let the conversation grow past a moving breakpoint
In an agent loop or chat, the conversation history keeps growing — and each turn's history is the next turn's stable prefix. The caching pattern: place a breakpoint after the stable prefix (system + tools + settled history) and let new turns accumulate after it. On the next request, that now-larger history is a byte-identical prefix, so it reads from cache; only the newest turn is written. As history settles, you move the breakpoint forward to keep capturing the growing stable region.
CONCEPT What invalidates it — and the 20-block window
Because caching is a prefix match (2.3), anything that alters the front of the prompt mid-run wipes the cache and forces a cold re-write:
- Switching models — caches are model-scoped. Moving from Opus to Haiku mid-loop can't read the other model's cache. Keep one model per loop; spawn a subagent for a cheaper side-task instead.
- Editing cached bytes — mutating the system prompt, adding/removing/reordering tools, or rewriting settled history changes the prefix and invalidates everything downstream.
- The 20-block lookback — a breakpoint walks back at most 20 content blocks to find a prior entry. A single turn that adds >20 blocks (common in tool-heavy loops) can outrun the window and silently miss. Place an intermediate breakpoint every ~15 blocks in long turns.
{"role":"system"} message
instead of editing the top-level system prompt (preserves the cached prefix on supporting models); use tool
search to append tool schemas rather than swapping the tool list.DIAGRAM The breakpoint advances each turn
whole prefix"| A2 classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; classDef neutral fill:#e7f1ff,stroke:#4a90d9,color:#1a4971; class BP1 ok; class BP2 ok; class BP3 ok; class MODELSWAP bad; class A1 neutral; class A2 neutral; class A3 neutral;
CODE A growing conversation with a moving breakpoint
Append each turn, and put the breakpoint on the last block of the newest settled turn. The ever-larger history reads from cache; only the new tail is written.
messages = [] def turn(user_text): messages.append({"role": "user", "content": user_text}) # Move the breakpoint to the last block of the newest turn messages[-1]["content"] = [ {"type": "text", "text": user_text, "cache_control": {"type": "ephemeral"}}, # advances each turn ] resp = client.messages.create( model="claude-opus-4-8", # SAME model every turn — never swap mid-loop max_tokens=1024, system=[{"type": "text", "text": FROZEN_SYSTEM, # never edited in place "cache_control": {"type": "ephemeral"}}], messages=messages, ) messages.append({"role": "assistant", "content": resp.content}) return resp # Turn 2+ reads the whole prior history from cache (~0.1×); # only the newest turn is written.
TRADE-OFFS Fixed model & prompt vs. mid-loop flexibility
✅ One model, frozen prefix
- History reads from cache every turn — huge savings in long loops
- Latency stays low as context grows
- Breakpoint simply advances with the conversation
⚠️ Changing things mid-loop
- Switching models → cold cache (model-scoped)
- Editing system / reordering tools → full prefix rebuild
- >20 blocks in one turn → lookback window overrun, silent miss
EXAM TRAPS
role:"system" message or use tool
search instead.PRACTICE
Q1 · In a multi-turn loop, where does the breakpoint go, and does it move?
On the last block of the newest settled turn. It moves forward each turn — the prior turn's suffix becomes the next turn's cached prefix, so the growing history reads from cache while only the new tail is written.
Q2 · Why does switching models mid-loop kill caching?
Caches are model-scoped — the entry written under one model can't be read by another. The new model starts cold and pays a full write. Keep one model per loop; delegate cheap side-tasks to a subagent instead.
Q3 · A tool-heavy turn adds 25 content blocks and the next request misses. Why?
The breakpoint's 20-block lookback window was overrun — it couldn't reach the prior cache entry 25 blocks back. Add an intermediate breakpoint (~every 15 blocks) so each stays within 20 of the previous cached block.