D5 · Context & Reliability › Cluster 2 · Prompt Caching

2.5 · Caching in Multi-Turn / Agent Loops

The moving breakpoint

CONCEPT Let the conversation grow past a moving breakpoint

In an agent loop or chat, the conversation history keeps growing — and each turn's history is the next turn's stable prefix. The caching pattern: place a breakpoint after the stable prefix (system + tools + settled history) and let new turns accumulate after it. On the next request, that now-larger history is a byte-identical prefix, so it reads from cache; only the newest turn is written. As history settles, you move the breakpoint forward to keep capturing the growing stable region.

🔑 The breakpoint isn't fixed to one spot — it advances as stable content accumulates. Each turn: the prior turn's suffix becomes this turn's cached prefix.

CONCEPT What invalidates it — and the 20-block window

Because caching is a prefix match (2.3), anything that alters the front of the prompt mid-run wipes the cache and forces a cold re-write:

Agent-safe moves: keep the model fixed; append a {"role":"system"} message instead of editing the top-level system prompt (preserves the cached prefix on supporting models); use tool search to append tool schemas rather than swapping the tool list.
Cache-wiping moves: swapping the model, editing the system prompt in place, reordering tools, or rewriting old turns. Each rebuilds the prefix from that point and pays a fresh write.

DIAGRAM The breakpoint advances each turn

flowchart TD subgraph R1["Request 1"] A1["system + tools"] --> B1["User(1)"] B1 --> BP1["🟢 breakpoint"] end subgraph R2["Request 2 — prior prefix now reads from cache"] A2["system + tools + User(1) + Asst(1)"] --> B2["User(2)"] B2 --> BP2["🟢 breakpoint moved forward"] end subgraph R3["Request 3"] A3["... + User(2) + Asst(2)"] --> B3["User(3)"] B3 --> BP3["🟢 breakpoint moved forward"] end BP1 -.->|"grows"| BP2 -.->|"grows"| BP3 MODELSWAP["switch model / edit system / reorder tools"] -.->|"invalidates
whole prefix"| A2 classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; classDef neutral fill:#e7f1ff,stroke:#4a90d9,color:#1a4971; class BP1 ok; class BP2 ok; class BP3 ok; class MODELSWAP bad; class A1 neutral; class A2 neutral; class A3 neutral;

CODE A growing conversation with a moving breakpoint

Append each turn, and put the breakpoint on the last block of the newest settled turn. The ever-larger history reads from cache; only the new tail is written.

messages = []

def turn(user_text):
    messages.append({"role": "user", "content": user_text})
    # Move the breakpoint to the last block of the newest turn
    messages[-1]["content"] = [
        {"type": "text", "text": user_text,
         "cache_control": {"type": "ephemeral"}},   # advances each turn
    ]
    resp = client.messages.create(
        model="claude-opus-4-8",           # SAME model every turn — never swap mid-loop
        max_tokens=1024,
        system=[{"type": "text", "text": FROZEN_SYSTEM,  # never edited in place
                 "cache_control": {"type": "ephemeral"}}],
        messages=messages,
    )
    messages.append({"role": "assistant", "content": resp.content})
    return resp

# Turn 2+ reads the whole prior history from cache (~0.1×);
# only the newest turn is written.

TRADE-OFFS Fixed model & prompt vs. mid-loop flexibility

✅ One model, frozen prefix

  • History reads from cache every turn — huge savings in long loops
  • Latency stays low as context grows
  • Breakpoint simply advances with the conversation

⚠️ Changing things mid-loop

  • Switching models → cold cache (model-scoped)
  • Editing system / reordering tools → full prefix rebuild
  • >20 blocks in one turn → lookback window overrun, silent miss

EXAM TRAPS

Trap: "The breakpoint stays put at the system prompt." → In a loop it moves forward onto the newest settled turn, so the growing history keeps reading from cache.
Trap: Switching models mid-conversation to "save money." → Caches are model-scoped; the new model starts cold and pays a full write. Keep one model per loop (use a subagent for cheap side-tasks).
Trap: Editing the system prompt or reordering tools between turns. → Both change the prefix bytes and invalidate the whole cache. Append a role:"system" message or use tool search instead.
Trap: Forgetting the 20-block lookback. → A single turn adding >20 blocks (many tool_use/tool_result pairs) can overrun the window and silently miss. Add an intermediate breakpoint every ~15 blocks.

PRACTICE

Q1 · In a multi-turn loop, where does the breakpoint go, and does it move?

On the last block of the newest settled turn. It moves forward each turn — the prior turn's suffix becomes the next turn's cached prefix, so the growing history reads from cache while only the new tail is written.

Q2 · Why does switching models mid-loop kill caching?

Caches are model-scoped — the entry written under one model can't be read by another. The new model starts cold and pays a full write. Keep one model per loop; delegate cheap side-tasks to a subagent instead.

Q3 · A tool-heavy turn adds 25 content blocks and the next request misses. Why?

The breakpoint's 20-block lookback window was overrun — it couldn't reach the prior cache entry 25 blocks back. Add an intermediate breakpoint (~every 15 blocks) so each stays within 20 of the previous cached block.