CONCEPT The prefix rule — one byte breaks the cache
Prompt caching is a prefix match. The cache key is derived from the exact bytes of the rendered prompt up to each breakpoint. Everything before a breakpoint must be byte-for-byte identical across requests to score a hit. A single differing byte anywhere in the prefix — a reordered JSON key, a timestamp, a swapped tool — invalidates the cache for that breakpoint and every breakpoint after it. This is the one invariant everything else follows from.
CONCEPT Render order — tools → system → messages
The prompt is assembled and hashed in a fixed order: tools → system → messages. That ordering dictates the canonical caching strategy: put the most static content first, the most volatile last, and drop your breakpoint at the boundary between them.
- tools — the tool catalog; render at position 0, usually the most stable.
- system — the system prompt; a breakpoint on its last block also caches the tools.
- messages — conversation turns; the newest, per-request content lives here, after the last breakpoint.
| Stability | Belongs… | Examples |
|---|---|---|
| Never changes | Early, before any breakpoint | Frozen system prompt, deterministic tool list |
| Per-session | After the global prefix | User profile, retrieved documents |
| Per-turn / per-request | After the last breakpoint | The new question, fresh tool results |
| Per-request noise | Eliminate or move to the very end | Timestamps, UUIDs, request IDs |
datetime.now() in the system prompt,
uuid4() in early content, json.dumps(d) without sort_keys=True,
interpolating a session ID into the system text, or a tool set that varies per user.DIAGRAM Static prefix cached, volatile suffix fresh
timestamps, per-request IDs"] Q --> R["🔴 processed fresh every call"] BAD["datetime.now() in system prompt"] -.->|"invalidates
whole prefix"| S classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; classDef neutral fill:#e7f1ff,stroke:#4a90d9,color:#1a4971; class BP ok; class R bad; class BAD bad; class T neutral; class S neutral; class D neutral; class Q neutral;
CODE A shared prefix with a varying suffix
Put the breakpoint at the end of the shared portion — never at the very end of the whole prompt, or every request writes a distinct entry and nothing is ever read.
resp = client.messages.create( model="claude-opus-4-8", max_tokens=1024, system=[{"type": "text", "text": FROZEN_SYSTEM, # no now()/uuid inside! "cache_control": {"type": "ephemeral"}}], messages=[{"role": "user", "content": [ {"type": "text", "text": shared_context, # same across requests "cache_control": {"type": "ephemeral"}}, # breakpoint ends the shared part {"type": "text", "text": varying_question}, # differs every call → no marker ]}], ) # Serialize tools deterministically so their bytes never shift: tools = sorted(tool_defs, key=lambda t: t["name"])
TRADE-OFFS Freezing the prefix costs flexibility
✅ Static-first discipline
- Reliable, repeated cache hits
- Cheap, fast large shared context
- Forces deterministic prompt assembly (a good habit)
⚠️ The cost of it
- Can't sprinkle timestamps / per-user data into the system prompt
- Changing tools or reordering them mid-run wipes the cache
- Dynamic context must move after the last breakpoint
EXAM TRAPS
messages.json.dumps without
sort_keys=True (or iterating a set) produces non-deterministic bytes → prefix
mismatch. Serialize deterministically.PRACTICE
Q1 · State the prefix rule in one sentence.
Everything before a cache breakpoint must be byte-for-byte identical across requests to hit the cache; any change in the prefix invalidates that breakpoint and all breakpoints after it.
Q2 · In what order are tools, system, and messages hashed, and why does it matter?
tools → system → messages. Static content should go first and volatile content last, so a breakpoint at the boundary reuses the frozen prefix while per-request data (after it) is processed fresh. Because tools render first, changing them invalidates the system and message caches too.
Q3 · Why does datetime.now() in the system prompt destroy caching?
It changes the prefix bytes on every request, so the cache key never matches — you pay the write premium each time and get no reads. Keep the system prompt frozen and inject the timestamp later in a message.