D5 · Context & Reliability › Cluster 2 · Prompt Caching

2.3 · What's Cacheable & the Prefix Rule

Static first, volatile last

CONCEPT The prefix rule — one byte breaks the cache

Prompt caching is a prefix match. The cache key is derived from the exact bytes of the rendered prompt up to each breakpoint. Everything before a breakpoint must be byte-for-byte identical across requests to score a hit. A single differing byte anywhere in the prefix — a reordered JSON key, a timestamp, a swapped tool — invalidates the cache for that breakpoint and every breakpoint after it. This is the one invariant everything else follows from.

⚠️ Any change anywhere in the prefix invalidates everything downstream. Caching rewards discipline: keep the prefix frozen and deterministic.

CONCEPT Render order — tools → system → messages

The prompt is assembled and hashed in a fixed order: tools → system → messages. That ordering dictates the canonical caching strategy: put the most static content first, the most volatile last, and drop your breakpoint at the boundary between them.

StabilityBelongs…Examples
Never changesEarly, before any breakpointFrozen system prompt, deterministic tool list
Per-sessionAfter the global prefixUser profile, retrieved documents
Per-turn / per-requestAfter the last breakpointThe new question, fresh tool results
Per-request noiseEliminate or move to the very endTimestamps, UUIDs, request IDs
Silent cache killers hiding in the prefix: datetime.now() in the system prompt, uuid4() in early content, json.dumps(d) without sort_keys=True, interpolating a session ID into the system text, or a tool set that varies per user.

DIAGRAM Static prefix cached, volatile suffix fresh

flowchart TD subgraph PREFIX["Cacheable prefix — byte-stable"] T["tools (deterministic order)"] --> S["system prompt (frozen)"] S --> D["retrieved docs / few-shot"] end D --> BP["🟢 cache_control breakpoint"] BP --> Q["volatile: today's question,
timestamps, per-request IDs"] Q --> R["🔴 processed fresh every call"] BAD["datetime.now() in system prompt"] -.->|"invalidates
whole prefix"| S classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; classDef neutral fill:#e7f1ff,stroke:#4a90d9,color:#1a4971; class BP ok; class R bad; class BAD bad; class T neutral; class S neutral; class D neutral; class Q neutral;

CODE A shared prefix with a varying suffix

Put the breakpoint at the end of the shared portion — never at the very end of the whole prompt, or every request writes a distinct entry and nothing is ever read.

resp = client.messages.create(
    model="claude-opus-4-8", max_tokens=1024,
    system=[{"type": "text", "text": FROZEN_SYSTEM,      # no now()/uuid inside!
             "cache_control": {"type": "ephemeral"}}],
    messages=[{"role": "user", "content": [
        {"type": "text", "text": shared_context,        # same across requests
         "cache_control": {"type": "ephemeral"}},     # breakpoint ends the shared part
        {"type": "text", "text": varying_question},     # differs every call → no marker
    ]}],
)

# Serialize tools deterministically so their bytes never shift:
tools = sorted(tool_defs, key=lambda t: t["name"])

TRADE-OFFS Freezing the prefix costs flexibility

✅ Static-first discipline

  • Reliable, repeated cache hits
  • Cheap, fast large shared context
  • Forces deterministic prompt assembly (a good habit)

⚠️ The cost of it

  • Can't sprinkle timestamps / per-user data into the system prompt
  • Changing tools or reordering them mid-run wipes the cache
  • Dynamic context must move after the last breakpoint

EXAM TRAPS

Trap: "I can cache the volatile part too." → Only a stable prefix caches. Anything that changes per request must sit after the last breakpoint, or you write a new entry every call and never read one.
Trap: Putting the current date/time or a UUID in the system prompt. → It changes the prefix bytes every request → zero cache hits. Inject dynamic context later in messages.
Trap: Assuming JSON order doesn't matter. → json.dumps without sort_keys=True (or iterating a set) produces non-deterministic bytes → prefix mismatch. Serialize deterministically.
Trap: "System comes before tools." → Render order is tools → system → messages. A change to the tool list invalidates the system and message caches too, because tools sit first.

PRACTICE

Q1 · State the prefix rule in one sentence.

Everything before a cache breakpoint must be byte-for-byte identical across requests to hit the cache; any change in the prefix invalidates that breakpoint and all breakpoints after it.

Q2 · In what order are tools, system, and messages hashed, and why does it matter?

tools → system → messages. Static content should go first and volatile content last, so a breakpoint at the boundary reuses the frozen prefix while per-request data (after it) is processed fresh. Because tools render first, changing them invalidates the system and message caches too.

Q3 · Why does datetime.now() in the system prompt destroy caching?

It changes the prefix bytes on every request, so the cache key never matches — you pay the write premium each time and get no reads. Keep the system prompt frozen and inject the timestamp later in a message.