Cache it, don't cut it
D2 · Models, Prompting & ContextGuide lines 2.5, 4.5
Many assistants send the same long block on every request: a system prompt, a policy, a rulebook, tool definitions. Only the user's short message changes. Paying full price for that same block thousands of times is the waste, and the tempting fixes are to cut it (send less) or downsize (use a smaller model). Both risk wrong answers, because the content was there for a reason.
Prompt caching attacks the waste directly. If the start of the prompt (the prefix) is identical on every request, Claude reads it from a cache: much cheaper, and the first words of the reply arrive sooner. The rule is ordering: static content first, anything that changes per request after it. One changing detail near the top, such as a date, a name or a different tool order, breaks the cache from that point on.
Caching is not always the answer. If nothing repeats, there is nothing to reuse. If the reference is far too big to send at all, retrieve only the relevant part. And if part of the block is genuinely never needed, removing it is fine: the trap is cutting content the answers need.
Change one fact
A pension provider's member assistant sends the same 10,000-token system prompt and scheme rulebook on every request, followed by the member's short question. Members wait on screen for each reply. Cost and reply time are both too high, and any part of the rulebook may be needed for a correct answer. What should the team change?
Words in the stem that signal this trap
- “the same long system prompt on every request”
- “followed by a short, varying message”
- “cost and latency are both concerns”
- “caching is on, but the hit rate is near zero”
- “each request carries a different document”
- “parts of it are never used”
Practice
A law firm's contract assistant sends the same 20,000-token clause library first in every request, then the lawyer's question. Prompt caching is on, and the prefix never changes. Lawyers send a question about every fifteen minutes, longer than the default cache lifetime, so most requests miss the cache. What should the architect change?
- Choose the longer cache lifetime, so the prefix stays cached between requests.
- Shorten the clause library by removing the clauses that lawyers cite least often.
- Move the lawyer's question so it comes after the static clause library.
- Switch to a smaller model so that each uncached request costs less.
Show answer
Answer: A. The deciding fact: the prefix is identical and correctly placed, but requests arrive further apart than the cache lasts. A longer cache lifetime keeps it warm between requests.
- A ✅ Choose the longer cache lifetime, so the prefix stays cached between requests.
- B ❌ Works to cut tokens, but it removes clauses some answers need, and the misses would continue.
- C ❌ Works when variable content sits in the prefix; here the order is already right.
- D ❌ Works if the eval holds, but it leaves the real cause, an expiring cache, untouched.
An appliance maker's support assistant sends a cached 14,000-token prefix with every chat. Half of it is manuals for products the company stopped supporting two years ago, and policy says the assistant must not advise on them. Costs are acceptable, but the assistant sometimes quotes the old manuals. What should the architect do?
- Keep the prefix as it is, since caching already makes the old manuals cheap to send.
- Add a system-prompt rule telling the assistant to ignore the unsupported manuals.
- Move the unsupported manuals to a retrieval index the assistant searches when needed.
- Remove the unsupported manuals from the prefix and cache the part that remains.
Show answer
Answer: D. The deciding fact: the old manuals are never needed and must not be used. Here cutting is right, because the removed text is content no answer should draw on.
- A ❌ Works for content that is needed; here cheap content still misleads the assistant.
- B ❌ Works sometimes, but a prompt rule is probabilistic and the manuals stay in front of the model.
- C ❌ Works for large content needed now and then; here the content should never be used at all.
- D ✅ Remove the unsupported manuals from the prefix and cache the part that remains.
A car-rental firm's booking assistant sends the same system prompt, 30 tool definitions and a fare-rules document on every request, then the customer's message. Prompt caching is on, but the hit rate is low. Logs show the tool definitions are added in a different order on each request. What should the architect change?
- Shorten every tool description so that each uncached request costs less.
- Send the tool definitions in one fixed order on every request.
- Turn caching off, since a prefix that keeps changing cannot be cached.
- Load the tools through tool search so fewer definitions are sent.
Show answer
Answer: B. The deciding fact: the content is the same each time, only the order varies, and any change early in the prefix breaks the cache. A fixed order makes the prefix identical.
- A ❌ Works to trim tokens, but descriptions help tool choice, and the cache would still miss.
- B ✅ Send the tool definitions in one fixed order on every request.
- C ❌ Works when the content truly differs per request; here only the order differs.
- D ❌ Works for hundreds of tools with a few used per task; 30 stable tools are cheaper cached.