WHERE Who stores the (prompt) cache?
Anthropic's servers — not your machine. When you mark a prefix with
cache_control, Anthropic stores the model's processed internal state for those tokens (the
attention KV cache from prefill) on their infrastructure. Your app stores nothing — it just re-sends the
same prefix bytes plus the marker, and Anthropic recognises the matching prefix and reuses its stored compute.
- Ephemeral — 5-minute TTL by default, or 1-hour. Each hit refreshes the TTL.
- Scoped to your organisation/workspace and the model — never shared across accounts.
- You don't manage, evict, or persist it. You only enable it by structuring the prompt.
cache_control cache is Anthropic-side and ephemeral. It makes re-sending
a prefix cheaper & faster — it does not shrink your context window.HOW What step gets skipped
A transformer processes a prompt in a step called prefill: it computes the attention key/value (KV) states for every input token before it can generate. For a long prompt, that's the expensive part. Caching stores the KV states for the cached prefix; on a hit, that prefill is skipped.
(compute KV for every token)"]:::bad --> A2["Generate"] end subgraph YES["Cache HIT"] B1["Load stored KV for the
cached prefix — prefill SKIPPED"]:::ok --> B2["Prefill only the NEW tokens"]:::warn --> B3["Generate"] end classDef ok fill:#dcfce7,stroke:#15803d,color:#14532d; classDef warn fill:#fef3c7,stroke:#b45309,color:#7c2d12; classDef bad fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d;
So on a hit you pay ~0.1× the input rate for the cached tokens (a "cache read") instead of full price, and
time-to-first-token drops because that prefill is skipped. A cache write (first time) costs ~1.25× (5-min TTL)
or ~2× (1-hour). Verify a hit with usage.cache_read_input_tokens — 0 across identical requests means a
silent invalidator is at work.
LEVELS How deep the cached prefix goes
Caching is prefix-based, in render order tools → system → messages, with up to 4
breakpoints. The deeper the stable prefix, the more prefill is skipped on a repeat:
| Cache level | On a repeat, prefill is skipped for… | Still processed fresh |
|---|---|---|
| None | nothing — 100% re-prefilled | everything |
| Tools only | the tool definitions | system + messages |
| Tools + system | tool defs + system prompt | all messages |
| …into messages (conversation prefix / a big document / few-shot) | tools + system + the cached message prefix (e.g. the whole prior conversation) | just the newest turn |
Invalidation hierarchy (why order matters)
| You change… | Invalidates |
|---|---|
| Tool definitions (add/remove/reorder) or the model | everything below |
| System prompt content | system + messages cache |
| Message content | only the messages cache |
datetime/uuid or an unsorted json.dumps into the system prompt — put volatile
content after the last breakpoint. Below the minimum prefix (~1024 tokens on Sonnet, ~4096 on Opus 4.8) it
silently won't cache at all.TWO KINDS "Prompt caching" vs an app-side cache you build
The word "cache" covers two different things — keep them apart:
| Prompt caching (Anthropic's) | App-side cache (yours) | |
|---|---|---|
| Who stores it | Anthropic | You (Redis / your DB / memory) |
| What you do | Enable it — byte-stable prefix + cache_control | Build & store it yourself |
| What it saves | Skips re-prefill of a repeated prefix (cost + latency) | Skips work entirely (even the LLM call) |
| Examples | System prompt, tool defs, a document you Q&A over, few-shot | Response/semantic cache · introspected schema cache · embedding cache |
Use Anthropic prompt caching when you re-send a large stable prefix across ≥2 requests within the TTL (chatbots, doc Q&A, agent loops). Build your own app-side cache when you want to skip calling the model at all (cache the final answer for identical/similar questions) or avoid recomputing an expensive input (a schema you introspected once, embeddings you already made).
MEMORY Where is "memory" stored — DB, vector DB, or graph?
It depends on the access pattern — memory isn't one storage. And note two different things called memory:
- Claude's memory tool (
memory_20250818) is a file-based interface — Claude reads/writes a/memoriesdirectory. It is client-executed: you implement the backend, so it lives on your infrastructure (local files, object storage, a DB blob — your choice). Anthropic doesn't store it. (Managed Agents has a separate hosted "memory store".) - Working (short-term) memory = just the
messagesyou re-send — no external store; keep it in the session (in-process / Redis / a DB row).
For long-term, cross-session memory, pick the store by how you'll retrieve:
| You need to recall… | Use | Why |
|---|---|---|
| Structured facts / preferences by key ("default warehouse = analytics_wh", "prefers ₹") | Relational / document / KV DB (Postgres, DynamoDB, Redis) | Exact lookup by user/key — simplest & most common |
| "Something similar to X" across many past notes/docs | Vector database (pgvector, Pinecone, Chroma) | Semantic recall by similarity — memory-as-RAG |
| Reasoning over relationships between entities (who-relates-to-whom, multi-hop) | Knowledge graph (Neo4j) | When the connections are the value; heavier |
EXAM ONE-LINERS
• Deeper cache level → more prefill skipped, but the prefix must be byte-stable; render order
tools → system → messages.• The memory tool is client-executed (you own the storage); pick DB vs vector vs graph by your retrieval pattern, not by default.
examples/context_bloat_techniques.py.