CCA-F › Domain 5 › Caching & Memory

🗄️ Prompt Caching & Memory — Where It Lives, How It Works

Who stores what · what step is skipped · which memory store to pick

WHERE Who stores the (prompt) cache?

Anthropic's servers — not your machine. When you mark a prefix with cache_control, Anthropic stores the model's processed internal state for those tokens (the attention KV cache from prefill) on their infrastructure. Your app stores nothing — it just re-sends the same prefix bytes plus the marker, and Anthropic recognises the matching prefix and reuses its stored compute.

🔑 The cache_control cache is Anthropic-side and ephemeral. It makes re-sending a prefix cheaper & faster — it does not shrink your context window.

HOW What step gets skipped

A transformer processes a prompt in a step called prefill: it computes the attention key/value (KV) states for every input token before it can generate. For a long prompt, that's the expensive part. Caching stores the KV states for the cached prefix; on a hit, that prefill is skipped.

flowchart LR subgraph NO["No cache — every request"] A1["PREFILL all input tokens
(compute KV for every token)"]:::bad --> A2["Generate"] end subgraph YES["Cache HIT"] B1["Load stored KV for the
cached prefix — prefill SKIPPED"]:::ok --> B2["Prefill only the NEW tokens"]:::warn --> B3["Generate"] end classDef ok fill:#dcfce7,stroke:#15803d,color:#14532d; classDef warn fill:#fef3c7,stroke:#b45309,color:#7c2d12; classDef bad fill:#fee2e2,stroke:#b91c1c,color:#7f1d1d;

So on a hit you pay ~0.1× the input rate for the cached tokens (a "cache read") instead of full price, and time-to-first-token drops because that prefill is skipped. A cache write (first time) costs ~1.25× (5-min TTL) or ~2× (1-hour). Verify a hit with usage.cache_read_input_tokens — 0 across identical requests means a silent invalidator is at work.

LEVELS How deep the cached prefix goes

Caching is prefix-based, in render order tools → system → messages, with up to 4 breakpoints. The deeper the stable prefix, the more prefill is skipped on a repeat:

Cache levelOn a repeat, prefill is skipped for…Still processed fresh
Nonenothing — 100% re-prefilledeverything
Tools onlythe tool definitionssystem + messages
Tools + systemtool defs + system promptall messages
…into messages (conversation prefix / a big document / few-shot)tools + system + the cached message prefix (e.g. the whole prior conversation)just the newest turn

Invalidation hierarchy (why order matters)

You change…Invalidates
Tool definitions (add/remove/reorder) or the modeleverything below
System prompt contentsystem + messages cache
Message contentonly the messages cache
Byte-stable prefix rule: any change in the prefix invalidates everything after it. Never interpolate a datetime/uuid or an unsorted json.dumps into the system prompt — put volatile content after the last breakpoint. Below the minimum prefix (~1024 tokens on Sonnet, ~4096 on Opus 4.8) it silently won't cache at all.

TWO KINDS "Prompt caching" vs an app-side cache you build

The word "cache" covers two different things — keep them apart:

Prompt caching (Anthropic's)App-side cache (yours)
Who stores itAnthropicYou (Redis / your DB / memory)
What you doEnable it — byte-stable prefix + cache_controlBuild & store it yourself
What it savesSkips re-prefill of a repeated prefix (cost + latency)Skips work entirely (even the LLM call)
ExamplesSystem prompt, tool defs, a document you Q&A over, few-shotResponse/semantic cache · introspected schema cache · embedding cache

Use Anthropic prompt caching when you re-send a large stable prefix across ≥2 requests within the TTL (chatbots, doc Q&A, agent loops). Build your own app-side cache when you want to skip calling the model at all (cache the final answer for identical/similar questions) or avoid recomputing an expensive input (a schema you introspected once, embeddings you already made).

In the Databricks NL→SQL agent, both appear: the SQL-guidelines system prompt uses Anthropic prompt caching, while the introspected Unity Catalog schema + sample values are an app-side cache you refresh only on DDL change.

MEMORY Where is "memory" stored — DB, vector DB, or graph?

It depends on the access pattern — memory isn't one storage. And note two different things called memory:

For long-term, cross-session memory, pick the store by how you'll retrieve:

You need to recall…UseWhy
Structured facts / preferences by key ("default warehouse = analytics_wh", "prefers ₹")Relational / document / KV DB (Postgres, DynamoDB, Redis)Exact lookup by user/key — simplest & most common
"Something similar to X" across many past notes/docsVector database (pgvector, Pinecone, Chroma)Semantic recall by similarity — memory-as-RAG
Reasoning over relationships between entities (who-relates-to-whom, multi-hop)Knowledge graph (Neo4j)When the connections are the value; heavier
Rule of thumb: start with the simplest that fits — usually a key-value / relational store. Add a vector store only when you genuinely need semantic recall, and a graph only when relationships / multi-hop reasoning are the point. Real systems often combine a DB for facts + a running summary + a vector store for episodic notes.
Never store secrets/PII in memory — memory replays into every future session that loads it.

EXAM ONE-LINERS

• Prompt caching is Anthropic-side, ephemeral, prefix-based; a hit skips prefill of the cached tokens (cost + latency) — it does not shrink the context window.
• Deeper cache level → more prefill skipped, but the prefix must be byte-stable; render order tools → system → messages.
• The memory tool is client-executed (you own the storage); pick DB vs vector vs graph by your retrieval pattern, not by default.