D5 · Context & Reliability › Cluster 7 · Cost & Performance Optimization

7.5 · Caching + Batch Economics

Stack the discounts

CONCEPT The caching price table

Prompt caching is a prefix match: a stable prefix (tools → system → early messages) is written to cache once, then reused. The economics are a set of multipliers on the model's base input rate:

OperationMultiplier on base inputMeaning
Cache write, 5-min TTL1.25×Premium to store the prefix (default TTL)
Cache write, 1-hour TTL2×Premium for the longer-lived entry
Cache read (hit)0.1×~90% cheaper than a fresh read
Uncached input1× (full)Baseline
🔑 Break-even: at the 5-min write premium (1.25×) plus one read (0.1×) = 1.35× vs. 2× for two uncached reads, caching pays off from the second request. The 1-hour TTL (2× write) needs at least three reads to win. Minimum cacheable prefix is ~1,024 tokens on Opus 4.8 / Sonnet; shorter prefixes silently don't cache.

CONCEPT Caching and batching stack — multiplicatively

These are independent discounts, and the multipliers combine. The Batch API's 50% cut (7.2) applies to all token usage, including cached reads. So a cached prefix read inside a batch is billed at 0.1× × 0.5 = 0.05× of base input — a 95% discount on that portion. This is the cheapest way to run a large eval that shares one big prompt: cache the shared context, then batch every variant against it.

Scenario (Opus 4.8, $5/MTok input)Effective rate on the shared prefix
Live, uncached1× → $5.00 / MTok
Live, cache read0.1× → $0.50 / MTok
Batch, uncached0.5× → $2.50 / MTok
Batch + cache read0.05× → $0.25 / MTok
⚠️ Caveat inside a batch: ephemeral cache entries can expire before the read if the batch runs long, and max_tokens: 0 cache pre-warming is not supported in a batch. Structure batches so cache-writing and cache-reading requests are submitted together, and lean on the 1-hour TTL when the batch may take a while.

CONCEPT When NOT to batch — and when NOT to cache

Don't batch a latency-sensitive path. The 50% discount costs you an async window of up to 24 hours — fine for evals, fatal for a chat reply, a voice turn, or any user waiting on a result. Interactive paths stay on live requests; make those fast with 7.3's levers (stream, right-size, cache the prefix). Don't cache a prefix that changes every call — you'd pay the 1.25×/2× write premium and never collect a read. And don't cache a prefix under the ~1,024-token minimum; it won't cache at all.

Rule of thumb: Interactive? Live + stream + cache the stable prefix. Non-urgent & bulk? Batch it — and if the batch shares a big prefix, cache that too for the stacked 95% discount.

DIAGRAM Route by urgency, stack the discounts

flowchart TD A["Workload with a large shared prefix"] --> Q{"Needs a fast reply?"} Q -->|"yes: interactive"| LIVE["🔵 Live request"] LIVE --> LC["Cache the prefix
reads 0.1× (~90% off)"] LC --> STREAM["🟢 Stream for low TTFT"] Q -->|"no: bulk / evals"| BATCH["🟢 Batch API
0.5× (50% off)"] BATCH --> BC["Cache the shared prefix
0.1× × 0.5 = 0.05×"] BC --> BEST["🟢 ~95% off the prefix"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef mid fill:#cce5ff,stroke:#3399ff,color:#004085; class LC ok; class STREAM ok; class BATCH ok; class BC ok; class BEST ok; class LIVE mid;

CODE Cache the shared prefix, then batch every variant

One big cached system prompt, reused across a whole batch of questions — 50% batch discount on top of the ~90% cache-read discount.

from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request

# The shared prefix — cached once, read by every request in the batch.
shared_system = [{
    "type": "text",
    "text": large_reference_doc,        # same across all requests
    "cache_control": {"type": "ephemeral", "ttl": "1h"},  # longer TTL for a slow batch
}]

batch = client.messages.batches.create(requests=[
    Request(
        custom_id=f"q-{i}",
        params=MessageCreateParamsNonStreaming(
            model="claude-opus-4-8",
            max_tokens=512,
            system=shared_system,               # cache read → 0.1×, then ×0.5 batch = 0.05×
            messages=[{"role": "user", "content": q}],
        ),
    )
    for i, q in enumerate(questions)
])
# Prefix billed at ~0.05× base; per-question tokens at 0.5× (batch).

TRADE-OFFS Stacking discounts vs. keeping it live

✅ Batch + cache (non-urgent)

  • ~95% off the shared prefix (0.1× × 0.5)
  • 50% off all other tokens
  • Ideal for evals over one big context

⚠️ Live + cache (interactive)

  • ~90% off the prefix, full price otherwise
  • But immediate — no 24h wait
  • Pair with streaming for low TTFT
🔑 The discounts are orthogonal: model tier (7.1), cache reads, batch, and fewer tokens (7.4) all multiply. Pick each dial by the workload — urgency decides batch, prefix stability decides cache.

EXAM TRAPS

Trap: "Batch and caching are alternatives — pick one." → They stack. A cached read inside a batch is billed at 0.1× × 0.5 = 0.05× base — both discounts apply at once.
Trap: "Caching is always free money." → Writes cost 1.25× (5-min) or 2× (1-hour). It only pays off if the prefix is stable and re-read — from the 2nd read (5-min) or 3rd (1-hour).
Trap: Batching an interactive path to save 50%. → Batch adds an async window up to 24 hours. Never put a user-facing, latency-sensitive request in a batch.
Trap: Assuming a cache always survives a batch. → Ephemeral entries can expire before a long batch reads them, and max_tokens: 0 pre-warming isn't supported in a batch. Use the 1-hour TTL and co-submit writer + reader requests.

PRACTICE

Q1 · What is the effective rate on a cached prefix read inside a batch, as a multiple of base input?

0.05× — the cache read multiplier (0.1×) times the batch discount (0.5×). About 95% off the base input rate on that prefix.

Q2 · When should you NOT use the Batch API?

On any latency-sensitive / interactive path — chat, voice, "do it now" actions. Batch is asynchronous with a window up to 24 hours; the 50% discount isn't worth making a waiting user sit through it. Keep those live and use streaming/caching to make them fast.

Q3 · A large eval runs 5,000 questions against one shared 30K-token document. How do you minimize cost?

Put the shared document in a cached system prefix (1-hour TTL, since the batch may run a while) and submit all questions in a batch. The prefix is billed at ~0.05× base (cache read × batch), and the rest at 0.5× — the two discounts stack. Right-size the model too (7.1) if the task allows.