D5 · Context & Reliability › Cluster 8 · Observability & Production Ops

8.1 · Usage Metrics (Tokens, Cache, Cost)

Measure every call

CONCEPT Every response carries a usage object

You cannot manage what you don't measure. Every Messages API response returns a usage object — non-streaming and streaming alike — and it is the ground truth for cost and cache health. The four fields you must know:

FieldWhat it countsRelative price
input_tokensUncached prompt tokens processed at full price1× input rate
output_tokensTokens the model generated (incl. thinking)1× output rate
cache_creation_input_tokensTokens written to the cache this request~1.25× (5m TTL) / 2× (1h TTL)
cache_read_input_tokensTokens served from the cache this request~0.1× input rate
⚠️ input_tokens is the uncached remainder only, not the total prompt. The full prompt size = input_tokens + cache_creation_input_tokens + cache_read_input_tokens. If an agent ran for hours but input_tokens shows 4K, the rest was served from cache — sum the three, don't read one field.

CONCEPT Deriving cost and cache hit-rate

The usage object is the raw material for the two numbers finance and SRE actually care about.

Cost is a weighted sum of the four fields against the model's per-token prices — cache reads at ~0.1× and cache writes at the 1.25×/2× premium, so caching only pays off after repeated reads (break-even is two requests on the 5-minute TTL, three on the 1-hour TTL).

Cache hit-rate tells you whether your prefix is actually being reused. Compute it as cache_read / (cache_read + cache_creation + input_tokens) across requests. If cache_read_input_tokens is zero across repeated identical-prefix requests, a silent invalidator is at work — a timestamp or UUID in the system prompt, unsorted JSON, or a varying tool set.

🔑 Log the raw usage object per call and aggregate downstream. Storing derived cost only throws away the detail you need when prices change or you want to attribute spend by tier.

DIAGRAM From usage to the two numbers that matter

flowchart TD R["API response.usage"] --> I["input_tokens
(uncached)"] R --> O["output_tokens"] R --> CW["cache_creation_input_tokens"] R --> CR["cache_read_input_tokens"] I --> COST["💰 Cost = Σ tokens × price
(reads ~0.1×, writes 1.25–2×)"] O --> COST CW --> COST CR --> COST I --> HIT["📊 Hit-rate = read /
(read + create + input)"] CW --> HIT CR --> HIT HIT --> AL{"read == 0
on repeat?"} AL -->|"yes"| BAD["🔴 Silent invalidator
audit the prefix"] AL -->|"no"| OK["🟢 Cache working"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class OK ok; class COST gate; class BAD bad;

CODE Read usage, compute cost + hit-rate

The response object exposes all four fields directly. Streaming works too: the final message from stream.get_final_message() carries the same usage.

# Prices per 1M tokens (Opus 4.8 example): $5 input / $25 output.
# Cache read ~0.1x input; cache write ~1.25x input (5-minute TTL).
IN, OUT, C_READ, C_WRITE = 5.0, 25.0, 0.5, 6.25

resp = client.messages.create(model="claude-opus-4-8", max_tokens=1024,
                              system=SYSTEM, messages=msgs)
u = resp.usage

def cost_usd(u):
    return (u.input_tokens*IN + u.output_tokens*OUT
          + u.cache_creation_input_tokens*C_WRITE
          + u.cache_read_input_tokens*C_READ) / 1_000_000

prompt_total = u.input_tokens + u.cache_creation_input_tokens + u.cache_read_input_tokens
hit_rate = u.cache_read_input_tokens / prompt_total if prompt_total else 0

emit_metric("llm.cost_usd", cost_usd(u))
emit_metric("llm.cache_hit_rate", hit_rate)
emit_metric("llm.output_tokens", u.output_tokens)  # feeds OTPM tracking (8.3)

TRADE-OFFS Log raw usage vs. log derived cost

✅ Log the raw usage object

  • Survives price changes — recompute cost anytime
  • Cache hit-rate and token throughput both derivable
  • Attribute spend by tenant, route, or model later

⚠️ Log only a computed dollar figure

  • Smaller records, one number to chart
  • But: hard-codes today's prices into history
  • Can't reconstruct hit-rate or ITPM/OTPM after the fact

EXAM TRAPS

Trap: Treating input_tokens as the whole prompt. → It's the uncached remainder. Total = input_tokens + cache_creation_input_tokens + cache_read_input_tokens.
Trap: Assuming the field names. → It's cache_creation_input_tokens and cache_read_input_tokens — not cache_write, cached_tokens, or prompt_tokens. Don't invent usage fields.
Trap: "Caching is always cheaper." → A cache write costs 1.25× (5m) or 2× (1h). It only pays off once you get enough reads — break-even is 2 requests on 5m TTL, 3 on 1h.
Trap: "Streaming responses don't report usage." → They do. Usage arrives on message_start (input) and cumulatively on message_delta (output); the final message carries the complete object.

PRACTICE

Q1 · Name the four token fields on the usage object.

input_tokens, output_tokens, cache_creation_input_tokens (tokens written to cache), and cache_read_input_tokens (tokens served from cache). The two cache fields are how you compute cache hit-rate.

Q2 · cache_read_input_tokens is 0 on every repeated request. What's wrong?

A silent cache invalidator is changing the prefix each time — a datetime.now() or UUID in the system prompt, unsorted json.dumps(), or a tool set that varies per request. Diff the rendered prompt bytes between two requests to find it.

Q3 · How do you compute total prompt size from the usage object?

Sum the three input-side fields: input_tokens + cache_creation_input_tokens + cache_read_input_tokens. Reading input_tokens alone undercounts because it excludes everything served from or written to cache.