CONCEPT What to instrument
Every lever in this cluster — model choice, effort, caching, streaming, state — is only tunable if
you can see its effect. Observability for agents has two halves: log tool calls and
trace each run (the structure of what happened), and read the usage
object (the token economics of each call).
usage field | Meaning |
|---|---|
input_tokens | Uncached input processed this call. |
output_tokens | Tokens generated (the pricey side, ~5×). |
cache_creation_input_tokens | Tokens written to cache (~1.25×). |
cache_read_input_tokens | Tokens served from cache (~0.1×) — your cache-hit signal. |
response._request_id on every call. It's what Anthropic
support needs to investigate a specific request — log it alongside the usage record.CONCEPT The metrics that matter
Aggregate the per-call data into run-level metrics. These are the numbers an architect reports on:
| Metric | Why it matters |
|---|---|
| Latency p50 / p95 / p99 | Tail latency (p95/p99), not the average, is what users feel — track all three. |
| Cost per run | Sum token cost across every call in a run — the true price of the agent, not per-call. |
| Error rate | Failed calls / runs — catches regressions, bad tools, and timeouts. |
| Resolution rate | Fraction of runs that actually solved the task — the quality metric that justifies cost. |
DIAGRAM Trace + usage per run
cost/run · latency p50/p95/p99
error rate · resolution rate"] AGG --> OBS([Dashboards / alerts]) classDef tool fill:#d4edda,stroke:#28a745,color:#155724; classDef agg fill:#d1ecf1,stroke:#17a2b8,color:#0c5460; class TC tool; class AGG agg;
CODE Logging usage & a trace record
Wrap each call: pull the usage fields and _request_id, emit one structured
record per call tagged with the run's trace_id, and record each tool call too.
import time, uuid def traced_call(trace_id, messages, tools): t0 = time.perf_counter() resp = client.messages.create( model="claude-opus-4-8", max_tokens=4096, tools=tools, messages=messages, ) u = resp.usage log.info("llm_call", extra={ "trace_id": trace_id, "request_id": resp._request_id, # for Anthropic support "input_tokens": u.input_tokens, "output_tokens": u.output_tokens, "cache_read": u.cache_read_input_tokens, # cache-hit signal (7.3) "cache_write": u.cache_creation_input_tokens, "stop_reason": resp.stop_reason, "latency_ms": round((time.perf_counter() - t0) * 1000), }) return resp def traced_tool(trace_id, name, args, fn): t0 = time.perf_counter() ok = True try: return fn(**args) except Exception: ok = False; raise finally: log.info("tool_call", extra={ "trace_id": trace_id, "tool": name, "ok": ok, "latency_ms": round((time.perf_counter() - t0) * 1000), }) trace_id = str(uuid.uuid4()) # one id ties the whole run together
EXAM TRAPS
_request_id. → Without it, Anthropic support
can't investigate a specific failing request.PRACTICE
Q1 · Your dashboard shows flat average latency but users complain of slowness. What are you missing?
The tail — track p95/p99, not just p50/mean. A small fraction of very slow runs (retries, big generations) drives the felt experience while the average stays calm.
Q2 · You need to prove a caching change actually helped. Which usage field, and where does it come from?
cache_read_input_tokens from the usage object (7.3). Log it per
call and confirm it's > 0 across identical-prefix requests; a run-level rise means the cache is being
reused.
Q3 · What single identifier lets you reconstruct an entire agent run from your logs, and what ties one call to Anthropic support?
A per-run trace_id you generate stitches all calls and tool invocations together;
the per-call response._request_id is what support uses to look up a specific request.