D5 · Context & Reliability › Cluster 8 · Observability & Production Ops

8.3 · Monitoring Reliability & SLOs

Error rate · latency · saturation

CONCEPT Pick SLIs, set SLOs, alert on them

An SLI is a measured signal; an SLO is the target you hold it to; an alert fires when the SLI breaches the SLO. For an LLM integration the signals map cleanly onto the classic reliability triad — errors, latency, saturation — plus a cost-efficiency signal unique to caching.

SLIWhat it measuresTypical SLO
Error rate by status code% of calls failing, split by 4xx vs 5xx/429e.g. <0.5% 5xx over 5 min
Latency p50 / p95 / p99Tail response time, not just the averagee.g. p95 < 8s
Token throughput vs rate limitsITPM / OTPM consumed against your ceilinge.g. <80% of OTPM
Cache hit-rateReads ÷ total prompt tokens (8.1)e.g. >70% on stable prefixes
⚠️ Split error rate by status code. A 400 is your bug (bad request, non-retryable); a 429 means you hit a rate limit; a 500/529 is Anthropic-side and retryable. Alerting on a single "error %" hides which of three very different problems you have.

CONCEPT Saturation = rate limits and the headers that warn you

The LLM-specific saturation signal is rate-limit headroom. Limits are enforced as requests-per-minute (RPM), input-tokens-per-minute (ITPM), and output-tokens-per-minute (OTPM). A 429 tells you you've hit one; the anthropic-ratelimit-* response headers tell you how close you are before you hit it, and a 429 carries a retry-after header saying how long to wait. Watch the remaining-quota headers so you alert on approaching saturation rather than on the outage itself.

🔑 Measure the tail (p95/p99), not the mean. LLM latency is highly variable — a healthy average can hide a p99 that's timing users out. Tail latency is where reliability actually lives.

DIAGRAM SLI → SLO → alert

flowchart TD LOG["Per-call logs (8.2):
status · latency · tokens · cache"] --> AGG["Aggregate into SLIs"] AGG --> E["Error rate
by status code"] AGG --> L["Latency
p50/p95/p99"] AGG --> S["Saturation
ITPM/OTPM vs limit"] AGG --> H["Cache hit-rate"] E --> B{"Breach SLO?"} L --> B S --> B H --> B B -->|"yes"| AL["🔴 Alert + page
429→back off · 5xx→retry · 400→fix"] B -->|"no"| OK["🟢 Within budget"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class OK ok; class AGG gate; class AL bad;

CODE Emit SLIs and watch rate-limit headroom

Record status, latency, and tokens per call; read the remaining-quota headers so a dashboard can alert before you saturate. On a 429, honor retry-after.

# Use the raw-response accessor to see the rate-limit headers.
raw = client.messages.with_raw_response.create(model="claude-opus-4-8",
          max_tokens=1024, messages=msgs)
h = raw.headers
emit("llm.otpm_remaining", int(h.get("anthropic-ratelimit-output-tokens-remaining", 0)))
emit("llm.rpm_remaining",   int(h.get("anthropic-ratelimit-requests-remaining", 0)))

try:
    msg = raw.parse()
    emit("llm.status", 200); emit("llm.output_tokens", msg.usage.output_tokens)
except anthropic.RateLimitError as e:
    emit("llm.status", 429)                       # saturation SLI
    wait = int(e.response.headers.get("retry-after", 60))
    sleep(wait)                                  # back off, then retry (429 is retryable)

TRADE-OFFS Alert on 5xx only vs. alert on every status

✅ Per-status-code SLIs + tail latency

  • 429 → back off; 5xx → retry; 4xx → fix your code
  • Rate-limit headers warn before saturation
  • p99 catches the timeouts a mean hides

⚠️ One blended "error %" + average latency

  • Simple single dashboard number
  • But: can't tell a client bug from an overload
  • Average latency masks a failing tail

EXAM TRAPS

Trap: Monitoring average latency only. → LLM latency has a long tail. Track p50/p95/p99; the p99 is what times users out.
Trap: Blending all errors into one rate. → Split by status code: 400 (your bug, non-retryable), 429 (rate limit — back off), 500/529 (Anthropic-side — retryable). They demand different responses.
Trap: Only reacting to 429s after they happen. → The anthropic-ratelimit-* headers report remaining RPM/ITPM/OTPM every call — alert on approaching the limit, not on the resulting outage.
Trap: Ignoring the retry-after header on a 429. → It tells you exactly how long to wait. Backing off blindly (or not at all) either wastes capacity or hammers a limited endpoint.

PRACTICE

Q1 · Why split error rate by status code instead of one blended figure?

Because the fix differs: 400 is a non-retryable client bug, 429 means back off (honor retry-after), and 500/529 are retryable Anthropic-side errors. A single "error %" tells you something's wrong but not what to do.

Q2 · Which latency statistic should the SLO target, and why not the mean?

Target the tail — p95 and p99. LLM response time is highly variable, so a healthy average can hide a p99 that's blowing past user timeouts. Reliability lives in the tail.

Q3 · Which signals tell you you're approaching rate-limit saturation?

The anthropic-ratelimit-* remaining headers (RPM / ITPM / OTPM) on every response, tracked as a saturation SLI. Alert when remaining quota drops (e.g. below 20%) so you throttle before the 429 outage rather than after.