CONCEPT Pick SLIs, set SLOs, alert on them
An SLI is a measured signal; an SLO is the target you hold it to; an alert fires when the SLI breaches the SLO. For an LLM integration the signals map cleanly onto the classic reliability triad — errors, latency, saturation — plus a cost-efficiency signal unique to caching.
| SLI | What it measures | Typical SLO |
|---|---|---|
| Error rate by status code | % of calls failing, split by 4xx vs 5xx/429 | e.g. <0.5% 5xx over 5 min |
| Latency p50 / p95 / p99 | Tail response time, not just the average | e.g. p95 < 8s |
| Token throughput vs rate limits | ITPM / OTPM consumed against your ceiling | e.g. <80% of OTPM |
| Cache hit-rate | Reads ÷ total prompt tokens (8.1) | e.g. >70% on stable prefixes |
CONCEPT Saturation = rate limits and the headers that warn you
The LLM-specific saturation signal is rate-limit headroom. Limits are enforced as
requests-per-minute (RPM), input-tokens-per-minute (ITPM), and
output-tokens-per-minute (OTPM). A 429 tells you you've hit one; the
anthropic-ratelimit-* response headers tell you how close you are before you hit it, and
a 429 carries a retry-after header saying how long to wait. Watch the remaining-quota headers so
you alert on approaching saturation rather than on the outage itself.
DIAGRAM SLI → SLO → alert
status · latency · tokens · cache"] --> AGG["Aggregate into SLIs"] AGG --> E["Error rate
by status code"] AGG --> L["Latency
p50/p95/p99"] AGG --> S["Saturation
ITPM/OTPM vs limit"] AGG --> H["Cache hit-rate"] E --> B{"Breach SLO?"} L --> B S --> B H --> B B -->|"yes"| AL["🔴 Alert + page
429→back off · 5xx→retry · 400→fix"] B -->|"no"| OK["🟢 Within budget"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class OK ok; class AGG gate; class AL bad;
CODE Emit SLIs and watch rate-limit headroom
Record status, latency, and tokens per call; read the remaining-quota headers so a dashboard can alert
before you saturate. On a 429, honor retry-after.
# Use the raw-response accessor to see the rate-limit headers. raw = client.messages.with_raw_response.create(model="claude-opus-4-8", max_tokens=1024, messages=msgs) h = raw.headers emit("llm.otpm_remaining", int(h.get("anthropic-ratelimit-output-tokens-remaining", 0))) emit("llm.rpm_remaining", int(h.get("anthropic-ratelimit-requests-remaining", 0))) try: msg = raw.parse() emit("llm.status", 200); emit("llm.output_tokens", msg.usage.output_tokens) except anthropic.RateLimitError as e: emit("llm.status", 429) # saturation SLI wait = int(e.response.headers.get("retry-after", 60)) sleep(wait) # back off, then retry (429 is retryable)
TRADE-OFFS Alert on 5xx only vs. alert on every status
✅ Per-status-code SLIs + tail latency
- 429 → back off; 5xx → retry; 4xx → fix your code
- Rate-limit headers warn before saturation
- p99 catches the timeouts a mean hides
⚠️ One blended "error %" + average latency
- Simple single dashboard number
- But: can't tell a client bug from an overload
- Average latency masks a failing tail
EXAM TRAPS
anthropic-ratelimit-* headers report remaining RPM/ITPM/OTPM every call — alert on approaching
the limit, not on the resulting outage.retry-after header on a 429. → It tells you exactly how
long to wait. Backing off blindly (or not at all) either wastes capacity or hammers a limited endpoint.PRACTICE
Q1 · Why split error rate by status code instead of one blended figure?
Because the fix differs: 400 is a non-retryable client bug, 429 means back off
(honor retry-after), and 500/529 are retryable Anthropic-side errors. A single "error %"
tells you something's wrong but not what to do.
Q2 · Which latency statistic should the SLO target, and why not the mean?
Target the tail — p95 and p99. LLM response time is highly variable, so a healthy average can hide a p99 that's blowing past user timeouts. Reliability lives in the tail.
Q3 · Which signals tell you you're approaching rate-limit saturation?
The anthropic-ratelimit-* remaining headers (RPM / ITPM / OTPM) on every
response, tracked as a saturation SLI. Alert when remaining quota drops (e.g. below 20%) so you throttle
before the 429 outage rather than after.