D5 · Context & Reliability › Cluster 6 · Reliability: Degradation & Recovery

6.5 · Streaming for Reliability

Deliver as you generate

CONCEPT Streaming isn't just for UX — it's a timeout defense

Set "stream": true and the Messages API returns the response incrementally over server-sent events (SSE) instead of one big blocking payload. The obvious win is UX (tokens appear live). The reliability win is bigger: a long non-streaming request holds one idle connection open for the whole generation, and some networks drop idle connections — so a long answer can fail or hit an HTTP timeout with nothing to show. Streaming keeps bytes flowing, so the connection never sits idle.

🔑 The docs are explicit: for large max_tokens or generations over ~10 minutes, use streaming (or the Batches API). The SDKs even refuse a non-streaming request they estimate will exceed the ~10-minute timeout — because idle-connection drops make it unreliable.

CONCEPT The event sequence, and how to recover from a drop

An SSE stream has a fixed high-level shape. Handle unknown event types gracefully — new ones may be added.

EventFiresCarries
message_startOnce, firstThe Message shell (empty content)
content_block_startEach block beginsBlock type + index
content_block_deltaPer token/chunkIncremental text / JSON / thinking delta
content_block_stopEach block ends—
message_deltaTop-level updatesstop_reason + cumulative usage
message_stopOnce, lastStream complete
⚠️ ping events appear anywhere (ignore them). And an error event can arrive mid-stream after the initial 200 — e.g. an overloaded_error (the streaming analog of a 529). A mid-stream error does not follow normal HTTP error handling; you must catch it on the event stream and fall back / retry (6.1) from the partial you have.
On a disconnect you hold the tokens received so far — reconnect and continue from the partial (6.4), or fall back (6.1). Let the SDK helper accumulate for you: .get_final_message() / .finalMessage() gives you the complete Message with streaming's timeout protection but without hand-handling every event.

DIAGRAM The SSE lifecycle (with a mid-stream error)

flowchart TD MS["message_start
(empty Message)"] --> CBS["content_block_start (index i)"] CBS --> CBD["content_block_delta ×N
(text / json / thinking)"] CBD --> CBST["content_block_stop"] CBST -->|"more blocks"| CBS CBST -->|"done"| MD["message_delta
(stop_reason + cumulative usage)"] MD --> MST["🟢 message_stop"] CBD -.->|"overloaded_error
mid-stream"| ERR["🔴 error event
→ keep partial, retry/fallback (6.1)"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class MST ok; class ERR bad;

CODE Stream with progress, then recover on error

Deltas render live (progress + no idle connection); a mid-stream error keeps the partial and hands off to the fallback ladder from 6.1.

def stream_answer(client, messages):
    partial = ""
    try:
        with client.messages.stream(
            model="claude-opus-4-8", max_tokens=64000,   # large → streaming a must
            messages=messages) as stream:
            for text in stream.text_stream:
                partial += text
                render(text)                          # show progress live
            final = stream.get_final_message()          # accumulated Message
            if final.stop_reason == "max_tokens":
                return continue_from(partial, messages)  # 6.4
            return partial
    except (anthropic.APIStatusError,               # incl. mid-stream overloaded
            anthropic.APIConnectionError):            # dropped connection
        return answer_with_fallback(messages, seed=partial)  # 6.1, reuse partial

TRADE-OFFS Streaming vs. one blocking call

✅ Streaming (SSE)

  • No idle connection → survives long generations
  • Live progress; partial output on disconnect
  • Required for large max_tokens / >10-min work

⚠️ Non-streaming

  • Simpler: one request, one response object
  • But idle connection can be dropped mid-generation
  • Mid-stream errors & timeouts surface as a hard failure
Default to streaming for anything long; use .get_final_message() when you only need the finished Message — you get the timeout protection without event-by-event handling. For huge, non-latency-sensitive jobs, the Batches API is the poll-based alternative.

EXAM TRAPS

Trap: Thinking streaming is only a UX nicety. → It's a reliability mechanism: it prevents idle-connection drops and HTTP timeouts on long outputs. Large max_tokens essentially requires it.
Trap: Assuming errors always come as an HTTP status. → After a 200, an error event (e.g. overloaded_error) can arrive mid-stream. Handle it on the event stream, keep the partial, retry/fallback (6.1).
Trap: Reading per-event usage as incremental. → The usage in message_delta is cumulative, not per-delta. Don't sum it across events.
Trap: Crashing on an unknown event type. → New event types can be added; handle them gracefully. Ignore ping events entirely.

PRACTICE

Q1 · Why stream a long generation instead of making one blocking call?

A blocking call holds one idle connection open for the whole generation; some networks drop idle connections, so a long answer can fail or hit an HTTP timeout with nothing returned. Streaming keeps bytes flowing (no idle drop), shows progress, and leaves you a usable partial on disconnect. For large max_tokens the SDKs require it.

Q2 · An overloaded_error arrives after you already got a 200. How?

On a streaming response the initial HTTP status is 200, but an error event can be emitted mid-stream (the SSE analog of a 529). It bypasses normal HTTP error handling — catch it on the event stream, keep the tokens you have, and retry or fall back (6.1).

Q3 · Name the streaming event sequence at a high level.

message_start → for each block: content_block_start, one or more content_block_delta, content_block_stop → one or more message_delta (carries stop_reason + cumulative usage) → message_stop. ping and error events may be interspersed.