D5 · Context & Reliability › Cluster 5 · Reliability: Errors & Retries

5.5 · Timeouts & Long Requests

Stream to avoid timeouts

CONCEPT Big max_tokens without streaming times out

A large max_tokens means a long generation, and a non-streaming request holds one idle HTTP connection open for the whole time. Two things go wrong past roughly 10 minutes: some networks silently drop idle connections (you get a failure or hang with no response), and the request exceeds practical timeout windows. Anthropic's guidance is explicit — avoid a large max_tokens on a non-streaming request. Instead, stream the response (or use the Batch API). Streaming keeps data flowing on the connection, so it never goes idle, and you can start processing tokens as they arrive.

🔑 The SDKs enforce this. They validate that a non-streaming Messages request isn't expected to exceed the ~10-minute timeout; a large max_tokens without stream=True is refused before it's even sent (Python raises a ValueError). Pass stream=True / use messages.stream(), or explicitly raise timeout, to proceed.

CONCEPT Streaming, or the Batch API — pick by latency need

There are two ways to handle a long request safely:

Interactive chat / long single response → stream. Overnight bulk generation where latency doesn't matter → Batch API (poll for results, half price).

CONCEPT Errors can still arrive mid-stream

A streaming request first returns HTTP 200, then delivers events. So an error can occur after the 200 — it arrives as an error event in the SSE stream, not as an HTTP status your normal try/except around the call would catch. Handle stream errors inside the iteration, and remember that if a stream is interrupted you may hold a partial response. Also note the default client timeout (10 minutes) is configurable, and its units differ by SDK — seconds in Python/Ruby, milliseconds in TypeScript.

DIAGRAM Route the long request

flowchart TD A["Request with large max_tokens"] --> Q{"Latency-sensitive?"} Q -->|"yes — user waiting"| S["🟢 Stream (SSE)
connection stays warm"] Q -->|"no — bulk / offline"| B["🟢 Batch API
poll for results, ~50% cost"] A -.->|"non-streaming + big max_tokens"| X["🔴 SDK refuses
(idle-timeout risk)"] S --> F["get_final_message()
for the whole reply"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class S ok; class B ok; class F ok; class X bad;

CODE Stream a 128K-token response, get the final message

Use the streaming helper so the connection never idles — then pull the complete message without handling individual events.

with client.messages.stream(
    model="claude-opus-4-8",
    max_tokens=128000,                         # large output → MUST stream
    messages=[{"role": "user", "content": "Write a detailed analysis…"}],
) as stream:
    for text in stream.text_stream:        # optional: render incrementally
        print(text, end="", flush=True)
    msg = stream.get_final_message()        # complete Message, no timeout risk

# Bulk, latency-insensitive? Use the Batch API instead — poll, don't hold a socket:
batch = client.messages.batches.create(requests=[...])   # ~50% cost, 24h window
# …poll client.messages.batches.retrieve(batch.id) until "ended", then read results

TRADE-OFFS Streaming vs. Batch API for long work

✅ Streaming (SSE)

  • Real-time — tokens as they generate
  • Connection stays warm, no idle-timeout drop
  • get_final_message() if you just want the whole reply

✅ Batch API (different job)

  • No live connection — poll for results
  • ~50% cheaper, up to 100K requests, 24h window
  • But: not real-time — wrong choice for interactive UX
⚠️ The failure mode to avoid: a big max_tokens on a non-streaming call. That's the one the SDK blocks and the network drops.

EXAM TRAPS

Trap: "Just raise max_tokens and wait." → A large max_tokens without streaming can exceed the non-streaming timeout and get dropped by idle-connection limits. Stream it (or use the Batch API).
Trap: Thinking streaming is only for showing tokens in a UI. → Even if you don't render deltas, streaming plus get_final_message() gives you timeout protection for long generations.
Trap: Wrapping a streaming call in a plain try/except and expecting to catch every error. → Streaming returns 200 first; errors can arrive as mid-stream error events, handled inside the iteration, not as an HTTP status.
Trap: Using streaming for huge offline bulk jobs. → For non-latency-sensitive volume, the Batch API is the right tool — poll for results, ~50% cheaper, no held connection.

PRACTICE

Q1 · Why must a request with a very large max_tokens stream?

A long non-streaming generation holds one idle connection open past the ~10-minute timeout; networks may drop it and the SDK refuses it. Streaming keeps the connection active with incremental events, so it never goes idle — use get_final_message() if you just want the whole reply.

Q2 · When would you choose the Batch API over streaming for a long request?

When the work is not latency-sensitive — bulk/offline generation. You submit and poll for results instead of holding a connection (removing network-reliability risk), at ~50% of standard cost. Streaming is for interactive, real-time responses.

Q3 · A streaming request returned 200, then failed. Where do you catch that?

Inside the stream iteration — as an error event in the SSE stream. Because the HTTP response already succeeded (200), the failure won't surface as a status code in a try/except around the call. You may also be left holding a partial response.