CONCEPT Big max_tokens without streaming times out
A large max_tokens means a long generation, and a non-streaming request holds one idle HTTP
connection open for the whole time. Two things go wrong past roughly 10 minutes: some networks
silently drop idle connections (you get a failure or hang with no response), and the request
exceeds practical timeout windows. Anthropic's guidance is explicit — avoid a large
max_tokens on a non-streaming request. Instead, stream the response (or
use the Batch API). Streaming keeps data flowing on the connection, so it never goes idle, and you can start
processing tokens as they arrive.
max_tokens without stream=True is refused
before it's even sent (Python raises a ValueError). Pass stream=True / use
messages.stream(), or explicitly raise timeout, to proceed.CONCEPT Streaming, or the Batch API — pick by latency need
There are two ways to handle a long request safely:
- Streaming (SSE) — for interactive/real-time work. Tokens arrive incrementally as
Server-Sent Events; the connection stays warm. If you don't need per-token handling, the SDK helpers
(
get_final_message()/finalMessage()) consume the stream for you and hand back the completeMessage— identical to a non-streaming call, but without the timeout risk. - Message Batches API — for non-latency-sensitive bulk work. You submit up to 100,000 requests, then poll for results rather than holding a connection, so network reliability stops mattering. Most batches finish within an hour (24h max), at ~50% of standard price.
CONCEPT Errors can still arrive mid-stream
A streaming request first returns HTTP 200, then delivers events. So an error can occur
after the 200 — it arrives as an error event in the SSE stream, not as an HTTP status your
normal try/except around the call would catch. Handle stream errors inside the iteration, and
remember that if a stream is interrupted you may hold a partial response. Also note the default client
timeout (10 minutes) is configurable, and its units differ by SDK — seconds in Python/Ruby,
milliseconds in TypeScript.
DIAGRAM Route the long request
connection stays warm"] Q -->|"no — bulk / offline"| B["🟢 Batch API
poll for results, ~50% cost"] A -.->|"non-streaming + big max_tokens"| X["🔴 SDK refuses
(idle-timeout risk)"] S --> F["get_final_message()
for the whole reply"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class S ok; class B ok; class F ok; class X bad;
CODE Stream a 128K-token response, get the final message
Use the streaming helper so the connection never idles — then pull the complete message without handling individual events.
with client.messages.stream( model="claude-opus-4-8", max_tokens=128000, # large output → MUST stream messages=[{"role": "user", "content": "Write a detailed analysis…"}], ) as stream: for text in stream.text_stream: # optional: render incrementally print(text, end="", flush=True) msg = stream.get_final_message() # complete Message, no timeout risk # Bulk, latency-insensitive? Use the Batch API instead — poll, don't hold a socket: batch = client.messages.batches.create(requests=[...]) # ~50% cost, 24h window # …poll client.messages.batches.retrieve(batch.id) until "ended", then read results
TRADE-OFFS Streaming vs. Batch API for long work
✅ Streaming (SSE)
- Real-time — tokens as they generate
- Connection stays warm, no idle-timeout drop
get_final_message()if you just want the whole reply
✅ Batch API (different job)
- No live connection — poll for results
- ~50% cheaper, up to 100K requests, 24h window
- But: not real-time — wrong choice for interactive UX
max_tokens on a non-streaming call.
That's the one the SDK blocks and the network drops.EXAM TRAPS
max_tokens and wait." → A large max_tokens
without streaming can exceed the non-streaming timeout and get dropped by idle-connection limits. Stream it
(or use the Batch API).get_final_message() gives you timeout protection for long generations.error events, handled inside
the iteration, not as an HTTP status.PRACTICE
Q1 · Why must a request with a very large max_tokens stream?
A long non-streaming generation holds one idle connection open past the ~10-minute timeout;
networks may drop it and the SDK refuses it. Streaming keeps the connection active with incremental events, so
it never goes idle — use get_final_message() if you just want the whole reply.
Q2 · When would you choose the Batch API over streaming for a long request?
When the work is not latency-sensitive — bulk/offline generation. You submit and poll for results instead of holding a connection (removing network-reliability risk), at ~50% of standard cost. Streaming is for interactive, real-time responses.
Q3 · A streaming request returned 200, then failed. Where do you catch that?
Inside the stream iteration — as an error event in the SSE stream. Because
the HTTP response already succeeded (200), the failure won't surface as a status code in a try/except around the
call. You may also be left holding a partial response.