D1 · Agentic Architecture › Cluster 7 · Performance

7.4 · Streaming in Long Agentic Runs

Stream big requests or they time out

CONCEPT Why stream at all

A large generation can take longer than an HTTP connection is willing to stay open waiting for a single response body. Streaming sends the response incrementally, keeping the connection alive. The architect's rule: stream any request with a large max_tokens (≳16K) to avoid HTTP timeouts. In long agentic runs — where turns can produce big outputs — this is not optional.

🔑 The trigger is large max_tokens (≳16K), not the model. A big generation may exceed the non-streaming timeout window; streaming keeps bytes flowing so the connection never idles out.

CONCEPT The event stream & get_final_message

A streamed request emits a sequence of typed events. You usually don't hand-assemble them — the SDK helper client.messages.stream(...) yields events and then stream.get_final_message() reconstructs the complete Message (content, stop_reason, usage) so the rest of your agent loop is unchanged.

EventWhat it signals
message_startThe response has begun; metadata arrives.
content_block_deltaIncremental chunks of a content block (text, tool input…).
message_deltaTop-level updates — carries stop_reason and usage.
message_stopThe stream is complete.

DIAGRAM Streaming a long turn

flowchart TD A([Big request
max_tokens ≳ 16K]) --> S["client.messages.stream(...)"] S --> E1["🔵 message_start"] E1 --> E2["🟢 content_block_delta
(repeats — keeps HTTP alive)"] E2 --> E3["🟠 message_delta
stop_reason + usage"] E3 --> E4["🔵 message_stop"] E4 --> F["stream.get_final_message()
→ complete Message"] F --> L([resume agent loop]) classDef s fill:#d1ecf1,stroke:#17a2b8,color:#0c5460; classDef d fill:#d4edda,stroke:#28a745,color:#155724; classDef m fill:#ffe8cc,stroke:#d9480f,color:#7a2e08; class E1 s; class E4 s; class E2 d; class E3 m;

CODE Streaming with get_final_message

Use the context-manager helper. Consume deltas if you want live output, then collapse the stream into a normal Message and continue exactly as in a non-streamed loop.

# Large max_tokens → stream to avoid HTTP timeouts on a long generation.
with client.messages.stream(
    model="claude-opus-4-8",
    max_tokens=32000,          # ≳16K → stream, don't block on one body
    tools=TOOLS,
    messages=messages,
) as stream:
    for event in stream:
        if event.type == "content_block_delta":
            render(event.delta)          # optional live UI; keeps conn warm
        elif event.type == "message_delta":
            # message_delta carries stop_reason + usage
            pass

    # Reconstruct the full Message — same shape as a non-streamed .create()
    final = stream.get_final_message()

print(final.stop_reason, final.usage.output_tokens)
messages.append({"role": "assistant", "content": final.content})
# ...run any tool the model chose, append tool_result, loop as usual

TRADE-OFFS Stream vs. block

✅ Streaming

  • Avoids HTTP timeouts on large max_tokens
  • Enables live, token-by-token UX
  • Same final Message via get_final_message()

⚠️ Cost of streaming

  • More client-side plumbing (event handling)
  • Overkill for tiny, fast responses
  • Partial output must be finalized before you act on it
Heuristic: small/fast response → create() is fine; large generation (max_tokens ≳ 16K) or a run that needs live output → stream.

EXAM TRAPS

Trap: Issuing a huge non-streamed request and blaming the model when it "hangs." → A large max_tokens (≳16K) can hit an HTTP timeout. Stream it.
Trap: Acting on partial deltas as if they were the final answer. → Finalize with stream.get_final_message() before you branch on stop_reason or run a tool.
Trap: Expecting usage/stop_reason on the first event. → They arrive on message_delta; get_final_message() is the reliable place to read them.

PRACTICE

Q1 · A report-generation call with max_tokens=40000 intermittently fails with a connection timeout. Fix?

Switch to client.messages.stream(...). A generation that large can exceed the non-streaming HTTP window; streaming keeps the connection alive with incremental content_block_delta events.

Q2 · How do you fold a streamed turn back into a normal agent loop?

Call stream.get_final_message() to get a complete Message, append final.content to messages, run any chosen tool, and loop — identical to a .create()-based loop.

Q3 · Which event carries stop_reason and usage?

message_delta. The full sequence is message_start → content_block_delta… → message_delta → message_stop; reading the finalized message is the simplest way to get both.