D5 · Context & Reliability › Cluster 6 · Reliability: Degradation & Recovery

6.4 · max_tokens Truncation & Continuation

Finish a cut-off response

CONCEPT stop_reason: "max_tokens" means the answer is cut off, not done

Every response carries a stop_reason. When Claude finishes naturally it's end_turn; when it wants a tool it's tool_use. But stop_reason: "max_tokens" means the model hit your max_tokens ceiling mid-generation — the text is truncated, possibly mid-sentence or mid-JSON. The single biggest reliability bug here is treating that partial output as a complete answer.

🔑 Always branch on stop_reason before consuming content. If it's max_tokens, the response is incomplete — you either raise the ceiling and retry, or continue generation from where it stopped.
⚠️ Truncated JSON / structured output is the nastiest case: json.loads() throws on a half-written object. Check stop_reason first; don't parse a max_tokens body as if it were whole.

CONCEPT Continue: feed the partial back, ask it to keep going

To recover a cut-off answer, append the model's partial output as an assistant turn, then add a short user message asking it to continue. The model sees its own prior text as context and picks up where it left off; you concatenate the two halves.

Keep the last message a user turn. On current models (Opus 4.8/4.7/4.6, Sonnet 5/4.6, Fable 5) an assistant message as the last entry is a prefill and returns a 400. So append {assistant: partial} then {user: "Continue from where you left off."} — the assistant block is history, not a prefill.

🔑 The other lever is prevention: right-size max_tokens. Too low → routine truncation and continuation round-trips. Too high without streaming → the SDK refuses / risks an HTTP timeout on long generations (see 5.5 / 6.5). Size it to the expected output, and stream when it's large.

DIAGRAM Detect truncation, then continue

flowchart TD R["Response"] --> S{"stop_reason?"} S -->|"end_turn"| DONE["🟢 Complete — use content"] S -->|"tool_use"| TOOL["Run tool, loop"] S -->|"max_tokens"| CUT["🟠 Truncated — output incomplete"] CUT --> APP["Append partial as ASSISTANT turn"] APP --> ASK["Add USER msg:
'Continue where you left off'"] ASK --> R2["Re-call → concatenate halves"] R2 --> S classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; class DONE ok; class CUT gate; class APP gate; class ASK gate;

CODE Detect max_tokens and continue to completion

Loop until end_turn, stitching each partial together. A capped retry count guards against a runaway continuation loop.

def complete(client, messages, max_tokens=2048, max_conts=4):
    full = ""
    for _ in range(max_conts + 1):
        resp = client.messages.create(
            model="claude-opus-4-8", max_tokens=max_tokens, messages=messages)
        part = next(b.text for b in resp.content if b.type == "text")
        full += part

        if resp.stop_reason != "max_tokens":            # end_turn → done
            return full

        # Truncated: feed the partial back as history, then a user nudge.
        # NOT a trailing assistant turn (that's a prefill → 400 on 4.x models).
        messages = messages + [
            {"role": "assistant", "content": part},
            {"role": "user",
             "content": "Continue exactly where you left off; do not repeat."},
        ]
    return full                                        # hit continuation cap

TRADE-OFFS Continue vs. just raise max_tokens

✅ Continue from the partial

  • Recovers work already generated — no wasted tokens
  • Handles outputs longer than any single ceiling
  • Keeps max_tokens modest per call

⚠️ Just crank max_tokens high

  • Simpler — one call, no stitching
  • But wastes budget when most answers are short
  • Very high values need streaming or hit HTTP timeouts
Right-size max_tokens to the typical output; keep continuation as the safety net for the occasional long answer. Stream (6.5) whenever the ceiling is large.

EXAM TRAPS

Trap: Treating a max_tokens response as complete. → It's truncated. Always check stop_reason; continue or raise the ceiling before using the content.
Trap: Continuing by putting the partial as the last assistant message. → That's a prefill and returns a 400 on Opus 4.8/4.7/4.6, Sonnet 5/4.6, and Fable 5. Append the partial as history, then end with a user "continue" turn.
Trap: Parsing truncated JSON. → A max_tokens cut-off can slice a structured output mid-object; json.loads() throws. Detect truncation first, then continue or increase max_tokens.
Trap: Fixing truncation by setting max_tokens enormous with no streaming. → Large ceilings without streaming risk idle-connection HTTP timeouts (the SDK even refuses them). Stream, or continue in modest chunks.

PRACTICE

Q1 · A response comes back with stop_reason: "max_tokens". Is it done?

No — it was cut off at the token ceiling and is incomplete (maybe mid-sentence or mid-JSON). Continue generation (feed the partial back + a user "continue" nudge) or raise max_tokens and retry. Never consume it as a finished answer.

Q2 · Why not just append the partial as a trailing assistant message and re-call?

A trailing assistant message is a prefill, which returns a 400 on current models (Opus 4.8/4.7/4.6, Sonnet 5/4.6, Fable 5). Keep the assistant partial as history and make the last message a user turn asking it to continue.

Q3 · How do you decide the right max_tokens?

Size it to the expected output length: too low forces frequent truncation and continuation round-trips; too high wastes budget and, without streaming, risks HTTP timeouts. Set a sensible default, keep continuation as the safety net, and stream for large ceilings.