D5 · Context & Reliability › Cluster 1 · Context Windows & Token Budgeting

1.4 · Budgeting a Growing Conversation

History grows every turn

CONCEPT The API is stateless — you resend the whole history

The Messages API has no server-side memory. Each turn, your input phase is all previous conversation history plus the current user message, and previous turns are preserved completely. So the input side grows monotonically: every user message and assistant reply accumulates in the window, and you re-send that entire transcript on the next call. The output of one turn becomes part of the input for the next.

⚠️ The cost consequence: because you re-send the full history each turn, those input tokens are re-sent and re-billed every turn. A 50-turn chat pays for the early turns 50 times over. Cost per turn rises as the conversation grows, even if each new message is short.

CONCEPT Watch the input side climb toward the window

The danger isn't a single giant message — it's accumulation. Each turn the input grows, the room left for the reply shrinks (1.3), and eventually input + max_tokens presses against the window. Two things help you stay ahead of it:

🔑 Prompt caching changes what you pay for re-sent tokens (cache reads are far cheaper), but cached tokens still occupy the window. Caching is a cost lever, not a context-window lever — the history still counts toward the limit.

DIAGRAM Each turn re-sends everything before it

flowchart TD T1["Turn 1 input
sys + U1"] --> A1["+ A1"] A1 --> T2["Turn 2 input
sys + U1 + A1 + U2"] T2 --> A2["+ A2"] A2 --> T3["Turn 3 input
sys + U1 + A1 + U2 + A2 + U3"] T3 --> GROW{"Approaching
the window?"} GROW -->|"no"| KEEP["🟢 Keep appending"] GROW -->|"yes"| MANAGE["🟠 Compact / trim
the history (1.5)"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class KEEP ok; class MANAGE gate;

CODE Budget check each turn as history accumulates

WINDOW, MAX_OUT, SAFETY = 1_000_000, 4096, 2_000
history = []   # you own the transcript — the API stores nothing

def turn(user_msg):
    history.append({"role": "user", "content": user_msg})

    # Input grows every turn — re-check before each send.
    n_in = client.messages.count_tokens(
        model="claude-opus-4-8", messages=history,
    ).input_tokens

    if n_in + MAX_OUT + SAFETY > WINDOW:
        compact(history)          # summarize/trim oldest turns (1.5)

    resp = client.messages.create(
        model="claude-opus-4-8", max_tokens=MAX_OUT, messages=history,
    )
    # Re-sent history is re-billed: this input was paid for last turn too.
    history.append({"role": "assistant", "content": resp.content})
    return resp

TRADE-OFFS Keep full history vs. manage it

✅ Append everything

  • Perfect recall of the whole conversation
  • Simplest possible loop
  • Cache reads soften the re-billing cost

⚠️ Unbounded growth

  • Per-turn cost climbs (re-sent, re-billed)
  • Latency and context rot rise with size
  • Eventually overflows the window (1.5)

EXAM TRAPS

Trap: "The API remembers the conversation, so I only send the new message." → It's stateless. You resend the entire history every turn; the model has no memory of prior calls.
Trap: "Early turns are paid for once." → Re-sent history is re-billed each turn. The longer the chat, the more times early tokens are charged.
Trap: "Prompt caching shrinks my context usage." → Caching cuts the price of re-sent tokens; those tokens still occupy the window. It's a cost lever, not a window lever.
Trap: "Context awareness is on for every model." → Only Sonnet 5 / 4.6 / 4.5 and Haiku 4.5 get the injected budget tags. Opus 4.7+, Fable 5, Mythos 5 don't — use task budgets (beta).

PRACTICE

Q1 · Why does per-turn cost rise as a conversation grows?

The API is stateless, so each turn you re-send the full history. Those input tokens are re-sent and re-billed every turn — early turns get paid for again and again — so cost climbs with length even when new messages are short.

Q2 · Does prompt caching reduce how much of the context window the history uses?

No. Caching reduces the cost of re-sent tokens (cheap cache reads), but cached tokens still occupy the context window. To reduce window usage you must trim or compact (1.5).

Q3 · How can you tell you're approaching the window mid-conversation?

Read usage on each response, or count_tokens before sending. On context-aware models (Sonnet 5/4.6/4.5, Haiku 4.5) the model itself tracks remaining budget via injected tags; on Opus 4.7+/Fable 5/Mythos 5, give it a task budget instead.