D5 · Context & Reliability › Cluster 3 · Managing Long Conversations

3.2 · Truncation / Sliding Window

Drop the oldest turns

CONCEPT Keep the last N turns, drop the oldest

The simplest way to keep a growing transcript inside the context window is a sliding window: keep the most recent turns and discard the oldest ones. You choose the bound — either a turn count ("keep the last N exchanges") or, better, a token budget ("keep as many recent turns as fit under X tokens"). Because you own the history (3.1), truncation is just slicing your messages list before you send it.

🔑 A token budget beats a turn count: turns vary wildly in size (a one-line reply vs. a huge tool result), so counting tokens keeps you reliably under the limit where counting turns can still overflow on one fat message.

Two invariants you must preserve when you slice:

CONCEPT Simple, but it forgets — and it breaks caching

Truncation's virtue is its cost: zero extra API calls, trivial to implement. Its flaw is equally blunt — anything you drop is gone forever. If the user said "my name is Alice" on turn 1 and that turn slides out of the window, the model can no longer answer "what's my name?" Truncation preserves recency at the cost of early context: names, decisions, constraints, and goals stated up front silently vanish.

Two failure signatures to recognize: (1) the model "forgets" facts established early in a long session; (2) a malformed request because you sliced through a tool_use/tool_result pair or left the first message as an assistant turn.
⚠️ Truncation breaks prompt caching if you trim the cached prefix. Caching is a prefix match — the cache key is the exact bytes from the front up to your breakpoint. Dropping the oldest turns changes the front of the prompt, so the cached prefix no longer matches and every request pays a full cache-write instead of a cheap cache-read. A sliding window and a stable cached prefix are in direct tension: you can't slide the front and keep it cached.

DIAGRAM The window slides forward, early turns fall off

flowchart TD SYS["System prompt
(pinned — never slides)"] --> WIN subgraph WIN["Sliding window: keep recent under token budget"] T3["Turn n-2"] --> T4["Turn n-1"] --> T5["Turn n (newest)"] end OLD["Turn 1 · Turn 2
(oldest)"] -.->|"exceed budget →
DROP, gone forever"| X["🗑️ Discarded
early context lost"] WIN --> SEND["Send: system + windowed turns"] X -.->|"also invalidates
the cached prefix"| CACHE["🔴 Cache miss
front bytes changed"] classDef keep fill:#d4edda,stroke:#28a745,color:#155724; classDef drop fill:#f8d7da,stroke:#dc3545,color:#721c24; classDef pin fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; class SYS,SEND pin; class T3,T4,T5 keep; class OLD,X,CACHE drop;

CODE Token-budgeted sliding window

Pin the system prompt, then keep the newest turns that fit a token budget — sliding only at turn boundaries and starting on a user turn so no tool pair is split.

# Keep system prompt pinned; slide messages under a token budget.
def truncate(system, messages, budget=120_000):
    # Walk from the NEWEST message backward, keeping turns that fit.
    kept, used = [], client.messages.count_tokens(
        model="claude-opus-4-8", system=system,
        messages=[{"role": "user", "content": "x"}]).input_tokens

    for msg in reversed(messages):
        cost = client.messages.count_tokens(
            model="claude-opus-4-8", messages=[msg]).input_tokens
        if used + cost > budget:
            break                              # drop this and everything older
        kept.append(msg); used += cost

    kept.reverse()
    # Invariant: window must start on a user turn (don't lead with assistant,
    # and never split a tool_use / tool_result pair).
    while kept and kept[0]["role"] != "user":
        kept.pop(0)
    return kept   # ⚠️ trimming the front = cached prefix no longer matches

TRADE-OFFS Truncation vs. keeping everything

✅ Sliding window

  • Dead simple, zero extra API calls, deterministic cost bound
  • Great when only recent context matters (support chat, live coding)
  • Never overflows if the budget is set below the window

⚠️ What it costs you

  • Dropped turns are lost — early facts/goals silently forgotten
  • Breaks prompt caching when the trimmed prefix changes
  • No "gist" preserved — it's a hard cut, not a compression
If losing early context is unacceptable, don't truncate — compress it into a summary (3.3) so the gist survives, or clear only stale tool output (3.4) instead of whole turns.

EXAM TRAPS

Trap: "A sliding window keeps a summary of the old turns." → No — truncation drops them entirely. Preserving the gist is summarization (3.3), a different strategy.
Trap: "Truncation is free of downsides because it's cheap." → It loses early context and it breaks prompt caching whenever you trim the cached prefix (a prefix change forces a fresh cache write).
Trap: Slicing mid-tool-pair or leaving an assistant message first. → The request 400s. Slide at turn boundaries; keep each tool_use with its tool_result; start the window on a user turn.
Trap: Counting turns instead of tokens. → One giant tool result can blow the window even with "only 10 turns." Bound the window by a token budget, not a turn count.

PRACTICE

Q1 · What is a sliding window and what does it sacrifice?

Keep the most recent turns (bounded by a token budget or turn count) and drop the oldest. It sacrifices early context — dropped turns are gone forever, so facts, goals, or decisions stated up front are silently forgotten. It preserves recency, not the full history.

Q2 · Why can truncation hurt your prompt-cache hit rate?

Caching is a prefix match keyed on the exact leading bytes. Dropping the oldest turns changes the front of the prompt, so the cached prefix no longer matches — you pay a full cache write instead of a cheap read. Sliding the front and keeping it cached are mutually exclusive.

Q3 · When is a plain sliding window the right choice — and when isn't it?

Right when only recent context matters and simplicity/cost win (support tickets, live coding sessions). Wrong when early context is load-bearing (long research or planning sessions) — there, summarize/compact (3.3) or use external memory (3.5) so the gist survives instead of being cut.