CONCEPT Keep the last N turns, drop the oldest
The simplest way to keep a growing transcript inside the context window is a sliding window:
keep the most recent turns and discard the oldest ones. You choose the bound — either a
turn count ("keep the last N exchanges") or, better, a token budget ("keep as
many recent turns as fit under X tokens"). Because you own the history (3.1), truncation is just slicing
your messages list before you send it.
Two invariants you must preserve when you slice:
- Keep the system prompt. It's not part of the sliding window — pin it and slide only
the
messages. - Don't split a tool-use pair. A
tool_useblock must keep its matchingtool_result; dropping one without the other produces a malformed request (400). Slide at turn boundaries, and start the window on auserturn.
CONCEPT Simple, but it forgets — and it breaks caching
Truncation's virtue is its cost: zero extra API calls, trivial to implement. Its flaw is equally blunt — anything you drop is gone forever. If the user said "my name is Alice" on turn 1 and that turn slides out of the window, the model can no longer answer "what's my name?" Truncation preserves recency at the cost of early context: names, decisions, constraints, and goals stated up front silently vanish.
tool_use/tool_result
pair or left the first message as an assistant turn.DIAGRAM The window slides forward, early turns fall off
(pinned — never slides)"] --> WIN subgraph WIN["Sliding window: keep recent under token budget"] T3["Turn n-2"] --> T4["Turn n-1"] --> T5["Turn n (newest)"] end OLD["Turn 1 · Turn 2
(oldest)"] -.->|"exceed budget →
DROP, gone forever"| X["🗑️ Discarded
early context lost"] WIN --> SEND["Send: system + windowed turns"] X -.->|"also invalidates
the cached prefix"| CACHE["🔴 Cache miss
front bytes changed"] classDef keep fill:#d4edda,stroke:#28a745,color:#155724; classDef drop fill:#f8d7da,stroke:#dc3545,color:#721c24; classDef pin fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; class SYS,SEND pin; class T3,T4,T5 keep; class OLD,X,CACHE drop;
CODE Token-budgeted sliding window
Pin the system prompt, then keep the newest turns that fit a token budget — sliding only at turn boundaries
and starting on a user turn so no tool pair is split.
# Keep system prompt pinned; slide messages under a token budget. def truncate(system, messages, budget=120_000): # Walk from the NEWEST message backward, keeping turns that fit. kept, used = [], client.messages.count_tokens( model="claude-opus-4-8", system=system, messages=[{"role": "user", "content": "x"}]).input_tokens for msg in reversed(messages): cost = client.messages.count_tokens( model="claude-opus-4-8", messages=[msg]).input_tokens if used + cost > budget: break # drop this and everything older kept.append(msg); used += cost kept.reverse() # Invariant: window must start on a user turn (don't lead with assistant, # and never split a tool_use / tool_result pair). while kept and kept[0]["role"] != "user": kept.pop(0) return kept # ⚠️ trimming the front = cached prefix no longer matches
TRADE-OFFS Truncation vs. keeping everything
✅ Sliding window
- Dead simple, zero extra API calls, deterministic cost bound
- Great when only recent context matters (support chat, live coding)
- Never overflows if the budget is set below the window
⚠️ What it costs you
- Dropped turns are lost — early facts/goals silently forgotten
- Breaks prompt caching when the trimmed prefix changes
- No "gist" preserved — it's a hard cut, not a compression
EXAM TRAPS
assistant message first. → The
request 400s. Slide at turn boundaries; keep each tool_use with its tool_result; start
the window on a user turn.PRACTICE
Q1 · What is a sliding window and what does it sacrifice?
Keep the most recent turns (bounded by a token budget or turn count) and drop the oldest. It sacrifices early context — dropped turns are gone forever, so facts, goals, or decisions stated up front are silently forgotten. It preserves recency, not the full history.
Q2 · Why can truncation hurt your prompt-cache hit rate?
Caching is a prefix match keyed on the exact leading bytes. Dropping the oldest turns changes the front of the prompt, so the cached prefix no longer matches — you pay a full cache write instead of a cheap read. Sliding the front and keeping it cached are mutually exclusive.
Q3 · When is a plain sliding window the right choice — and when isn't it?
Right when only recent context matters and simplicity/cost win (support tickets, live coding sessions). Wrong when early context is load-bearing (long research or planning sessions) — there, summarize/compact (3.3) or use external memory (3.5) so the gist survives instead of being cut.