D5 · Context & Reliability › Cluster 3 · Managing Long Conversations

3.3 · Summarization / Compaction

Compress instead of drop

CONCEPT Replace old turns with an LLM-generated summary

Truncation (3.2) throws early turns away. Summarization — also called compaction — keeps their meaning: you feed the old turns to the model, ask it to write a compact summary, then replace those turns with the summary in your messages list. The transcript shrinks, but the gist survives: decisions made, facts established, the user's goal, where the work stands.

🔑 The trade at the heart of the technique: you spend an extra LLM call (and some fidelity) to buy back context that truncation would have lost. Compression, not deletion.

A typical pattern: keep the last few turns verbatim (recency matters), summarize everything older into a single block, and prepend that block so the model still "remembers" the arc of the conversation. You can do this client-side (your own summarize-then-splice loop) or lean on Anthropic's server-side compaction, a beta feature that automatically summarizes earlier context on the server as the conversation approaches the context-window limit.

Server-side compaction (beta header compact-2026-01-12, on Fable 5 / Opus 4.8 / 4.7 / 4.6 / Sonnet 5 / 4.6) returns a compaction block in the response. Critical: append the full response.content — not just the text — back to messages. The API uses that block to replace the compacted history on the next request; keep only the text and you silently lose the compaction state.

CONCEPT The trade-off vs. raw truncation

Summarization sits between "keep everything" (accurate but overflows) and "drop the oldest" (cheap but forgets). Versus raw truncation:

Truncation (3.2)Summarization / Compaction (3.3)
Old contextDeleted — goneCompressed — gist retained
Extra costNoneAn extra LLM call to summarize
FidelityPerfect for kept turns, zero for droppedLossy: details/exact wording can blur
Best whenOnly recency mattersEarly goals/decisions must survive
⚠️ Summaries are lossy and can drift: exact numbers, names, or verbatim requirements may get smoothed over or, worse, hallucinated. Anything that must be recalled exactly is better written to durable external memory (3.5) than trusted to a prose summary. Summarizing also, like truncation, rewrites the prompt prefix — so it invalidates prompt caching from the point where content changed.

DIAGRAM Old turns collapse into one summary block

flowchart TD OLD["Turns 1…k
(old history)"] -->|"feed to model:
'summarize this'"| SUM["📝 Summary block
decisions · facts · goal · state"] SUM --> NEW["messages = [ summary ] + recent turns"] RECENT["Turns k+1…n
(kept verbatim)"] --> NEW NEW --> SEND["Send: smaller transcript,
gist preserved"] OLD -.->|"contrast: truncation would
DELETE turns 1…k"| DROP["🗑️ gist lost"] classDef old fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; classDef good fill:#d4edda,stroke:#28a745,color:#155724; classDef bad fill:#f8d7da,stroke:#dc3545,color:#721c24; class OLD old; class SUM,NEW,SEND,RECENT good; class DROP bad;

CODE Client-side summarize-and-splice

When the transcript gets big, summarize the old turns with a cheap call, then replace them with the summary and keep the recent tail verbatim.

# Compress old turns into a summary; keep recent turns verbatim.
KEEP_RECENT = 6

def compact(messages):
    if len(messages) <= KEEP_RECENT:
        return messages
    old, recent = messages[:-KEEP_RECENT], messages[-KEEP_RECENT:]

    summary = client.messages.create(
        model="claude-opus-4-8", max_tokens=1024,
        system="Summarize this conversation. Preserve decisions, facts, "
               "the user's goal, and open tasks. Be faithful; do not invent.",
        messages=old,
    ).content[0].text

    # Replace the old turns with ONE summary block, keep the recent tail.
    return [
        {"role": "user",
         "content": f"[Summary of earlier conversation]\n{summary}"},
        *recent,
    ]

# Server-side alternative (beta): let the API compact for you.
# resp = client.beta.messages.create(
#     betas=["compact-2026-01-12"], model="claude-opus-4-8", max_tokens=1024,
#     context_management={"edits": [{"type": "compact_20260112"}]}, messages=messages)
# messages.append({"role": "assistant", "content": resp.content})  # keep the compaction block!

TRADE-OFFS Compress vs. drop vs. keep

✅ Summarization / compaction

  • Keeps the gist of the whole conversation, not just the tail
  • Lets a session run far past the raw window limit
  • Server-side compaction automates it — no client bookkeeping

⚠️ What it costs

  • An extra model call (latency + tokens) to produce the summary
  • Lossy: exact details can blur or be hallucinated
  • Rewrites the prefix → invalidates prompt cache from that point
Pair it with external memory (3.5): compaction keeps the active context small, while memory files hold the exact facts that must survive summarization intact.

EXAM TRAPS

Trap: "Summarization and truncation are the same thing." → Truncation deletes old turns; summarization compresses them into a block. One loses the gist, the other preserves it at the cost of an extra call and some fidelity.
Trap: With server-side compaction, appending only the summary text. → You must append the full response.content including the compaction block; the API needs it to replace the compacted history next turn. Keep only the text and you lose the compaction state.
Trap: Trusting a summary for exact recall. → Summaries are lossy and can drift or hallucinate. Store numbers, IDs, and verbatim requirements in external memory (3.5), not prose.
Trap: Assuming compaction is free of cache impact. → It rewrites the prefix, so it invalidates prompt caching from the changed point onward — same as truncation.

PRACTICE

Q1 · How does summarization differ from truncation, and what's the core trade-off?

Truncation drops old turns; summarization replaces them with an LLM-generated summary that preserves the gist (decisions, facts, goal, state). The trade-off: you spend an extra model call and accept some fidelity loss in exchange for not forgetting early context.

Q2 · With Anthropic's server-side compaction, what must you preserve on each turn?

The full response.content — including the compaction block — must be appended back to messages. The API uses that block to replace the compacted history on the next request. Appending only the text string silently loses the compaction state.

Q3 · Why isn't a summary a safe place for exact facts, and what fixes that?

Summaries are lossy: exact numbers, names, and verbatim requirements can be smoothed over or hallucinated. Write those to durable external memory (3.5) — files Claude reads back exactly — while using compaction to keep the active transcript small.