D5 · Context & Reliability › Cluster 7 · Cost & Performance Optimization

7.4 · Token-Efficient Patterns

Fewer tokens, same result

CONCEPT Every token is billed and read — on both sides

You pay for input and output tokens, and both slow the call (7.3). Token efficiency is the one lever that helps every request regardless of model, cache, or batch: cut the tokens and you cut cost and latency at once. The moves fall into two buckets — shrink what you send (prompts, context, tool results) and shrink what you ask the model to produce (concise, structured, capped output).

PatternSideWhat it cuts
Concise prompts — drop boilerplate & repetitionInputWasted instruction tokens
Don't dump huge tool resultsInputReturn only the fields needed; filter/summarize first
Ask for concise outputOutputProse the caller never reads
Structure over prose (JSON / labels)OutputFraming words around the actual answer
Cap max_tokensOutputHard ceiling — model can't run long
🔑 Structured output is often the biggest output win: a JSON object or a single label carries the answer with almost no surrounding words, and it's easier to parse downstream.

CONCEPT The biggest waste is uncontrolled context growth

In agent loops the input grows every turn — old tool results, verbose observations, and re-sent history pile up, and you pay for all of it on every subsequent call. Two habits keep it lean: don't dump raw tool output into context (return the 3 fields the model needs, not a 5,000-token API blob), and ask for structured, minimal responses so the assistant turns you re-send stay small. Pair this with a capped max_tokens so a single call can't balloon, and with prompt caching (7.5) so the stable part you must re-send is billed at ~0.1×.

Design tools to return distilled results. A search() tool that returns titles + IDs beats one that returns full documents — the model asks for the full doc only when it actually needs it.
Anti-pattern: piping a giant JSON API response straight back as a tool result. You pay for those thousands of tokens on this turn and every turn after, since they stay in history.

DIAGRAM Trim both sides of the call

flowchart TD IN["Input tokens"] --> CALL{{"Messages API call"}} CALL --> OUT["Output tokens"] P1["concise prompt
(no boilerplate)"] -.-> IN P2["filtered tool results
(not raw dumps)"] -.-> IN P3["cached stable prefix"] -.-> IN P4["ask for concise output"] -.-> OUT P5["structure over prose"] -.-> OUT P6["cap max_tokens"] -.-> OUT CALL --> WIN["🟢 Lower cost + lower latency"] classDef lever fill:#d4edda,stroke:#28a745,color:#155724; classDef win fill:#d4edda,stroke:#28a745,color:#155724; class P1 lever; class P2 lever; class P3 lever; class P4 lever; class P5 lever; class P6 lever; class WIN win;

CODE Structure + cap beats free-form prose

Ask for a small structured answer and cap the length — the model returns the data, not a paragraph.

# WASTEFUL: open-ended prose, no ceiling → pays for framing words.
# messages=[{"role":"user","content":"Tell me everything about this ticket"}]

# EFFICIENT: structured output + a hard cap.
resp = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=128,                           # ceiling — can't run long
    output_config={"format": {
        "type": "json_schema",
        "schema": {
            "type": "object",
            "properties": {
                "category": {"type": "string"},
                "priority": {"type": "string", "enum": ["low", "high"]},
            },
            "required": ["category", "priority"],
            "additionalProperties": False,
        },
    }},
    messages=[{"role": "user", "content": f"Classify: {ticket}"}],
)  # returns {"category": "...", "priority": "..."} — no prose to pay for

TRADE-OFFS Terse vs. useful

✅ Aggressive token trimming

  • Lower cost on every call, input and output
  • Faster (fewer tokens to read and generate)
  • Structured output is easy to parse
  • Keeps agent context from ballooning

⚠️ Cutting too far

  • Stripping needed context degrades answers
  • Over-tight max_tokens truncates output
  • Forcing structure on a genuinely open task can lose nuance
🔑 Cut framing and redundancy, not signal. The goal is the same result in fewer tokens — not a worse result that happens to be shorter.

EXAM TRAPS

Trap: "Token efficiency is only about a shorter prompt." → It's both sides. Output tokens are billed too (and at a higher rate) — concise, structured, capped output often saves more than trimming the prompt.
Trap: Dumping raw tool results back into context. → In a loop you re-pay for that bulk every turn. Return only the fields needed; filter or summarize large results before they enter context.
Trap: Leaving max_tokens huge "so it never truncates." → An uncapped output can run far longer than needed. Set a ceiling sized to the task; raise it only when genuinely truncated.
Trap: Confusing "fewer tokens" with "a cheaper model." → They're different dials (7.1). Token efficiency reduces the count you pay for at any model tier; they stack.

PRACTICE

Q1 · Name two input-side and two output-side token-efficiency patterns.

Input: concise prompts (drop boilerplate) and not dumping raw tool results (filter to needed fields). Output: ask for concise/structured answers (JSON or a label instead of prose) and cap max_tokens.

Q2 · Why does structured output usually save more than a "be brief" instruction?

Structured output (JSON, an enum label) carries the answer with almost no surrounding framing words, and output tokens are billed at the higher rate. It also parses reliably downstream, avoiding a re-ask.

Q3 · In an agent loop, why is dumping a huge tool result especially costly?

Because history is re-sent every turn — a 5,000-token blob is paid for on this call and every subsequent one. Returning only the needed fields keeps the re-sent context small across the whole run.