D4 · Prompt Engineering › Cluster 5 · Chain of Thought & Extended Thinking

5.5 · CoT Trade-offs & When Not to Use

Latency & tokens vs accuracy

CONCEPT Reasoning is a spend, not a default

Every form of reasoning in this cluster — prompted CoT, structured tags, native extended thinking — buys accuracy with output tokens and latency. That's a great trade on genuinely hard problems and a pure tax on easy ones. The engineering discipline is to switch reasoning on where it moves the needle and leave it off where it doesn't — not to sprinkle "think step by step" everywhere.

🔑 The decision rule: does the task have a real risk of a wrong one-shot answer? If yes, reasoning earns its cost. If a direct answer is already reliable, reasoning is latency and spend for nothing.

CONCEPT When to skip it

Turn reasoning off (or never add it) for tasks that are simple, latency-sensitive, or high-volume:

Skip / disable reasoning for: classification into known labels, extraction, format conversion, lookups, sentiment tagging, short factual answers, and real-time UX where every hundred milliseconds counts.
Keep reasoning for: multi-step math, logical proofs, multi-factor analysis, agentic tool loops (5.4), and anything where a wrong answer is expensive and a naive answer is unreliable.

Two dials let you turn the cost down without turning reasoning fully off: lower the effort level, or (on models that support it) cap the manual budget_tokens. On current models, effort: "low" is the light-touch setting; "high"/"max" is for correctness-critical work.

DIAGRAM Decide, then dial

flowchart TD A["Task"] --> Q{"Real risk of a
wrong one-shot answer?"} Q -->|"no"| OFF["🟢 No reasoning
direct answer — fast & cheap"] Q -->|"yes"| ON{"Latency-sensitive?"} ON -->|"very"| LOW["🔵 Reason lightly
effort: low / small budget"] ON -->|"correctness matters most"| HIGH["🟠 Reason deeply
effort: high / max"] classDef ok fill:#d4edda,stroke:#28a745,color:#155724; classDef mid fill:#cfe2ff,stroke:#0d6efd,color:#084298; classDef gate fill:#ffe8cc,stroke:#ff9f40,color:#9a4a0a; class OFF ok; class LOW mid; class HIGH gate;

CODE Match the reasoning level to the task

One codebase, three tiers: no reasoning for classification, light reasoning for a moderate task, deep reasoning where correctness dominates cost.

# Tier 1 — simple classification: NO reasoning. Fast, cheap, reliable.
client.messages.create(model="claude-haiku-4-5", max_tokens=10,
    messages=[{"role": "user", "content": "Label sentiment (pos/neg/neutral): 'Loved it!'"}])

# Tier 2 — moderate task, latency-sensitive: light reasoning.
client.messages.create(model="claude-opus-4-8", max_tokens=4000,
    thinking={"type": "adaptive"},
    output_config={"effort": "low"},           # dial the cost down
    messages=[{"role": "user", "content": "Summarize and flag any billing risk..."}])

# Tier 3 — correctness > cost: deep reasoning.
client.messages.create(model="claude-opus-4-8", max_tokens=16000,
    thinking={"type": "adaptive"},
    output_config={"effort": "max"},           # spend for accuracy
    messages=[{"role": "user", "content": "Audit this contract for conflicting clauses..."}])
⚠️ Reasoning tokens count toward max_tokens and the bill even when you set display: "omitted" — hiding the reasoning text does not make it free. Budget max_tokens generously so a long think doesn't truncate the answer.

TRADE-OFFS The knobs at a glance

SettingAccuracyLatency / tokensUse when
No reasoningFine on easy tasksLowestClassify, extract, convert, look up
Prompted CoTBetter on hard tasksHigher (inline reasoning)No native thinking, or you want visible steps
Adaptive + effort: lowGoodModerateModerate tasks, latency-sensitive
Adaptive + effort: high/maxBestHighestCorrectness matters more than cost

EXAM TRAPS

Trap: "More reasoning is always better." → On simple tasks it adds latency and tokens with no accuracy gain. Match the reasoning level to the task's difficulty.
Trap: Thinking display: "omitted" saves money. → It only hides the reasoning text; the thinking tokens are still generated and billed.
Trap: Setting a tiny max_tokens with deep reasoning on. → A long think can eat the budget and truncate the answer. Reasoning tokens count toward max_tokens.
Trap: Treating effort as all-or-nothing. → It's a dial — low for latency-sensitive work, high/max when correctness outweighs cost. You don't have to choose between "off" and "full."

PRACTICE

Q1 · A high-volume sentiment classifier is slow and expensive after someone added extended thinking. Fix?

Remove reasoning entirely — classification into known labels is a simple task where a direct answer is equally accurate. Reasoning here is pure latency and token cost with no accuracy benefit.

Q2 · Does display: "omitted" reduce your bill?

No. It only suppresses the reasoning text in the response (helping time-to-first-token when streaming). The full thinking tokens are still generated and billed.

Q3 · You need maximum accuracy on a legal audit and cost is secondary. What settings?

Adaptive thinking with output_config: {effort: "high"} or "max", and a generous max_tokens so the reasoning doesn't truncate the answer. This is exactly the task where the token/latency spend earns its keep.