CONCEPT Two clocks: time-to-first-token and total time
Latency has two parts. Time-to-first-token (TTFT) — how long before anything appears — is dominated by how many input tokens must be read before generation starts. Total time is TTFT plus generation, and generation time scales with how many output tokens the model produces. The levers below attack one clock or the other. Note the Batch API (7.2) is the opposite trade — it minimizes cost by accepting high latency, so none of this applies to batched work.
| Lever | Attacks | Why it helps |
|---|---|---|
| Faster / smaller model | TTFT + total | Smaller models generate faster; Haiku < Sonnet < Opus in latency |
| Stream the response | Perceived TTFT | User sees tokens as they arrive, not after the full reply |
| Cache the prompt prefix | TTFT | Cached tokens are read far faster than fresh ones |
| Minimize input tokens | TTFT | Fewer tokens to read before the first output token |
| Constrain output length | Total time | Fewer output tokens = less generation time |
| Keep the tool set small | TTFT | Tool schemas are input tokens the model reads every call |
CONCEPT Cut the tokens the model has to read and write
Right-size the model. A smaller model is faster end-to-end; if Haiku clears the task, it's
both cheaper (7.1) and quicker. Shrink the input: trim boilerplate, don't dump giant tool
results into context, and put a large stable prefix behind cache_control so repeat calls
skip re-reading it. Keep the tool set small — every tool definition is input tokens read on
every request, and a bloated tool list slows TTFT and muddies tool selection. Cap the output:
set a sensible max_tokens and ask for concise, structured answers so the model doesn't generate a
paragraph where a label will do.
DIAGRAM Where each lever lands
(prompt + tools)"] READ --> FT(["First token — TTFT"]) FT --> GEN["Generate output tokens"] GEN --> DONE(["Full response"]) L1["smaller model"] -.-> READ L2["cache the prefix"] -.-> READ L3["fewer input tokens
+ small tool set"] -.-> READ L4["STREAM"] -.-> FT L5["cap max_tokens
+ concise output"] -.-> GEN L6["smaller model"] -.-> GEN classDef lever fill:#d4edda,stroke:#28a745,color:#155724; classDef stage fill:#cce5ff,stroke:#3399ff,color:#004085; class L1 lever; class L2 lever; class L3 lever; class L4 lever; class L5 lever; class L6 lever; class READ stage; class GEN stage;
CODE A low-latency call: stream, cache, cap
Stream so TTFT is low, cache the stable prefix so reads are fast, and bound the output.
# Stream → first token reaches the user almost immediately. with client.messages.stream( model="claude-haiku-4-5", # smaller = faster generation max_tokens=256, # cap output → bounded total time system=[{ "type": "text", "text": stable_instructions, # big + unchanging → cache it "cache_control": {"type": "ephemeral"}, # cached reads = faster TTFT }], tools=core_tools, # keep the set SMALL — each schema is input tokens messages=[{"role": "user", "content": user_msg}], ) as stream: for text in stream.text_stream: print(text, end="", flush=True) # render tokens live
TRADE-OFFS Fast vs. thorough
✅ Optimizing for latency
- Streaming makes the UI feel instant
- Smaller model + cached prefix cut real time
- Capped, concise output finishes sooner
⚠️ The cost of over-optimizing
- Too-small a model may need a retry (slower overall)
- Over-tight
max_tokenstruncates the answer - Streaming complicates error handling mid-response
EXAM TRAPS
PRACTICE
Q1 · Which levers lower TTFT vs. total generation time?
TTFT: smaller model, cache the prefix, fewer input tokens, small tool set, and (perceived) streaming. Total generation time: smaller model, and capping/constraining output length so fewer output tokens are produced.
Q2 · Why does streaming help even though the model isn't faster?
It cuts perceived latency: the user sees the first tokens the moment generation starts rather than waiting for the entire response, so the interface feels responsive even if total time is unchanged.
Q3 · How does a large tool set hurt latency?
Tool definitions are input tokens the model must read on every request, raising TTFT; a big list also degrades tool selection. Keep the exposed set small (and pair with prompt caching if the tool list is stable).