D5 · Context & Reliability › Cluster 7 · Cost & Performance Optimization

7.3 · Reducing Latency

Make it feel fast

CONCEPT Two clocks: time-to-first-token and total time

Latency has two parts. Time-to-first-token (TTFT) — how long before anything appears — is dominated by how many input tokens must be read before generation starts. Total time is TTFT plus generation, and generation time scales with how many output tokens the model produces. The levers below attack one clock or the other. Note the Batch API (7.2) is the opposite trade — it minimizes cost by accepting high latency, so none of this applies to batched work.

LeverAttacksWhy it helps
Faster / smaller modelTTFT + totalSmaller models generate faster; Haiku < Sonnet < Opus in latency
Stream the responsePerceived TTFTUser sees tokens as they arrive, not after the full reply
Cache the prompt prefixTTFTCached tokens are read far faster than fresh ones
Minimize input tokensTTFTFewer tokens to read before the first output token
Constrain output lengthTotal timeFewer output tokens = less generation time
Keep the tool set smallTTFTTool schemas are input tokens the model reads every call
🔑 Streaming is the highest-leverage move for perceived latency. It doesn't make the model faster, but the user sees the first words almost immediately instead of staring at a spinner until the whole answer lands.

CONCEPT Cut the tokens the model has to read and write

Right-size the model. A smaller model is faster end-to-end; if Haiku clears the task, it's both cheaper (7.1) and quicker. Shrink the input: trim boilerplate, don't dump giant tool results into context, and put a large stable prefix behind cache_control so repeat calls skip re-reading it. Keep the tool set small — every tool definition is input tokens read on every request, and a bloated tool list slows TTFT and muddies tool selection. Cap the output: set a sensible max_tokens and ask for concise, structured answers so the model doesn't generate a paragraph where a label will do.

Pair the levers: a right-sized model + cached prefix + streamed, length-capped output is fast on both clocks and cheap at the same time.

DIAGRAM Where each lever lands

flowchart TD REQ["Request sent"] --> READ["Read input tokens
(prompt + tools)"] READ --> FT(["First token — TTFT"]) FT --> GEN["Generate output tokens"] GEN --> DONE(["Full response"]) L1["smaller model"] -.-> READ L2["cache the prefix"] -.-> READ L3["fewer input tokens
+ small tool set"] -.-> READ L4["STREAM"] -.-> FT L5["cap max_tokens
+ concise output"] -.-> GEN L6["smaller model"] -.-> GEN classDef lever fill:#d4edda,stroke:#28a745,color:#155724; classDef stage fill:#cce5ff,stroke:#3399ff,color:#004085; class L1 lever; class L2 lever; class L3 lever; class L4 lever; class L5 lever; class L6 lever; class READ stage; class GEN stage;

CODE A low-latency call: stream, cache, cap

Stream so TTFT is low, cache the stable prefix so reads are fast, and bound the output.

# Stream → first token reaches the user almost immediately.
with client.messages.stream(
    model="claude-haiku-4-5",            # smaller = faster generation
    max_tokens=256,                       # cap output → bounded total time
    system=[{
        "type": "text",
        "text": stable_instructions,      # big + unchanging → cache it
        "cache_control": {"type": "ephemeral"},  # cached reads = faster TTFT
    }],
    tools=core_tools,                    # keep the set SMALL — each schema is input tokens
    messages=[{"role": "user", "content": user_msg}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)   # render tokens live

TRADE-OFFS Fast vs. thorough

✅ Optimizing for latency

  • Streaming makes the UI feel instant
  • Smaller model + cached prefix cut real time
  • Capped, concise output finishes sooner

⚠️ The cost of over-optimizing

  • Too-small a model may need a retry (slower overall)
  • Over-tight max_tokens truncates the answer
  • Streaming complicates error handling mid-response
🔑 The lowest-latency correct answer wins. A fast wrong answer that forces a re-ask is slower than getting it right once.

EXAM TRAPS

Trap: "Streaming makes the model generate faster." → It does not change total generation time. It lowers perceived latency by showing the first tokens immediately instead of after the full reply.
Trap: Only tuning output size. → TTFT is driven by input tokens — prompt length and tool schemas. Trimming input and caching the prefix speeds up the first token; capping output speeds up the total.
Trap: Loading every possible tool "just in case." → Each tool definition is input the model reads every call, hurting TTFT and tool-selection quality. Keep the set small and focused.
Trap: Using the Batch API to reduce latency. → Batch increases latency (async, up to 24h) to cut cost. It's the opposite lever — never a latency fix.

PRACTICE

Q1 · Which levers lower TTFT vs. total generation time?

TTFT: smaller model, cache the prefix, fewer input tokens, small tool set, and (perceived) streaming. Total generation time: smaller model, and capping/constraining output length so fewer output tokens are produced.

Q2 · Why does streaming help even though the model isn't faster?

It cuts perceived latency: the user sees the first tokens the moment generation starts rather than waiting for the entire response, so the interface feels responsive even if total time is unchanged.

Q3 · How does a large tool set hurt latency?

Tool definitions are input tokens the model must read on every request, raising TTFT; a big list also degrades tool selection. Keep the exposed set small (and pair with prompt caching if the tool list is stable).