Home › Key concepts › LLM cost: tokens, caching and routing

Key concept · Walk (intermediate) · the same on every platform

LLM cost: tokens, caching and routing

Know the cost per answer, then cut it.

What it is

Cost is tokens in plus tokens out, times the model's price. You cut it by sending less context, caching repeated prompts, routing easy requests to smaller models, and answering in code when you can.

Why it matters

Costs that look tiny per call become large at scale. Knowing the cost per answer is part of the design.

An example

Routing 70% of questions to a small model and caching the system prompt cuts monthly cost by 60% with no drop in evaluation scores.

The common mistake

Using the largest model for everything because it was the default.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Model gatewayone door to many models, with budgets and fallbacksBedrock cross-region inference + API GatewayAPI Management AI gatewayApigeeMosaic AI GatewayLiteLLM
Model APIcall a large language modelAmazon BedrockAzure OpenAI in Microsoft FoundryGemini on Vertex AIFoundation Model APIsOllama, or a free hosted open model

Related ideas

Where it shows up

Part of the Walk level on these paths: AI Engineering (safety and proof) · LLM Engineering (serving models).