Home › Key concepts › LLM cost: tokens, caching and routing
Key concept · Walk (intermediate) · the same on every platform
LLM cost: tokens, caching and routing
Know the cost per answer, then cut it.
What it is
Cost is tokens in plus tokens out, times the model's price. You cut it by sending less context, caching repeated prompts, routing easy requests to smaller models, and answering in code when you can.
Why it matters
Costs that look tiny per call become large at scale. Knowing the cost per answer is part of the design.
An example
Routing 70% of questions to a small model and caching the system prompt cuts monthly cost by 60% with no drop in evaluation scores.
The common mistake
Using the largest model for everything because it was the default.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Model gatewayone door to many models, with budgets and fallbacks | Bedrock cross-region inference + API Gateway | API Management AI gateway | Apigee | Mosaic AI Gateway | LiteLLM |
| Model APIcall a large language model | Amazon Bedrock | Azure OpenAI in Microsoft Foundry | Gemini on Vertex AI | Foundation Model APIs | Ollama, or a free hosted open model |
Related ideas
Where it shows up
Part of the Walk level on these paths: AI Engineering (safety and proof) · LLM Engineering (serving models).