Home › Key concepts › Caching
Key concept · Walk (intermediate) · the same on every platform
Caching
Keep a copy of a result so you don't compute it again.
What it is
A cache stores recent results close to where they are needed. It speeds things up and cuts cost, but a cached answer can be out of date, so every cache needs a rule for when to expire.
Why it matters
The fastest, cheapest call is the one you don't make. This applies to web pages, queries and model calls alike.
An example
The same 'what is my policy excess?' answer is served from cache for an hour instead of calling the model each time.
The common mistake
Caching without expiry, then serving yesterday's price.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Run an appsomewhere for your code to live and answer requests | Lambda / ECS Fargate | Container Apps / Functions | Cloud Run | Databricks Apps | Docker |
| Model gatewayone door to many models, with budgets and fallbacks | Bedrock cross-region inference + API Gateway | API Management AI gateway | Apigee | Mosaic AI Gateway | LiteLLM |
Related ideas
Where it shows up
Part of the Walk level on these paths: LLM Engineering (serving models) · Software Engineering (building services).