Home › Key concepts › Caching

Key concept · Walk (intermediate) · the same on every platform

Caching

Keep a copy of a result so you don't compute it again.

What it is

A cache stores recent results close to where they are needed. It speeds things up and cuts cost, but a cached answer can be out of date, so every cache needs a rule for when to expire.

Why it matters

The fastest, cheapest call is the one you don't make. This applies to web pages, queries and model calls alike.

An example

The same 'what is my policy excess?' answer is served from cache for an hour instead of calling the model each time.

The common mistake

Caching without expiry, then serving yesterday's price.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Run an appsomewhere for your code to live and answer requestsLambda / ECS FargateContainer Apps / FunctionsCloud RunDatabricks AppsDocker
Model gatewayone door to many models, with budgets and fallbacksBedrock cross-region inference + API GatewayAPI Management AI gatewayApigeeMosaic AI GatewayLiteLLM

Related ideas

Where it shows up

Part of the Walk level on these paths: LLM Engineering (serving models) · Software Engineering (building services).