Home › Key concepts › Serving: batching, KV cache, latency vs throughput
Key concept · Run (advanced) · the same on every platform
Serving: batching, KV cache, latency vs throughput
How to answer many requests fast and cheap on GPUs.
What it is
Batching processes many requests together to keep the GPU busy. The KV cache stores work already done on the context so each new token is quick. Latency is how long one user waits; throughput is how many tokens per second the server produces in total. You trade one for the other.
Why it matters
Serving choices often change cost per token by five to ten times.
An example
vLLM's continuous batching serves 40 users on one GPU, each seeing the first words in under a second.
The common mistake
Measuring only one request on an idle server and calling it the speed.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Model hosting on GPUsrun open-weight models yourself, fast and cheap per token | SageMaker endpoints / EKS + vLLM | Foundry managed compute / AKS + vLLM | Vertex AI endpoints / GKE + vLLM | Provisioned throughput serving | vLLM |
Related ideas
Where it shows up
Part of the Run level on these paths: LLM Engineering (serving models).