Home › Key concepts › Serving: batching, KV cache, latency vs throughput

Key concept · Run (advanced) · the same on every platform

Serving: batching, KV cache, latency vs throughput

How to answer many requests fast and cheap on GPUs.

What it is

Batching processes many requests together to keep the GPU busy. The KV cache stores work already done on the context so each new token is quick. Latency is how long one user waits; throughput is how many tokens per second the server produces in total. You trade one for the other.

Why it matters

Serving choices often change cost per token by five to ten times.

An example

vLLM's continuous batching serves 40 users on one GPU, each seeing the first words in under a second.

The common mistake

Measuring only one request on an idle server and calling it the speed.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Model hosting on GPUsrun open-weight models yourself, fast and cheap per tokenSageMaker endpoints / EKS + vLLMFoundry managed compute / AKS + vLLMVertex AI endpoints / GKE + vLLMProvisioned throughput servingvLLM

Related ideas

Where it shows up

Part of the Run level on these paths: LLM Engineering (serving models).