Home › Key concepts › Quantisation

Key concept · Walk (intermediate) · the same on every platform

Quantisation

Store a model's numbers with fewer bits so it runs smaller and faster.

What it is

Model weights are usually 16-bit numbers. Quantisation stores them in 8 or 4 bits. The model uses a half or a quarter of the memory and runs faster, with a small loss of quality you should measure.

Why it matters

It lets good open models run on cheaper hardware, even laptops.

An example

A model that needs 28 GB in 16-bit fits in about 8 GB at 4-bit.

The common mistake

Quantising and never checking quality on your own task.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Model hosting on GPUsrun open-weight models yourself, fast and cheap per tokenSageMaker endpoints / EKS + vLLMFoundry managed compute / AKS + vLLMVertex AI endpoints / GKE + vLLMProvisioned throughput servingvLLM

Related ideas

Where it shows up

Part of the Walk level on these paths: LLM Engineering (serving models).