AI Universe · Compute & infrastructure
Compute & infrastructure zoom 2
Where does it run? 4 concept buckets. Each one is a job you can learn once; underneath it, who does that job on each platform.
Concept
Chips
The hardware that does the maths: thousands of small calculations at once.
GPUsTPUs and other AI chipsmemory size and bandwidth
Physical: who does this job, where
| Open source | — |
|---|---|
| Azure | ND-series GPU VMs |
| AWS | Trainium, Inferentia, GPU instances |
| GCP | TPUs, GPU VMs |
| Vendors | NVIDIA, AMD |
Concept
Clusters and cloud
Many machines rented and wired together for training or serving.
GPU clustersjob schedulingnetworking between GPUs
Physical: who does this job, where
| Open source | Kubernetes, Ray, Slurm |
|---|---|
| Azure | Azure ML compute, AKS |
| AWS | SageMaker HyperPod, EKS |
| GCP | Vertex AI training, GKE |
| Vendors | CoreWeave, Lambda, Modal |
Concept
Serving engine
Software that loads a model and answers requests fast and cheaply.
batchingKV cachequantisation
Physical: who does this job, where
| Open source | vLLM, SGLang, llama.cpp, Ollama |
|---|---|
| Azure | Managed online endpoints |
| AWS | SageMaker endpoints |
| GCP | Vertex AI endpoints |
| Vendors | NVIDIA TensorRT-LLM / NIM |
Concept
On the device
Running a smaller model on a phone or laptop instead of a data centre.
small modelsquantised filesprivacy and offline use
Physical: who does this job, where
| Open source | llama.cpp, MLX, ONNX Runtime |
|---|---|
| Azure | Phi models, Foundry Local |
| AWS | — |
| GCP | Gemini Nano |
| Vendors | Apple Intelligence |