AI Universe · Compute & infrastructure

Compute & infrastructure zoom 2

Where does it run? 4 concept buckets. Each one is a job you can learn once; underneath it, who does that job on each platform.

Concept

Chips

The hardware that does the maths: thousands of small calculations at once.

GPUsTPUs and other AI chipsmemory size and bandwidth

Physical: who does this job, where

Open source—
AzureND-series GPU VMs
AWSTrainium, Inferentia, GPU instances
GCPTPUs, GPU VMs
VendorsNVIDIA, AMD

Concept

Clusters and cloud

Many machines rented and wired together for training or serving.

GPU clustersjob schedulingnetworking between GPUs

Physical: who does this job, where

Open sourceKubernetes, Ray, Slurm
AzureAzure ML compute, AKS
AWSSageMaker HyperPod, EKS
GCPVertex AI training, GKE
VendorsCoreWeave, Lambda, Modal

Concept

Serving engine

Software that loads a model and answers requests fast and cheaply.

batchingKV cachequantisation

Physical: who does this job, where

Open sourcevLLM, SGLang, llama.cpp, Ollama
AzureManaged online endpoints
AWSSageMaker endpoints
GCPVertex AI endpoints
VendorsNVIDIA TensorRT-LLM / NIM

Concept

On the device

Running a smaller model on a phone or laptop instead of a data centre.

small modelsquantised filesprivacy and offline use

Physical: who does this job, where

Open sourcellama.cpp, MLX, ONNX Runtime
AzurePhi models, Foundry Local
AWS—
GCPGemini Nano
VendorsApple Intelligence

Zoom in further

Data →

Compute & infrastructureDataModelModel accessThe AI application (agent harness)ExperienceAssurance & operations