AI Universe · Model access

Model access zoom 2

How does the app reach a model? 4 concept buckets. Each one is a job you can learn once; underneath it, who does that job on each platform.

Concept

Inference API

The request–response door to a model: messages in, text or tool calls out.

chat-completions shapestreamingtool calling

Physical: who does this job, where

Open sourceOpenAI-compatible servers (vLLM, Ollama)
AzureFoundry Models / Azure OpenAI
AWSBedrock
GCPVertex AI
VendorsOpenAI, Anthropic, Groq APIs

Concept

Gateway

One governed door for every model call: keys, limits, logging, screening.

allow-listrate and token limitskey managementfailover

Physical: who does this job, where

Open sourceLiteLLM
AzureAPI Management AI gateway
AWSBedrock + API Gateway
GCPApigee
VendorsPortkey, Kong, Cloudflare AI Gateway

Concept

Routing and model choice

Deciding which model answers which request.

rules firsta router modelcheapest model that passes

Physical: who does this job, where

Open sourceRouteLLM, your own rules
AzureFoundry model router
AWSBedrock intelligent prompt routing
GCP—
VendorsOpenRouter, Martian

Concept

Caching

Not paying twice for the same work.

prompt (prefix) cachingsemantic caching

Physical: who does this job, where

Open sourceRedis, GPTCache
AzureAPIM semantic cache
AWSBedrock prompt caching
GCPVertex context caching
Vendorsprovider prompt caching

Zoom in further

← Model · The AI application (agent harness) →

Compute & infrastructureDataModelModel accessThe AI application (agent harness)ExperienceAssurance & operations