Home › AI gateways

AI · Concepts

AI gateways: which one, where it runs, what it costs

An AI gateway is a control point in front of model calls: one address, one set of keys, limits, usage records, logs and checks. Organisations often end up with several (a cloud's, a data platform's, an open-source one), and the same model reachable through all of them. This page gives a way to choose, covers guardrails and cost, and treats developer tools such as coding assistants separately from your own agents.

Written 19 November 2026. Vendor features and prices come from the authors' knowledge and product help, not from testing every product, and they change often. The last section lists what to check before relying on it.

The short answer

You are making three separate decisions, in this order. Do not let one decide the others.

1 · The modelChoose by running your own test set on each candidate: accuracy, tool use, cost per task, speed. Not by platform.
2 · The routeChoose the host by contract, data terms, feature parity, capacity and existing commitments. This sets who bills the tokens.
3 · The gatewayChoose by who enforces policy and owns the audit trail. Use one enforcement point per path.

The gateways compared

They do the same job: take a chat request, apply limits and checks, forward it, record usage. They differ in who runs them, how you sign in, how policy is written and where the logs go. A policy written for one is not portable to another.

GatewayWho runs itWhere you configure itHow callers sign inWhich modelsUsage and logsGuardrailsFits
Databricks AI GatewayDatabricks, managedSettings on a model endpoint (UI, API or CLI)Workspace token or OAuthDatabricks-hosted models, plus external models you registerDatabricks system tables and log tables in DeltaBuilt-in safety and PII options on supported endpointsAgents whose models and data live in Databricks
Azure API Management (AI gateway features)Azure, managedPolicies on an API, written in XMLSubscription key in a header, plus Entra sign-inWhatever backend you point it atApplication Insights and Log AnalyticsAzure AI Content Safety, called from a policyApps that run on Azure
AWS AgentCore Gateway and BedrockAWS, managedGateway targets and guardrail settingsAWS identity (signed requests) or a tokenBedrock models and OpenAI-compatible providersCloudWatchBedrock GuardrailsApps that run on AWS
LiteLLM proxyYou host itOne YAML fileVirtual keys it issues, with budgetsOver a hundred providersIts own database, or Langfuse and similarIntegrations with outside guardrail toolsLaptops, prototypes, developer tools, and teams with no enterprise gateway
Kong AI GatewayYou host it, or Kong's cloudPlugins in declarative configKey, OAuth or OIDC pluginsSeveral providers through its AI pluginKong's log and metrics pluginsPrompt-guard and PII plugins; can call cloud guardrail services (some in paid tiers)A company that already runs Kong and wants one policy across clouds

What decides it

FactorWhy it matters
Who owns policy and the audit trailIf a security or platform team already owns a central gateway, model traffic goes through it, even if another would be neater.
Where the caller and the model sitDatabricks agent to a Databricks-hosted model, Azure app to Azure's model service, or an app outside both.
Where identity livesEntra passthrough, workspace identity, or a key.
Who has to run itA managed service costs less in people than one you patch and keep available yourself.
Data termsRegion, retention, zero-retention terms, private networking, and whether the provider is already an approved subprocessor.
Existing commitmentsA spend commitment or negotiated discount on one cloud can make one route much cheaper.
Feature parityA model's newest features (prompt caching, extended thinking, batch, structured output) tend to arrive first on the provider's own API and can lag on hosted versions.
Built-in guardrailsA tie-breaker, not the decider: see below.

A default that works for most organisations

One enforcement point per path. If a central gateway must be the door, the platform gateway behind it observes and passes through. Stacking enforcement doubles latency, duplicates limits and splits the logs.

Guardrails: built in, and what they cannot do

Each route offers some: a safety filter and PII options (Databricks), Azure AI Content Safety with jailbreak and groundedness checks (Azure), Bedrock Guardrails (AWS), plugins and integrations (Kong, LiteLLM), and the safety training built into the model. Treat them as a tie-breaker, not the decider.

What it costs

You pay in layers: the tokens (set by the host), the gateway (a tier fee, a licence, or hosting plus people), and logging and network transfer. A gateway in front of a host does not change the host's token price.

ItemCost shape
Databricks AI GatewayNo separate gateway charge that we know of. You pay for model usage, plus storage and compute for any log tables.
Azure API ManagementPriced per tier, from cheap per-call tiers to thousands of dollars a month at the top. The cheapest tier lacks token-limit and per-key rate-limit policies, so the tier you need decides the cost.
LiteLLMThe software is free and adds no per-token markup. You pay for the containers, a database, high availability and logs, then people's time to run, patch and secure it. An enterprise licence exists for features such as single sign-on and stronger audit, by quote.
KongOpen source is free. Enterprise and cloud versions are licensed, typically a large annual figure. You also pay hosting and operations.
The model tokensUsually the largest cost, and it does not change with the gateway. The host sets the per-token price, and the same model can cost differently on different hosts.

These are shapes, not quotes. Get a price for your expected volume from each route before you decide.

Developer tools are a separate problem

Coding assistants and chat tools carry your source code, so the same question comes up for them. There are two kinds of traffic.

Which gateway for coding traffic

OptionWhere it runsGood when
LiteLLM proxyContainers in your own cloud account with a small database, behind your network and single sign-onNo enterprise AI gateway yet, or you want per-developer keys and budgets quickly. It accepts Anthropic's request format, so streaming, tool calls and prompt caching pass through.
Kong or Azure API ManagementWherever your enterprise gateway already runsOne is already the approved door. It must pass the provider's request format and streaming through unchanged, so test that first.
The model through your cloud (Bedrock, Vertex, or Microsoft's Foundry, where offered)Nothing new to install; the tool uses your cloud identityYou already have a cloud agreement. Spend, limits and logs come from the cloud's tools.
Databricks AI GatewayPart of DatabricksNot a first choice for developer coding traffic; it fits agents inside Databricks.

What it costs, and what to protect

Questions

If the same model is on Azure, Databricks and through Kong, which do I call?

Choose the model by running your own test set. Then choose the route by contract, data terms, feature parity and capacity. Then choose the gateway by who enforces policy. Re-run the test set on each route, because the host or gateway can add filters or change limits.

Should I stack two gateways?

Avoid it. Two gateways on one path doubles latency, duplicates limits and splits the logs, so nobody can say which one blocked a call. If a central gateway must be the door, the platform gateway behind it should observe and pass through.

Which gateway do we use for our own agents?

On Databricks we use the Databricks AI Gateway, which is attached to the hosted model endpoint with usage tracking on by default; we configure nothing else on it. On Azure the design uses API Management. On a laptop, LiteLLM. The harness hides the difference behind one model port.

Does a gateway replace our own guardrails?

No. A gateway can't know that a drug-safety draft's statistic must equal what a tool returned, or that a draft must never say a drug causes an event. Those checks stay in the harness.

Check before you rely on this

Where to go next