Home › AI gateways
AI · Concepts
AI gateways: which one, where it runs, what it costs
An AI gateway is a control point in front of model calls: one address, one set of keys, limits, usage records, logs and checks. Organisations often end up with several (a cloud's, a data platform's, an open-source one), and the same model reachable through all of them. This page gives a way to choose, covers guardrails and cost, and treats developer tools such as coding assistants separately from your own agents.
Written 19 November 2026. Vendor features and prices come from the authors' knowledge and product help, not from testing every product, and they change often. The last section lists what to check before relying on it.
The short answer
You are making three separate decisions, in this order. Do not let one decide the others.
The gateways compared
They do the same job: take a chat request, apply limits and checks, forward it, record usage. They differ in who runs them, how you sign in, how policy is written and where the logs go. A policy written for one is not portable to another.
| Gateway | Who runs it | Where you configure it | How callers sign in | Which models | Usage and logs | Guardrails | Fits |
|---|---|---|---|---|---|---|---|
| Databricks AI Gateway | Databricks, managed | Settings on a model endpoint (UI, API or CLI) | Workspace token or OAuth | Databricks-hosted models, plus external models you register | Databricks system tables and log tables in Delta | Built-in safety and PII options on supported endpoints | Agents whose models and data live in Databricks |
| Azure API Management (AI gateway features) | Azure, managed | Policies on an API, written in XML | Subscription key in a header, plus Entra sign-in | Whatever backend you point it at | Application Insights and Log Analytics | Azure AI Content Safety, called from a policy | Apps that run on Azure |
| AWS AgentCore Gateway and Bedrock | AWS, managed | Gateway targets and guardrail settings | AWS identity (signed requests) or a token | Bedrock models and OpenAI-compatible providers | CloudWatch | Bedrock Guardrails | Apps that run on AWS |
| LiteLLM proxy | You host it | One YAML file | Virtual keys it issues, with budgets | Over a hundred providers | Its own database, or Langfuse and similar | Integrations with outside guardrail tools | Laptops, prototypes, developer tools, and teams with no enterprise gateway |
| Kong AI Gateway | You host it, or Kong's cloud | Plugins in declarative config | Key, OAuth or OIDC plugins | Several providers through its AI plugin | Kong's log and metrics plugins | Prompt-guard and PII plugins; can call cloud guardrail services (some in paid tiers) | A company that already runs Kong and wants one policy across clouds |
What decides it
| Factor | Why it matters |
|---|---|
| Who owns policy and the audit trail | If a security or platform team already owns a central gateway, model traffic goes through it, even if another would be neater. |
| Where the caller and the model sit | Databricks agent to a Databricks-hosted model, Azure app to Azure's model service, or an app outside both. |
| Where identity lives | Entra passthrough, workspace identity, or a key. |
| Who has to run it | A managed service costs less in people than one you patch and keep available yourself. |
| Data terms | Region, retention, zero-retention terms, private networking, and whether the provider is already an approved subprocessor. |
| Existing commitments | A spend commitment or negotiated discount on one cloud can make one route much cheaper. |
| Feature parity | A model's newest features (prompt caching, extended thinking, batch, structured output) tend to arrive first on the provider's own API and can lag on hosted versions. |
| Built-in guardrails | A tie-breaker, not the decider: see below. |
A default that works for most organisations
- Enterprise apps and agents calling external or cross-cloud models: the existing enterprise gateway, whichever the security team owns. Egress rules, data-loss checks and quotas belong there.
- Agents inside Databricks calling Databricks-hosted models: Databricks AI Gateway, for per-endpoint limits, guardrails and cost in system tables.
- Apps on Azure calling Azure's model service: Azure API Management.
- Laptops and prototypes: LiteLLM.
Guardrails: built in, and what they cannot do
Each route offers some: a safety filter and PII options (Databricks), Azure AI Content Safety with jailbreak and groundedness checks (Azure), Bedrock Guardrails (AWS), plugins and integrations (Kong, LiteLLM), and the safety training built into the model. Treat them as a tie-breaker, not the decider.
- Generic checks miss domain rules. A gateway cannot know that a drug-safety draft's statistic must equal what a tool returned. Those checks live in the agent harness, on every cloud.
- False positives are real. A generic safety filter can block legitimate text in some fields. Pharmacovigilance text routinely contains terms such as suicidal ideation. Test any filter on your own text; a filter that blocks work gets switched off.
- Compare: categories covered; block, flag or mask modes; whether tool outputs and retrieved text are checked for hidden instructions and not only the user's prompt; false-positive rate on your data; whether every decision is logged; added latency and per-call cost.
What it costs
You pay in layers: the tokens (set by the host), the gateway (a tier fee, a licence, or hosting plus people), and logging and network transfer. A gateway in front of a host does not change the host's token price.
| Item | Cost shape |
|---|---|
| Databricks AI Gateway | No separate gateway charge that we know of. You pay for model usage, plus storage and compute for any log tables. |
| Azure API Management | Priced per tier, from cheap per-call tiers to thousands of dollars a month at the top. The cheapest tier lacks token-limit and per-key rate-limit policies, so the tier you need decides the cost. |
| LiteLLM | The software is free and adds no per-token markup. You pay for the containers, a database, high availability and logs, then people's time to run, patch and secure it. An enterprise licence exists for features such as single sign-on and stronger audit, by quote. |
| Kong | Open source is free. Enterprise and cloud versions are licensed, typically a large annual figure. You also pay hosting and operations. |
| The model tokens | Usually the largest cost, and it does not change with the gateway. The host sets the per-token price, and the same model can cost differently on different hosts. |
These are shapes, not quotes. Get a price for your expected volume from each route before you decide.
Developer tools are a separate problem
Coding assistants and chat tools carry your source code, so the same question comes up for them. There are two kinds of traffic.
- Tools that call a model API and let you set the endpoint and key, or use a cloud provider's model service (for example Claude Code and the OpenAI command-line tools). These can go through a gateway.
- Tools that call their own vendor's backend: ChatGPT and Claude in the browser, and most editor assistants. A gateway cannot sit in front of them. You control them with the vendor's enterprise plan (single sign-on, retention and no-training terms), network and data-loss controls, and a written list of approved tools.
Which gateway for coding traffic
| Option | Where it runs | Good when |
|---|---|---|
| LiteLLM proxy | Containers in your own cloud account with a small database, behind your network and single sign-on | No enterprise AI gateway yet, or you want per-developer keys and budgets quickly. It accepts Anthropic's request format, so streaming, tool calls and prompt caching pass through. |
| Kong or Azure API Management | Wherever your enterprise gateway already runs | One is already the approved door. It must pass the provider's request format and streaming through unchanged, so test that first. |
| The model through your cloud (Bedrock, Vertex, or Microsoft's Foundry, where offered) | Nothing new to install; the tool uses your cloud identity | You already have a cloud agreement. Spend, limits and logs come from the cloud's tools. |
| Databricks AI Gateway | Part of Databricks | Not a first choice for developer coding traffic; it fits agents inside Databricks. |
What it costs, and what to protect
- The model tokens are the big cost and the gateway does not change them. Heavy daily coding use can run to tens or hundreds of dollars per developer per month.
- Running LiteLLM: a small cloud bill (containers, a database, logs), plausibly one to a few hundred dollars a month, rising with traffic. This is an estimate. The larger cost is the people who run, patch, secure and keep it available.
- Check how developers are billed. On seat subscriptions the traffic goes to the vendor under each person's sign-in, and a gateway cannot meter it per developer the way it does for API keys. Using a gateway usually means moving to API-key billing. Compare the two on real usage.
- Protect the logs. Prompts contain source code. Restrict who can read the gateway's logs, set a short retention, and keep developer traffic in its own namespace with its own keys, budgets and retention, apart from production agents.
- Confirm your vendor terms say API and commercial inputs are not used for training.
Questions
Choose the model by running your own test set. Then choose the route by contract, data terms, feature parity and capacity. Then choose the gateway by who enforces policy. Re-run the test set on each route, because the host or gateway can add filters or change limits.
Should I stack two gateways?Avoid it. Two gateways on one path doubles latency, duplicates limits and splits the logs, so nobody can say which one blocked a call. If a central gateway must be the door, the platform gateway behind it should observe and pass through.
Which gateway do we use for our own agents?On Databricks we use the Databricks AI Gateway, which is attached to the hosted model endpoint with usage tracking on by default; we configure nothing else on it. On Azure the design uses API Management. On a laptop, LiteLLM. The harness hides the difference behind one model port.
Does a gateway replace our own guardrails?No. A gateway can't know that a drug-safety draft's statistic must equal what a tool returned, or that a draft must never say a drug causes an event. Those checks stay in the harness.
Check before you rely on this
- Each product's current guardrail features and which tier includes them.
- Current prices, from each vendor, for your expected volume.
- Whether each developer tool you use supports a custom endpoint today; test one small team first.
- That a gateway passes streaming, tool calls and prompt caching through for your model.
Where to go next
- The agent harness on Databricks: how a real agent uses a gateway and handles guardrails, concern by concern.
- Build an agent on Databricks: each Databricks AI service explained.
- AI governance: running AI you can trust.
- Agent guardrails.