Agent Harness Tutorial · Tech stack
L2 · Model invocation
Every model call goes through one door. Family: Think · one turn, in order.
Why our choiceOne door for allow-list, limits, token metrics and Content Safety; the agent never holds the provider key. The model per job is picked offline by the golden set.
| Option | Where from | Verdict | Why |
|---|---|---|---|
| APIM AI gateway | ours | ✓ chosen | Consumption, Central India |
| Groq gpt-oss-120b | ours | ✓ chosen | behind APIM today |
| Pydantic AI | ours | ✓ chosen | model layer only |
| APIM AI gateway | Azure | ✓ used | one governed door |
| Foundry Models | Azure | ◐ planned | GPT, Claude, nano |
| Azure Model router | Azure | ✗ dropped | not deterministic |
| APIM backend pools | Azure | ◐ planned | failover, tier 10 |
| Pydantic AI | open source | ✓ used | only model_request |
| LiteLLM proxy | open source | · considered | not picked |
| vLLM on AKS | open source | · considered | not picked |
How it works inside the harness: Model call.
Compared: How do agents reach models, and who hosts them?
| Option | From | Verdict | Role | Limits and token metrics | Content screening | Failover | Lock-in | Status here |
|---|---|---|---|---|---|---|---|---|
| APIM AI gateway | Azure | chosen | one door for every model call | rate limit, token metric | calls Content Safety | backend pools (planned) | medium: policies are APIM config | built (Consumption tier) |
| LiteLLM proxy | open source | exit path | OpenAI-style proxy, many providers | budgets, rate limits, spend | via guardrail hooks | fallbacks between models | low: open source | the gateway in the open stack |
| Portkey gateway | vendor | not evaluated | gateway, open core plus SaaS | limits, cost logs | via guardrail plug-ins | fallbacks, retries | medium: SaaS features | named only |
| Direct SDK from each agent | open source | rejected | each agent calls the provider | none in one place | none in one place | each agent codes its own | provider keys in every agent | rejected |
| Groq | vendor | dev backend | hosts open-weight models | Groq's own rate limits | none: APIM adds it | single backend | low: OpenAI-compatible API | built: gpt-oss-120b |
| Foundry Models | Azure | planned | GPT, Claude, small models | via APIM | built-in content filters | via APIM backend pools | low: one catalogue, many models | recommended (Phase 2) |
| Azure OpenAI | Azure | not default | GPT deployments on Azure | via APIM | built-in content filters | via APIM backend pools | high: re-tune prompts, evals | Claude in Foundry preferred |
DecisionEvery model call goes through the APIM AI gateway, today to Groq and later to Foundry Models, so limits, metrics and screening live in one place. LiteLLM is the gateway we would use off Azure; business logic stays out of the gateway so that move stays cheap.
Compared: What decides the route, and which model answers?
| Option | From | Verdict | Decides | When | Network call | Same input, same answer | Status |
|---|---|---|---|---|---|---|---|
| Rules (regex router) | ours | chosen | route: refuse, hand off, direct, loop | every turn, before any model | none | yes | runs today (layer 1) |
| Local classifier (Model2Vec + sklearn) | open source | planned | the intent when rules are unsure | every message, about 1 ms | none: a versioned file | yes | Tier 6; needs ~500 labels |
| Small LLM (gpt-5-nano labels) | vendor | planned | route when classifier unsure | rarely | one small call | not guaranteed | Tier 6 |
| ModelSelector (golden set) | ours | planned | which model each job uses | offline, before each release | offline only | yes: same cases, same verdict | planned: a CI job |
| Azure Model router | Azure | rejected | which model answers a prompt | run time, every call | every call | not guaranteed | not used: model is Groq |
| fastText / Azure CLU | vendor | rejected | the intent | every message | CLU: yes | yes | archived; CLU retiring 2029 |
DecisionRules route every message today; a local classifier, then a small LLM only when unsure, are planned for Tier 6. Model choice is made offline by the ModelSelector on the golden set; Azure Model router is not used because its choice is not reproducible.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
model:
adapter: apim-ai-gateway
base_url: {env: APIM_GROQ_BASE_URL}
api_key: {key_vault: apim-subscription-key} <!-- To Groq, with the key from Key Vault; the agent's subscription key stays here. -->
<set-backend-service backend-id="groq" />
<set-header name="Authorization" exists-action="override"><value>Bearer {{groq-api-key}}</value></set-header>
<set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />