cleandataengineering.aiAgent Harness, drawn simply
You are here: Tech stack › L2 · Model invocation

Agent Harness Tutorial · Tech stack

L2 · Model invocation

Every model call goes through one door. Family: Think · one turn, in order.

AZURE OPTIONSAPIM AI gateway✓ used: one governed doorFoundry Models◐ planned: GPT, Claude, nanoAzure Model router✗ dropped: not deterministicAPIM backend pools◐ planned: failover, tier 10OUR CHOICEAPIM AI gatewayConsumption, Central IndiaGroq gpt-oss-120bbehind APIM todayPydantic AImodel layer onlyOPEN-SOURCE OPTIONSPydantic AI✓ used: only model_requestLiteLLM proxy· considered: not pickedvLLM on AKS· considered: not pickedWHO DOES THE WORKAzure gives 60% · we code once 30% · each agent team still does 10%
Dark boxes are what we run. Solid boxes are used or planned; dashed boxes were considered or dropped (crossed). The bar: grey is what Azure gives, dark is what we code once, hatched is what each agent team writes.
Why our choiceOne door for allow-list, limits, token metrics and Content Safety; the agent never holds the provider key. The model per job is picked offline by the golden set.
OptionWhere fromVerdictWhy
APIM AI gatewayours✓ chosenConsumption, Central India
Groq gpt-oss-120bours✓ chosenbehind APIM today
Pydantic AIours✓ chosenmodel layer only
APIM AI gatewayAzure✓ usedone governed door
Foundry ModelsAzure◐ plannedGPT, Claude, nano
Azure Model routerAzure✗ droppednot deterministic
APIM backend poolsAzure◐ plannedfailover, tier 10
Pydantic AIopen source✓ usedonly model_request
LiteLLM proxyopen source· considerednot picked
vLLM on AKSopen source· considerednot picked

How it works inside the harness: Model call.

Compared: How do agents reach models, and who hosts them?

APIM AI gatewayAzurechosenLiteLLM proxyopen sourceexit pathPortkey gatewayvendornot evaluatedDirect SDK fro…open sourcerejectedGroqvendordev backendFoundry ModelsAzureplannedAzure OpenAIAzurenot default
Dark: what we chose. Crossed and dashed: rejected or not for us. From the Series.
OptionFromVerdictRoleLimits and token metricsContent screeningFailoverLock-inStatus here
APIM AI gatewayAzurechosenone door for every model callrate limit, token metriccalls Content Safetybackend pools (planned)medium: policies are APIM configbuilt (Consumption tier)
LiteLLM proxyopen sourceexit pathOpenAI-style proxy, many providersbudgets, rate limits, spendvia guardrail hooksfallbacks between modelslow: open sourcethe gateway in the open stack
Portkey gatewayvendornot evaluatedgateway, open core plus SaaSlimits, cost logsvia guardrail plug-insfallbacks, retriesmedium: SaaS featuresnamed only
Direct SDK from each agentopen sourcerejectedeach agent calls the providernone in one placenone in one placeeach agent codes its ownprovider keys in every agentrejected
Groqvendordev backendhosts open-weight modelsGroq's own rate limitsnone: APIM adds itsingle backendlow: OpenAI-compatible APIbuilt: gpt-oss-120b
Foundry ModelsAzureplannedGPT, Claude, small modelsvia APIMbuilt-in content filtersvia APIM backend poolslow: one catalogue, many modelsrecommended (Phase 2)
Azure OpenAIAzurenot defaultGPT deployments on Azurevia APIMbuilt-in content filtersvia APIM backend poolshigh: re-tune prompts, evalsClaude in Foundry preferred
DecisionEvery model call goes through the APIM AI gateway, today to Groq and later to Foundry Models, so limits, metrics and screening live in one place. LiteLLM is the gateway we would use off Azure; business logic stays out of the gateway so that move stays cheap.

Compared: What decides the route, and which model answers?

Rules (regex rout…ourschosenLocal classifier …open sourceplannedSmall LLM (gpt-5-…vendorplannedModelSelector (go…oursplannedAzure Model routerAzurerejectedfastText / Azure …vendorrejected
Dark: what we chose. Crossed and dashed: rejected or not for us. From the Series.
OptionFromVerdictDecidesWhenNetwork callSame input, same answerStatus
Rules (regex router)ourschosenroute: refuse, hand off, direct, loopevery turn, before any modelnoneyesruns today (layer 1)
Local classifier (Model2Vec + sklearn)open sourceplannedthe intent when rules are unsureevery message, about 1 msnone: a versioned fileyesTier 6; needs ~500 labels
Small LLM (gpt-5-nano labels)vendorplannedroute when classifier unsurerarelyone small callnot guaranteedTier 6
ModelSelector (golden set)oursplannedwhich model each job usesoffline, before each releaseoffline onlyyes: same cases, same verdictplanned: a CI job
Azure Model routerAzurerejectedwhich model answers a promptrun time, every callevery callnot guaranteednot used: model is Groq
fastText / Azure CLUvendorrejectedthe intentevery messageCLU: yesyesarchived; CLU retiring 2029
DecisionRules route every message today; a local classifier, then a small LLM only when unsure, are planned for Tier 6. Model choice is made offline by the ModelSelector on the golden set; Azure Model router is not used because its choice is not reproducible.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

  model:
    adapter: apim-ai-gateway
    base_url: {env: APIM_GROQ_BASE_URL}
    api_key: {key_vault: apim-subscription-key}
    <!-- To Groq, with the key from Key Vault; the agent's subscription key stays here. -->
    <set-backend-service backend-id="groq" />
    <set-header name="Authorization" exists-action="override"><value>Bearer {{groq-api-key}}</value></set-header>
    <set-header name="Ocp-Apim-Subscription-Key" exists-action="delete" />