Agent Harness Tutorial · Tech stack
Every comparison, side by side
Every technology choice in the harness, as a head-to-head table: the options, the same criteria for each, a verdict, and when we would switch. Tables from the Series say so; the others were added here and are marked "added".
- Who owns the control loop?
- What keeps a run alive while it waits days for a person?
- How do agents reach models, and who hosts them?
- What decides the route, and which model answers?
- Who provides each guardrail check?
- Which library checks and gets identity tokens?
- How does the agent call tools?
- Where does conversation state live?
- Where do knowledge passages live and get searched?
- Where do traces, metrics and alerts go?
- Which tools run and judge our evaluations?
- How do we replay a run without live calls?
- How is cost measured and capped?
- What retries and fails over when a call breaks?
- How are inputs, outputs and records typed?
- What queues turns when load spikes?
- How do config and releases change safely?
- How do agents hand work to other agents?
- Which channels and which human desk?
- How is personal data found and masked?
Compared: Who owns the control loop?
| Option | From | Verdict | Who owns the loop | Any model | Can tests reach every step | Lock-in | Observability | Best for |
|---|---|---|---|---|---|---|---|---|
| Plain Python (hand-written) | ours | chosen | us | yes | yes | none | our OTel spans | assured, transactional agents |
| Pydantic AI Agent | open source | model layer; A1–A3 | framework; we can step it | yes | yes if we step the loop | low: open source, any cloud | OTel built in | typed, testable Python agents |
| LangGraph | open source | workflow-shaped only | framework runs our graph | yes | yes, node by node | medium: graph model, LangSmith | full only with LangSmith | workflow agents with approvals |
| Claude Agent SDK | vendor | internal agents only | the SDK | Claude only | partly: the loop is hidden | high: Claude-only | partial traces | coding, ops, research agents |
| Anthropic tool runner (thin helper) | vendor | small Claude agents | us, with hooks | Claude only | yes: the steps are ours | medium: Claude-only | our OTel spans | small Claude-only agents |
| Microsoft Agent Framework | open source | .NET teams only | framework | yes | trajectory checks re-proved | medium: framework model | OTel, Azure-native | .NET or Azure-first teams |
| Foundry Agent Service (prompt agent) | Azure | rejected | Azure | Foundry models | no: loop and retries hidden | high: Azure-only service | Foundry tracing | low-code agents, not assured ones |
On its concern page: L4 · Control loop
Compared: What keeps a run alive while it waits days for a person?
| Option | From | Verdict | Where state lives | New infrastructure | Rules on our code | Approvals and long waits | Ops burden for us | Exit cost |
|---|---|---|---|---|---|---|---|---|
| DBOS | open source | chosen | our Azure PostgreSQL | none: reuses our PostgreSQL | light | durable sleep, messages, events | lowest | low: open source, plain Postgres |
| Temporal | open source | for long sagas | Temporal's store + visibility index | Temporal Cloud, or AKS cluster | strict: no clock, random, I/O | signals, updates, timers: richest | low on Cloud; high self-hosted | low in code; in-flight stays |
| Durable Task Scheduler | Azure | only if Azure-only | managed by Azure | a DTS resource | strict, like Temporal | external events, durable timers | low | high: own model, Azure-only |
| Azure Durable Functions | Azure | if already on Functions | Azure Storage, SQL or DTS | run the agent on Functions | strict replay rules | external events, durable timers | low: serverless | medium: tied to Functions |
| Logic Apps Standard | Azure | back-office only | managed by Azure | a Logic Apps resource | not code-first: designer | built-in approval emails | low | high: designer workflows |
| Restate | open source | watch | the Restate server's storage | Restate server or Cloud | durable steps via its context | durable calls, timers | another service to run, secure | medium: younger ecosystem |
| Prefect | open source | data and eval jobs | Prefect server database | Prefect server or Cloud | light: Python flows, tasks | built for batch, not people | server plus workers | medium |
| DIY on PostgreSQL | ours | simplest waits only | our PostgreSQL | none | none: it is all ours | status rows + scheduled job | we own every bug | none |
On its concern page: L14 · Human-in-the-loop
Compared: How do agents reach models, and who hosts them?
| Option | From | Verdict | Role | Limits and token metrics | Content screening | Failover | Lock-in | Status here |
|---|---|---|---|---|---|---|---|---|
| APIM AI gateway | Azure | chosen | one door for every model call | rate limit, token metric | calls Content Safety | backend pools (planned) | medium: policies are APIM config | built (Consumption tier) |
| LiteLLM proxy | open source | exit path | OpenAI-style proxy, many providers | budgets, rate limits, spend | via guardrail hooks | fallbacks between models | low: open source | the gateway in the open stack |
| Portkey gateway | vendor | not evaluated | gateway, open core plus SaaS | limits, cost logs | via guardrail plug-ins | fallbacks, retries | medium: SaaS features | named only |
| Direct SDK from each agent | open source | rejected | each agent calls the provider | none in one place | none in one place | each agent codes its own | provider keys in every agent | rejected |
| Groq | vendor | dev backend | hosts open-weight models | Groq's own rate limits | none: APIM adds it | single backend | low: OpenAI-compatible API | built: gpt-oss-120b |
| Foundry Models | Azure | planned | GPT, Claude, small models | via APIM | built-in content filters | via APIM backend pools | low: one catalogue, many models | recommended (Phase 2) |
| Azure OpenAI | Azure | not default | GPT deployments on Azure | via APIM | built-in content filters | via APIM backend pools | high: re-tune prompts, evals | Claude in Foundry preferred |
On its concern page: L2 · Model invocation
Compared: What decides the route, and which model answers?
| Option | From | Verdict | Decides | When | Network call | Same input, same answer | Status |
|---|---|---|---|---|---|---|---|
| Rules (regex router) | ours | chosen | route: refuse, hand off, direct, loop | every turn, before any model | none | yes | runs today (layer 1) |
| Local classifier (Model2Vec + sklearn) | open source | planned | the intent when rules are unsure | every message, about 1 ms | none: a versioned file | yes | Tier 6; needs ~500 labels |
| Small LLM (gpt-5-nano labels) | vendor | planned | route when classifier unsure | rarely | one small call | not guaranteed | Tier 6 |
| ModelSelector (golden set) | ours | planned | which model each job uses | offline, before each release | offline only | yes: same cases, same verdict | planned: a CI job |
| Azure Model router | Azure | rejected | which model answers a prompt | run time, every call | every call | not guaranteed | not used: model is Groq |
| fastText / Azure CLU | vendor | rejected | the intent | every message | CLU: yes | yes | archived; CLU retiring 2029 |
On its concern page: Router
Compared: Who provides each guardrail check?
| Option | From | Verdict | Kind | Where it runs | What it catches | Why |
|---|---|---|---|---|---|---|
| Our rules (harness + agent) | ours | chosen | rules in code | every position, before to reply | 30 checks, all built | every check that protects money |
| Azure Content Safety (via APIM) | Azure | chosen | service | gateway: prompt and reply | Prompt Shields, harm categories | allow-list, rate limit, harms |
| Llama Prompt Guard 2 | open source | to build | small classifier model | pre_model and tool results | prompt injection | fast; covers tool results (F-132) |
| Presidio | open source | to decide | classic ML (spaCy) | before the model | personal data to mask | the model may need identifiers |
| LLM Guard | open source | rejected | scanner library | not placed | invisible text, secrets, topics | normalised() and router cover it |
| Llama Guard | open source | rejected | LLM safety classifier | not placed | harmful content | too slow for every turn |
| NeMo Guardrails | open source | rejected | dialogue rails framework | not placed | topic and dialogue rails | wants to own the loop |
| Guardrails AI | open source | rejected | validator framework | not placed | output validators | overlaps LLM Guard |
On its concern page: L7 · Policy enforcement
Compared: Which library checks and gets identity tokens?
| Option | From | Verdict | What it does | Checks tokens users send | Gets tokens for us | Hides failure kinds | Why |
|---|---|---|---|---|---|---|---|
| PyJWT + PyJWKClient | open source | chosen | checks signature, issuer, audience | yes, locally, keys cached | no | no | no Entra call per request |
| Plain httpx form posts | ours | chosen | sign-in, refresh, on-behalf-of | no | yes: one form post each | no: refused, misconfigured, down | code branches on each failure |
| Azure SDK (azure-identity) | Azure | drop here | gets tokens to call Azure | no | yes: managed identity | partly | wrong tool for checking tokens |
| MSAL | vendor | rejected | sign-in, refresh, OBO, caching | no | yes, with caching, retries | yes: hidden caching, retries | hides the failures we branch on |
| Identity port (Verifier, Exchange, Login) | ours | keep | one interface, many adapters | via the adapter | via the adapter | no | leaving Entra is configuration |
| Keycloak adapter | open source | reference agent | local identity provider | via the Verifier port | via the Login port | no | runs the reference agent off Azure |
On its concern page: L16 · Identity & authz
Compared: How does the agent call tools?
| Option | From | Verdict | What it is | Process boundary | Per-tool metadata | Auth on every call | Verdict reason |
|---|---|---|---|---|---|---|---|
| MCP Python SDK (mcp) | open source | chosen | official MCP client and server | yes: its own server | _meta: side effect, scope | bearer token checked again | MCP is the only path |
| FastMCP | open source | alternative | higher-level MCP server library | yes | yes | yes, via its auth | listed beside the official SDK |
| Function tools in process | open source | rejected | Python functions given to model | no: same process | only what we add | inherits the agent's rights | no far-end checks |
| OpenAPI / REST tools | open source | rejected | HTTP API described by OpenAPI | yes | no standard place | yes | not shaped by intent |
| APIM REST-as-MCP | Azure | rejected | APIM exposes REST as MCP | yes | no | at the gateway | wraps REST, not intent |
| Foundry tools | Azure | only on Foundry | tools in Foundry Agent Service | Foundry runs them | Foundry's own | Foundry connections | only if the loop moves there |
On its concern page: L3 · Tool layer
Compared: Where does conversation state live?
| Option | From | Verdict | Holds here | One transaction with ledger | Fits DBOS and pgvector | HA on Azure | Why |
|---|---|---|---|---|---|---|---|
| Azure PostgreSQL Flexible Server | Azure | chosen | state, ledger, approvals, findings | yes | yes | zone-redundant HA | one store, Entra login, backups |
| Cosmos DB | Azure | for global scale | nothing today | within one partition only | no | multi-region built in | only for global scale |
| Azure Managed Redis | Azure | optional | counters and short caches only | no | no | zone redundancy available | fast, but not the record |
| SQLite / file store | open source | local only | reference agent, on a laptop | yes, one process | no | none | simple local runs |
| In-memory store | ours | tests only | test state | n/a | no | none | fast, nothing survives a restart |
On its concern page: L5 · State & memory
Compared: Where do knowledge passages live and get searched?
| Option | From | Verdict | Where it runs | Hybrid keyword + vector | Entitlement filter | Extra infrastructure | When to pick |
|---|---|---|---|---|---|---|---|
| pgvector in Azure PostgreSQL | open source | chosen | inside our PostgreSQL | yes, with Postgres full text | a SQL filter before ranking | none: one store, one backup | default |
| Azure AI Search | Azure | later | managed search service | yes, plus semantic ranker | filters on indexed fields | a search service | corpus outgrows Postgres |
| Qdrant on Container Apps | open source | alternative | a container we run | yes, dense plus sparse | payload filters | a service plus its storage | dedicated vector store needed |
| Pinecone | vendor | not evaluated | managed SaaS | yes, sparse-dense | metadata filters | a SaaS outside Azure | named only |
On its concern page: 18 · Knowledge & retrieval
Compared: Where do traces, metrics and alerts go?
| Option | From | Verdict | Holds | Where used | Customer's words | Alerts | Status |
|---|---|---|---|---|---|---|---|
| Application Insights (OTel distro) | Azure | chosen | spans: one story per conversation | Azure | not captured (privacy) | log alerts, budget alert | on Azure today |
| Managed Prometheus + Grafana | Azure | planned | counters, alert rules, dashboards | Azure | never | same rules as reference | planned, Tier 12 |
| Langfuse | open source | reference agent | traces and scores | reference agent, locally | on sampled turns, redacted | via the watch job | second exporter option |
| Prometheus + Grafana (self-run) | open source | reference agent | counters, 21 alerts, dashboard | reference agent, locally | never | Alertmanager | Docker compose |
| Foundry tracing view | Azure | with Foundry | agent traces in Foundry | Azure | not decided | no | comes with a Foundry model |
| OTel Collector + Jaeger | open source | alternative | spans, routed anywhere | anywhere | as configured | no | listed open source option |
| LangSmith | vendor | not used | traces, datasets (paid) | with LangGraph | as configured | yes | not used |
On its concern page: L11 · Observability
Compared: Which tools run and judge our evaluations?
| Option | From | Verdict | Job here | Drives our agent loop | Needs a judge model | When it runs | Status |
|---|---|---|---|---|---|---|---|
| pydantic-evals | open source | chosen | offline eval runner | yes, through our harness | for soft graders only | pull request, release | used |
| DeepEval | open source | alternative | soft graders, case drafting | with an adapter | yes, for LLM metrics | release | alternative runner |
| Ragas | open source | alternative | retrieval scores, case drafting | no | yes, for most metrics | release | alternative to Foundry RAG |
| promptfoo | open source | prompt-only jobs | model × test matrix; red team | no: needs an adapter | for model-graded checks | pull request | for prompt-only jobs |
| Inspect | open source | alternative | eval framework, soft graders | with an adapter | for model-graded scorers | release | alternative runner |
| Azure AI Foundry evaluation | Azure | planned | generic LLM judges | no: judges our records | yes, or Azure-billed | release, sampled online | planned, Tier 12 |
| PyRIT (AI Red Teaming Agent) | open source | to adopt | broad attack library | via our oracles | yes, attacker model | weekly, before release | to adopt |
| garak | open source | to adopt | LLM vulnerability scanner | no: probes the model | no for most probes | weekly, before release | to adopt |
On its concern page: L12 · Eval harness
Compared: How do we replay a run without live calls?
| Option | From | Verdict | What it records | Level | On a miss | Status |
|---|---|---|---|---|---|---|
| Own cassettes at the model choke point | ours | chosen | request fingerprint → response | model calls | run fails, never goes live | built; used in release gates |
| VCR.py (vcrpy) | open source | to adopt | HTTP requests and responses | HTTP, for tool calls | configurable record mode | to adopt for tools |
| Pydantic AI TestModel / FunctionModel | open source | chosen | nothing: scripted replies | model layer | n/a | used in tests |
| Langfuse / LangSmith replay | vendor | not used | traced prompts | one model call, in a UI | n/a: re-runs live | not used |
On its concern page: L9 · Determinism & replay
Compared: How is cost measured and capped?
| Option | From | Verdict | Grain | Stops a turn | Source | Status |
|---|---|---|---|---|---|---|
| Own Meter + cost framework | ours | chosen | per step, turn, successful task | yes: $0.50 per turn | price table × tokens | built (framework v0) |
| APIM token metrics | Azure | chosen | per API and subscription | no: Consumption lacks it | gateway token counts | built |
| Langfuse cost tracking | open source | alternative | per trace and generation | no | its model price table | listed open source option |
| Helicone | vendor | not evaluated | per request via its proxy | rate limits only | its price table | not evaluated |
| Azure Cost Management | Azure | chosen | per resource, per day | no: alerts only | the Azure bill | built: $10 budget alert |
On its concern page: L13 · Cost accounting
Compared: What retries and fails over when a call breaks?
| Option | From | Verdict | Where it acts | Knows our typed failures | Shares one retry budget | Status |
|---|---|---|---|---|---|---|
| ResilientLLM (own wrapper) | ours | chosen | inside the LLMClient port | yes | yes | built |
| tenacity | open source | alternative | any Python call | only what we tell it | if we wire it | listed open source option |
| Provider SDK retries | vendor | rejected | inside the openai SDK | no | no: invisible to us | off: max_retries=0 |
| Pydantic AI fallback model | open source | chosen | inside the model adapter | partly | yes, one budget | design: degradation path |
| APIM backend pools + breaker | Azure | planned | at the gateway | no | must share our budget | planned, Tier 10 |
| Health probes + Front Door | Azure | chosen | containers and regions | no | n/a | final (design) |
On its concern page: L10 · Failure handling
Compared: How are inputs, outputs and records typed?
| Option | From | Verdict | Validation | JSON Schema for tools | Immutability | Status |
|---|---|---|---|---|---|---|
| Pydantic (frozen models) | open source | chosen | full, with clear errors | yes: Field descriptions | yes: frozen=True | built: every contract |
| jsonschema | open source | chosen | validates against a schema | uses the tool's schema | n/a | built: tool arguments |
| dataclasses + jsonschema | open source | not chosen | none built in; via jsonschema | by hand | yes: frozen=True | not chosen |
| attrs (+ cattrs) | open source | not evaluated | validators you write | no built-in | yes: frozen | not evaluated |
| msgspec | open source | not evaluated | typed decoding, very fast | yes | yes: frozen Structs | not evaluated |
| APIM schema validation | Azure | at the edge | at the gateway | uses API schema | n/a | design: edge check |
On its concern page: L6 · I/O contracts
Compared: What queues turns when load spikes?
| Option | From | Verdict | Ordering and priority | Dead-letter | Scales workers (KEDA) | Extra infrastructure | Status |
|---|---|---|---|---|---|---|---|
| Azure Service Bus Premium | Azure | chosen | sessions; priority via queues | built in | yes | a namespace | final (design) |
| PostgreSQL as queue (turns table) | ours | chosen | leases; one turn per chat | ours: failed status | yes, query-based | none | turn status, to build |
| Redis Streams | open source | not evaluated | per stream, consumer groups | ours: pending list | yes | Redis we must keep durable | Redis is for counters only |
| Azure Storage Queues | Azure | not evaluated | no ordering guarantee | no built-in dead-letter | yes | a storage account | too plain for deadlines |
| APIM rate limits | Azure | chosen | not a queue: 429 on overload | n/a | no | none: already there | built: 30 calls a minute |
On its concern page: L8 · Concurrency & flow
Compared: How do config and releases change safely?
| Option | From | Verdict | Job | Azure-native | Portable off Azure | Ops burden | Status |
|---|---|---|---|---|---|---|---|
| Azure App Configuration | Azure | chosen | flags, model per job, thresholds | yes | no | low: managed | built: agent.enabled, limits |
| OpenFeature + flagd | open source | alternative | standard flag API + flag daemon | no | yes | we run flagd | listed open source option |
| Unleash | open source | alternative | feature-flag server and UI | no | yes | we run a server, or SaaS | listed open source option |
| Container Apps revisions | Azure | chosen | canary traffic split, rollback | yes | no | low | final |
| Bicep + azd | Azure | chosen | infrastructure as code | yes | no: Azure only | low | built |
| Terraform | open source | alternative | infrastructure as code | via azurerm provider | yes: many clouds | state file to manage | "Bicep or Terraform" |
| Argo CD | open source | not used | GitOps deploys to Kubernetes | no | yes | needs Kubernetes (AKS) | listed open source option |
On its concern page: L15 · Release & config
Compared: How do agents hand work to other agents?
| Option | From | Verdict | Where agents run | Open protocol | Works across vendors | Typed hand-off and limits | Status |
|---|---|---|---|---|---|---|---|
| A2A SDK endpoints on Container Apps | open source | chosen | our Container Apps | yes: A2A | yes | ours: ticket, depth, fan-out | final (design) |
| Foundry connected agents | Azure | only on Foundry | Foundry Agent Service | no: Foundry's own | Foundry agents only | Foundry's | if the loop moves to Foundry |
| Microsoft Agent Framework A2A | open source | .NET teams only | anywhere | yes: A2A support | yes | partly framework, partly ours | listed Azure option |
| In-process hand-off (one service) | ours | inside one agent | one process | no | no | ours | fine within one agent |
| Azure API Center (registry) | Azure | chosen | registry, not a runtime | lists A2A endpoints | yes | no | final: who exists, which version |
On its concern page: 17 · Agent-to-agent
Compared: Which channels and which human desk?
| Option | From | Verdict | Customer window | Staff desk and hand-off | Channels | Verdict reason |
|---|---|---|---|---|---|---|
| Next.js window + Chatwoot desk | open source | chosen | best: our cards, upload, brand | good: queues, hand-off | web, WhatsApp, SMS, email | our default |
| Next.js window only | ours | rejected | best | none: build it | web only | a contact centre isn't our job |
| Chatwoot widget + desk | open source | rejected for customers | generic | good | web, WhatsApp, SMS, email | claims needs rich cards, upload |
| Azure Communication Services | Azure | chosen | no window | no desk | WhatsApp, SMS, email | the channel pipes |
| Azure Bot Service | Azure | Teams only | no | no human desk | Teams connector | never as the desk |
| Dynamics 365 Contact Center | vendor | where it exists | its own | best: staff already use it | many | only if already owned |
On its concern page: CH · Channels & human desk
Compared: How is personal data found and masked?
| Option | From | Verdict | Where it runs | India IDs (Aadhaar, PAN) | Network call | Handles free text | Status |
|---|---|---|---|---|---|---|---|
| Regex redact() (ours) | ours | chosen | every payload and log line | yes, Aadhaar Verhoeff-checked | none | known patterns only | built |
| no_pii_echo reply rule | ours | chosen | post_model and reply | yes | none | known patterns only | built |
| Microsoft Presidio | open source | to decide | in our process (spaCy) | built-in IN_PAN, IN_AADHAAR | none | yes: names, places | to decide: model needs IDs |
| Azure AI Language PII | Azure | planned | Azure service | yes, India entity types | yes | yes | design: mask before model |
On its concern page: PV · Privacy & data protection