Agent Harness Tutorial · Tech stack
L12 · Eval harness
Golden sets and graders that gate every release. Family: Change · between releases.
Why our choiceDomain checks have to be deterministic and owned. Azure judges plug in behind the same Evaluator port and YAML.
| Option | Where from | Verdict | Why |
|---|---|---|---|
| AgentTwin | ours | ✓ chosen | simulated world, 29 scenarios |
| pytest + 4 CI gates | ours | ✓ chosen | |
| pydantic-evals | ours | ✓ chosen | offline runner, ModelSelector |
| Foundry evaluation | Azure | ◐ planned | generic judges, tier 12 |
| AI Red Teaming Agent | Azure | ◐ planned | |
| AgentTwin · pytest · pydantic-evals | open source | ✓ used | |
| promptfoo, PyRIT, garak | open source | ◐ planned | red team |
| DeepEval, Inspect | open source | · considered | not picked |
How it works inside the harness: Eval.
Compared: Which tools run and judge our evaluations?
| Option | From | Verdict | Job here | Drives our agent loop | Needs a judge model | When it runs | Status |
|---|---|---|---|---|---|---|---|
| pydantic-evals | open source | chosen | offline eval runner | yes, through our harness | for soft graders only | pull request, release | used |
| DeepEval | open source | alternative | soft graders, case drafting | with an adapter | yes, for LLM metrics | release | alternative runner |
| Ragas | open source | alternative | retrieval scores, case drafting | no | yes, for most metrics | release | alternative to Foundry RAG |
| promptfoo | open source | prompt-only jobs | model × test matrix; red team | no: needs an adapter | for model-graded checks | pull request | for prompt-only jobs |
| Inspect | open source | alternative | eval framework, soft graders | with an adapter | for model-graded scorers | release | alternative runner |
| Azure AI Foundry evaluation | Azure | planned | generic LLM judges | no: judges our records | yes, or Azure-billed | release, sampled online | planned, Tier 12 |
| PyRIT (AI Red Teaming Agent) | open source | to adopt | broad attack library | via our oracles | yes, attacker model | weekly, before release | to adopt |
| garak | open source | to adopt | LLM vulnerability scanner | no: probes the model | no for most probes | weekly, before release | to adopt |
DecisionAgentTwin plus pytest drive the agent in a simulated world, with pydantic-evals as the runner and our own hard-check graders; Foundry evaluators supply generic judges once calibrated. promptfoo stays for prompt-only jobs, and PyRIT and garak are added for weekly red-team runs.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
apiVersion: gates/v0 agent: motor-claims-fnol aoas: ../clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml profile: harness-profile.yaml # `agenttwin.Binding`: `(live, *, wrap, clock, model) -> Subject`. binding: evals.gate_binding:open_subject tests: # Each test carries its `discharges` ids as JUnit properties (tests/conftest.py). command: uv run python -m pytest -q -p no:cacheprovider --junitxml={junit} structure: checks: types: uv run mypy imports: uv run lint-imports complexity: uv run ruff check src tests evals # One module per harness layer (AHC taxonomy/layers.yaml). The engine of every # layer is the installed library, agent_harness — named where it lives, in the # reference agent's checkout this project depends on by path — and a layer # whose values are this agent's names its own module beside it. layers: L1: ../reference-agent/packages/agent-harness/src/agent_harness/context L2: ../reference-agent/packages/agent-harness/src/agent_harness/llm L3: ../reference-agent/packages/agent-harness/src/agent_harness/tools L4: - ../reference-agent/packages/agent-harness/src/agent_harness/loop - src/claims_fnol/router L5: ../reference-agent/packages/agent-harness/src/agent_harness/state L6: - ../reference-agent/packages/agent-harness/src/agent_harness/contracts - src/claims_fnol/contracts L7: - ../reference-agent/packages/agent-harness/src/agent_harness/policy