cleandataengineering.aiAgent Harness, drawn simply
You are here: Tech stack › L12 · Eval harness

Agent Harness Tutorial · Tech stack

L12 · Eval harness

Golden sets and graders that gate every release. Family: Change · between releases.

AZURE OPTIONSFoundry evaluation◐ planned: generic judges, tier 12AI Red Teaming Agent◐ plannedOUR CHOICEAgentTwinsimulated world, 29 scenariospytest + 4 CI gatespydantic-evalsoffline runner, ModelSelectorOPEN-SOURCE OPTIONSAgentTwin · pytest · pydantic-evals✓ usedpromptfoo, PyRIT, garak◐ planned: red teamDeepEval, Inspect· considered: not pickedWHO DOES THE WORKAzure gives 25% · we code once 45% · each agent team still does 30%
Dark boxes are what we run. Solid boxes are used or planned; dashed boxes were considered or dropped (crossed). The bar: grey is what Azure gives, dark is what we code once, hatched is what each agent team writes.
Why our choiceDomain checks have to be deterministic and owned. Azure judges plug in behind the same Evaluator port and YAML.
OptionWhere fromVerdictWhy
AgentTwinours✓ chosensimulated world, 29 scenarios
pytest + 4 CI gatesours✓ chosen
pydantic-evalsours✓ chosenoffline runner, ModelSelector
Foundry evaluationAzure◐ plannedgeneric judges, tier 12
AI Red Teaming AgentAzure◐ planned
AgentTwin · pytest · pydantic-evalsopen source✓ used
promptfoo, PyRIT, garakopen source◐ plannedred team
DeepEval, Inspectopen source· considerednot picked

How it works inside the harness: Eval.

Compared: Which tools run and judge our evaluations?

pydantic-eva…open sourcechosenDeepEvalopen sourcealternativeRagasopen sourcealternativepromptfooopen sourceprompt-only jobsInspectopen sourcealternativeAzure AI Fou…AzureplannedPyRIT (AI Re…open sourceto adoptgarakopen sourceto adopt
Dark: what we chose. Crossed and dashed: rejected or not for us. From the Series.
OptionFromVerdictJob hereDrives our agent loopNeeds a judge modelWhen it runsStatus
pydantic-evalsopen sourcechosenoffline eval runneryes, through our harnessfor soft graders onlypull request, releaseused
DeepEvalopen sourcealternativesoft graders, case draftingwith an adapteryes, for LLM metricsreleasealternative runner
Ragasopen sourcealternativeretrieval scores, case draftingnoyes, for most metricsreleasealternative to Foundry RAG
promptfooopen sourceprompt-only jobsmodel × test matrix; red teamno: needs an adapterfor model-graded checkspull requestfor prompt-only jobs
Inspectopen sourcealternativeeval framework, soft graderswith an adapterfor model-graded scorersreleasealternative runner
Azure AI Foundry evaluationAzureplannedgeneric LLM judgesno: judges our recordsyes, or Azure-billedrelease, sampled onlineplanned, Tier 12
PyRIT (AI Red Teaming Agent)open sourceto adoptbroad attack libraryvia our oraclesyes, attacker modelweekly, before releaseto adopt
garakopen sourceto adoptLLM vulnerability scannerno: probes the modelno for most probesweekly, before releaseto adopt
DecisionAgentTwin plus pytest drive the agent in a simulated world, with pydantic-evals as the runner and our own hard-check graders; Foundry evaluators supply generic judges once calibrated. promptfoo stays for prompt-only jobs, and PyRIT and garak are added for weekly red-team runs.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

apiVersion: gates/v0
agent: motor-claims-fnol
aoas: ../clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml
profile: harness-profile.yaml

# `agenttwin.Binding`: `(live, *, wrap, clock, model) -> Subject`.
binding: evals.gate_binding:open_subject

tests:
  # Each test carries its `discharges` ids as JUnit properties (tests/conftest.py).
  command: uv run python -m pytest -q -p no:cacheprovider --junitxml={junit}

structure:
  checks:
    types: uv run mypy
    imports: uv run lint-imports
    complexity: uv run ruff check src tests evals
  # One module per harness layer (AHC taxonomy/layers.yaml). The engine of every
  # layer is the installed library, agent_harness — named where it lives, in the
  # reference agent's checkout this project depends on by path — and a layer
  # whose values are this agent's names its own module beside it.
  layers:
    L1: ../reference-agent/packages/agent-harness/src/agent_harness/context
    L2: ../reference-agent/packages/agent-harness/src/agent_harness/llm
    L3: ../reference-agent/packages/agent-harness/src/agent_harness/tools
    L4:
      - ../reference-agent/packages/agent-harness/src/agent_harness/loop
      - src/claims_fnol/router
    L5: ../reference-agent/packages/agent-harness/src/agent_harness/state
    L6:
      - ../reference-agent/packages/agent-harness/src/agent_harness/contracts
      - src/claims_fnol/contracts
    L7:
      - ../reference-agent/packages/agent-harness/src/agent_harness/policy