cleandataengineering.aiAgent Harness, drawn simply
You are here: End to end › Eval › Offline › The gate

Agent Harness Tutorial · End to end · Zoom level 3

The release gate

The gate is a file (gates.yaml) that lists what must be true before a release. CI runs it on every push. Images are only published when everything is green.

Behaviour29 of 29 scenarios passpassFeatures750 tests, 0 failurespassStructure16 of 16 layers mappedpassHarness81 duties: 54 proved, 23 accepted gaps, 4 n/apass
From the latest verdict file in the claims agent's reports folder.

The two release checks in evaluators.yaml, tool_selection (did it pick the right tool?) and must_include (did the reply say what it must?), need a pass rate of 1.0. One miss blocks the release.

Uncertain is not a pass. A result the gate can't decide goes to a person. And the line can only go up: a release may never be worse than the last one.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

gates.yaml:5gates.yaml: what must be true before a release
apiVersion: gates/v0
agent: motor-claims-fnol
aoas: ../clean-ai-engineering/drafts/examples/motor-claims-fnol.aoas.yaml
profile: harness-profile.yaml

# `agenttwin.Binding`: `(live, *, wrap, clock, model) -> Subject`.
binding: evals.gate_binding:open_subject

tests:
  # Each test carries its `discharges` ids as JUnit properties (tests/conftest.py).
  command: uv run python -m pytest -q -p no:cacheprovider --junitxml={junit}

structure:
  checks:
    types: uv run mypy
    imports: uv run lint-imports
    complexity: uv run ruff check src tests evals
  # One module per harness layer (AHC taxonomy/layers.yaml). The engine of every
  # layer is the installed library, agent_harness — named where it lives, in the
  # reference agent's checkout this project depends on by path — and a layer
  # whose values are this agent's names its own module beside it.
  layers:
    L1: ../reference-agent/packages/agent-harness/src/agent_harness/context
    L2: ../reference-agent/packages/agent-harness/src/agent_harness/llm
    L3: ../reference-agent/packages/agent-harness/src/agent_harness/tools
    L4:
      - ../reference-agent/packages/agent-harness/src/agent_harness/loop
      - src/claims_fnol/router
    L5: ../reference-agent/packages/agent-harness/src/agent_harness/state
    L6:
      - ../reference-agent/packages/agent-harness/src/agent_harness/contracts
      - src/claims_fnol/contracts
    L7:
      - ../reference-agent/packages/agent-harness/src/agent_harness/policy

Tech choices (what we chose, Azure and open-source options): L15 · Release & config