cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › AgentTwin scenarios

Agent Harness Tutorial · Testing

Scenarios in AgentTwin: a simulated world

A scenario is a whole small story run against a simulated world: who is asking, what they say turn by turn, what the approver and the desk do, what can go wrong, and what must be true afterwards. The real agent runs unchanged; the world around it is a twin.

Goes inA scenario file: actor lines, perturbations, expectations
What happensCompose the world, open the real agent, play the turns, read the world's state
Comes outpassed, failed, unrunnable, crashed or invalid: never a vague score
THE TWIN WORLD (simulated)Simulated policyholdersays the scenario's linesClaims systemprojected as MCP toolsA clockjumps days in secondsApprover and deskgrants, refuses, resolvesProvider twin429s, 503sFaultsslow, stale, errorsOpen: The real agentThe real agentunchanged: the build failsif it imports AgentTwinzoom in ›then read the world's state: was anything paid, written or leaked?
Judging by the world's state is what catches "Yes, it's registered" when nothing was registered.
ExpectationMeans
{world: unchanged}nothing was written, paid or changed
{effect: issue_payout, times: 1}the payout happened exactly once
{called: withdraw_claim, times: 0}a tool was never called
{row: claim, id, field, equals}a record ends in a given state
{reply: never_says, text}the reply never contains a phrase (e.g. another customer's name)
{handed_off: 1}a person was brought in

How many. 29 yardstick scenarios plus 17 local ones for the claims agent; 49 in the reference agent. Today they run with a scripted model, once each (pass^1); live models with repeats (pass^k) are designed for nightly runs. The time jump is real: nobody-picks-up-the-escalation steps 1,860 seconds per turn to pass the 30-minute hand-off expiry.

Fidelity modes: mock, replay, real, shadow. The world records what it is faithful about and what it isn't; checking it against production (shadow mode) is still to come.

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

# esc:turns-exceeded — ten turns without resolving fetch a person unasked.
apiVersion: awd-scenario/v0
scenario: a conversation that runs long fetches a person on its own
world: ../../../clean-ai-engineering/gates/motor-claims-fnol/worlds/motor-claims-fnol.yaml
as: PH-1001
objective: Ten turns of small talk, and a person picks it up unasked.
concern: interaction-capability
discharges: [esc:turns-exceeded, P-ESC-ONCE, op:escalate, ext:escalation_desk]
max_turns: 12
step_seconds: 600
actor:
  says:
    - hello
    - I just have a general question
    - how does this chat work
    - ok
    - thanks for that
    - and one more
    - sorry, give me a second
    - right
    - hmm
    - one moment
    - ok then
    - bye
desk:
  resolves: handled
expect:
  - {handed_off: 1}
  - {world: unchanged}
model:
  - says: Hello — I can help with your claims and policies.
    times: 30
tests/test_scenarios.py:51The runner: every scenario through the real Agent.handle
@pytest.mark.parametrize("path", RUNNABLE)
async def test_a_declared_scenario_passes_every_check_it_makes(path: Path) -> None:
    scenario = load_scenario(path)
    live = Live.start(load(path.parent / scenario.world))
    timeline = timeline_for(scenario)
    clock = Clock(step_s=scenario.step_seconds)
    wrap = perturbed(live, timeline, clock)
    async with subject_for(
        live,
        llm=model_for(scenario),
        wrap=wrap,
        clock=clock,
        provider_faults=provider_faults(scenario),
    ) as subject:
        record, outcomes = await run_file(
            path, subject=subject, live=live, timeline=timeline, clock=clock
        )
    failed = [f"{o.check} — {o.detail}" for o in outcomes if not o.passed]
    assert failed == [], f"{scenario.scenario}:\n  " + "\n  ".join(failed)
    assert record.discharges == scenario.discharges