Agent Harness Tutorial · Testing
Scenarios in AgentTwin: a simulated world
A scenario is a whole small story run against a simulated world: who is asking, what they say turn by turn, what the approver and the desk do, what can go wrong, and what must be true afterwards. The real agent runs unchanged; the world around it is a twin.
| Expectation | Means |
|---|---|
| {world: unchanged} | nothing was written, paid or changed |
| {effect: issue_payout, times: 1} | the payout happened exactly once |
| {called: withdraw_claim, times: 0} | a tool was never called |
| {row: claim, id, field, equals} | a record ends in a given state |
| {reply: never_says, text} | the reply never contains a phrase (e.g. another customer's name) |
| {handed_off: 1} | a person was brought in |
How many. 29 yardstick scenarios plus 17 local ones for the claims agent; 49 in the reference agent. Today they run with a scripted model, once each (pass^1); live models with repeats (pass^k) are designed for nightly runs. The time jump is real: nobody-picks-up-the-escalation steps 1,860 seconds per turn to pass the 30-minute hand-off expiry.
Fidelity modes: mock, replay, real, shadow. The world records what it is faithful about and what it isn't; checking it against production (shadow mode) is still to come.
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
tests/scenarios/a-long-conversation-fetches-a-person.yaml:1A scenario, as written# esc:turns-exceeded — ten turns without resolving fetch a person unasked.
apiVersion: awd-scenario/v0
scenario: a conversation that runs long fetches a person on its own
world: ../../../clean-ai-engineering/gates/motor-claims-fnol/worlds/motor-claims-fnol.yaml
as: PH-1001
objective: Ten turns of small talk, and a person picks it up unasked.
concern: interaction-capability
discharges: [esc:turns-exceeded, P-ESC-ONCE, op:escalate, ext:escalation_desk]
max_turns: 12
step_seconds: 600
actor:
says:
- hello
- I just have a general question
- how does this chat work
- ok
- thanks for that
- and one more
- sorry, give me a second
- right
- hmm
- one moment
- ok then
- bye
desk:
resolves: handled
expect:
- {handed_off: 1}
- {world: unchanged}
model:
- says: Hello — I can help with your claims and policies.
times: 30tests/test_scenarios.py:51The runner: every scenario through the real Agent.handle@pytest.mark.parametrize("path", RUNNABLE)
async def test_a_declared_scenario_passes_every_check_it_makes(path: Path) -> None:
scenario = load_scenario(path)
live = Live.start(load(path.parent / scenario.world))
timeline = timeline_for(scenario)
clock = Clock(step_s=scenario.step_seconds)
wrap = perturbed(live, timeline, clock)
async with subject_for(
live,
llm=model_for(scenario),
wrap=wrap,
clock=clock,
provider_faults=provider_faults(scenario),
) as subject:
record, outcomes = await run_file(
path, subject=subject, live=live, timeline=timeline, clock=clock
)
failed = [f"{o.check} — {o.detail}" for o in outcomes if not o.passed]
assert failed == [], f"{scenario.scenario}:\n " + "\n ".join(failed)
assert record.discharges == scenario.discharges