Agent Harness Tutorial · Testing
Failure drills
A failure drill breaks something on purpose and checks the agent still does the right thing: no double payout, no lost conversation, an honest reply. Several are already covered in-process; the live drills against Azure are still to build.
| Drill | How it is tested today | Live on Azure |
|---|---|---|
| Crash before the checkpoint is saved | test_redelivery.py: redelivery keeps the key; no second claim | to build |
| The same message twice | ledger tests; flow 17 live | done once by hand |
| Approve twice | scenario the-handler-approves-it-twice: paid once | to build |
| Model fails / throttles | scenarios the-model-fails-twice, the-provider-throttles | to build (fault injection at APIM) |
| Browser closed and reopened | test_resume.py | flow 3 live |
| Slow and stale reads, channel faults | test_break_it.py (Slow, StaleRead) | to build |
| Database failover · deploy mid-conversation · storm at 3× peak | — | to build (Azure Chaos Studio, Load Testing) |
How it is done: the real code
Cut from the live repositories each time the site is built, so it always matches what runs.
tests/test_redelivery.py:143A crash before the checkpoint does not register a claim twiceasync def test_a_crash_before_the_checkpoint_does_not_register_a_claim_twice(
case: str, again: Again, same_key: bool, claims: int
) -> None:
world = live()
clock = Monotonic()
ledger = InMemoryRequests(now=clock)
store = InMemoryCheckpointStore()
server = project(world, scopes=SCOPES, authorise=as_policyholder, unknown_record="result")
async with connect(server, requests=InMemoryRequests()) as projected:
tools = Keys(Entities(projected, entities_of(world)))
dying = ep.build(
llm=ScriptedClient(script()), tools=tools, store=Dies(), deliveries=Gone(ledger)
) # type: ignore[arg-type]
with pytest.raises(Crash):
await dying.handle(REPORT, identity=me(), delivery_id="msg-1")
assert world.count("register_claim") == 1, "the effect landed before the crash"
clock.at += CLAIM_TTL_S + 1 # the dead process's claim lapses
delivery, run_id, conversation = again()
restarted = ep.build(
llm=ScriptedClient(script()), tools=tools, store=store, deliveries=ledger
)
_, after = await restarted.handle(
REPORT,
identity=me(),
delivery_id=delivery,
run_id=run_id,
conversation=conversation, # type: ignore[arg-type]
)
(first_key, first_ref), (second_key, second_ref) = tools.sent
assert (first_key == second_key) is same_key, case
assert world.count("register_claim") == claims, case
assert (first_ref == second_ref) is same_key, case