cleandataengineering.aiAgent Harness, drawn simply
You are here: Testing › Failure drills

Agent Harness Tutorial · Testing

Failure drills

A failure drill breaks something on purpose and checks the agent still does the right thing: no double payout, no lost conversation, an honest reply. Several are already covered in-process; the live drills against Azure are still to build.

DrillHow it is tested todayLive on Azure
Crash before the checkpoint is savedtest_redelivery.py: redelivery keeps the key; no second claimto build
The same message twiceledger tests; flow 17 livedone once by hand
Approve twicescenario the-handler-approves-it-twice: paid onceto build
Model fails / throttlesscenarios the-model-fails-twice, the-provider-throttlesto build (fault injection at APIM)
Browser closed and reopenedtest_resume.pyflow 3 live
Slow and stale reads, channel faultstest_break_it.py (Slow, StaleRead)to build
Database failover · deploy mid-conversation · storm at 3× peak—to build (Azure Chaos Studio, Load Testing)

How it is done: the real code

Cut from the live repositories each time the site is built, so it always matches what runs.

tests/test_redelivery.py:143A crash before the checkpoint does not register a claim twice
async def test_a_crash_before_the_checkpoint_does_not_register_a_claim_twice(
    case: str, again: Again, same_key: bool, claims: int
) -> None:
    world = live()
    clock = Monotonic()
    ledger = InMemoryRequests(now=clock)
    store = InMemoryCheckpointStore()
    server = project(world, scopes=SCOPES, authorise=as_policyholder, unknown_record="result")
    async with connect(server, requests=InMemoryRequests()) as projected:
        tools = Keys(Entities(projected, entities_of(world)))
        dying = ep.build(
            llm=ScriptedClient(script()), tools=tools, store=Dies(), deliveries=Gone(ledger)
        )  # type: ignore[arg-type]
        with pytest.raises(Crash):
            await dying.handle(REPORT, identity=me(), delivery_id="msg-1")
        assert world.count("register_claim") == 1, "the effect landed before the crash"

        clock.at += CLAIM_TTL_S + 1  # the dead process's claim lapses
        delivery, run_id, conversation = again()
        restarted = ep.build(
            llm=ScriptedClient(script()), tools=tools, store=store, deliveries=ledger
        )
        _, after = await restarted.handle(
            REPORT,
            identity=me(),
            delivery_id=delivery,
            run_id=run_id,
            conversation=conversation,  # type: ignore[arg-type]
        )

    (first_key, first_ref), (second_key, second_ref) = tools.sent
    assert (first_key == second_key) is same_key, case
    assert world.count("register_claim") == claims, case
    assert (first_ref == second_ref) is same_key, case