AgentTwin
Test an AI agent in a simulated world and judge what changed, not only what it said.
1 · Purpose
AgentTwin places a real AI agent in a simulated world: its customers, the systems it calls, their data and their faults. It then checks what actually changed in that world, such as a refund issued without authority or one owed and never issued. It is for teams that build or buy agents that take actions, and for anyone who must judge an agent they did not build. Agents now move money and change records, and reading their replies is not enough to know whether they kept the rules.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Make an injected read error come back as an error
- Add a scenario to the lending-library test world
- Write a short how-to for running one τ² example
4 · Requirements
What it must do
- Reads a world description (AWD) that names the agent's systems, records, rules and faults.
- Serves that world to the agent as the tools it calls, including as an MCP (Model Context Protocol) server.
- Plays customers and approvers from scripts, or hands the customer to a model for open-ended conversations.
- Injects faults such as slowness, lost replies and stale reads.
- Judges each run with four state checks: rule-breaking writes, owed actions, false statements and other customers' data.
- Plants injection attacks drawn from PyRIT, AgentDojo or built-in templates, and judges them by what changed.
- Reports which scenarios a suite is missing, using pairwise coverage.
How well it must do it
- It is a description format first; the simulator shows the format can be run.
- Outside tools are adopted behind seams; only the format, the stand-in, the judges and the mode switch are its own.
- Same world and seed give the same result, so failures can be reproduced.
- Its own tests use a lending library, so the format is checked on a domain it was not written for.
- Every adoption is labelled as adopted or still planned.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS |
|---|---|---|---|
| World loader | Python, JSON Schema ✓ | — | — |
| Stand-in systems | Python, official MCP SDK ✓ | — | — |
| Actors | scripts; LangWatch Scenario for model-played customers ✓ | — | — |
| Fault injector | built-in perturbations ✓; Toxiproxy (planned) | — | — |
| Agent under test | — | the motor-claims agent built for Azure, run in-process ✓ | not built |
| Judges | Python state checks; pass^k from τ²-bench ✓ | — | — |
| Run record | pytest and JSON ✓ | — | — |
7 · What needs to be done
Format and simulator 4 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| World format, schema and loader | done | M | Specification design |
| Stand-in systems served over MCP | done | M | Python |
| Four state judges and the run command | done | M | Python |
| Release 0.13.0 with 397 passing tests | done | S | Testing |
Agents we did not build 2 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Judge τ²-bench's airline, retail and telecom agents from their policies alone | done | L | AI testing |
| Explain every disagreement with τ²-bench's own verdicts | done | M | Analysis |
| Run one more public benchmark agent through AgentTwin | todo | L | AI testing |
Adopt, don't build 1 of 5 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Adopt PyRIT, AgentDojo, Hypothesis, allpairspy and LangWatch Scenario | done | M | Python |
| Adopt an LLM grader such as DeepEval or Inspect behind a seam | todo | M | AI evaluation |
| Adopt Toxiproxy for network faults | todo | S | Testing |
| Adopt VCR.py for record and replay | todo | S | Python |
| Adopt SDV for realistic data volume | todo | M | Data engineering |
Format gaps 0 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Answer an injected read error as an error, not a refusal | todo | S | Python |
| Let the format express rules about a change (new versus old values) | todo | M | Specification design |
| Describe a customer who acts with their own tools | todo | L | Specification design |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Simulator developer | 2 | 3-5 hours/week | Builds stand-ins, judges and adapters to adopted tools. |
| World author | 2 | 2-4 hours/week | Writes worlds and scenarios for a new domain. |
| Benchmark analyst | 1 | 2-4 hours/week | Runs public agents through AgentTwin and explains each disagreement. |
9 · What you'll learn
- Why judging an agent by the state it leaves beats judging its words.
- How to turn an agent's policy into a testable world.
- How to plant injection attacks and judge them fairly.
- How pairwise coverage shows which scenarios a suite is missing.
- How to adopt open-source tools behind seams instead of rebuilding them.
Level-up questions
- What can a final-state judge never see, and how does the ownership judge see it?
- Why does a mock that drifts from the real system make a test suite a toy?
- When should a customer be scripted, and when played by a model?
- How would you write a world for a domain you know, such as hotel bookings?