AgentTwin

Test an AI agent in a simulated world and judge what changed, not only what it said.

Active · join meAI Engineering

7 of 15 tasks done

1 · Purpose

AgentTwin places a real AI agent in a simulated world: its customers, the systems it calls, their data and their faults. It then checks what actually changed in that world, such as a refund issued without authority or one owed and never issued. It is for teams that build or buy agents that take actions, and for anyone who must judge an agent they did not build. Agents now move money and change records, and reading their replies is not enough to know whether they kept the rules.

For: Agent developers, QA teams, independent assessors, buyers checking a vendor's agent.

2 · Business benefits

Catch costly actions before launchRule-breaking writes, such as a refund over a limit, are found in simulation, not in the ledger.
Find what was never doneIt flags actions that were owed to a customer and never taken, which a reply check cannot see.
Privacy checks that leave a traceIt notices when an agent reads another customer's record, even when nothing changed.
Judge an agent you did not buildAn assessor needs only the agent's policy and tool definitions, not its code.
No paid test infrastructureRuns on a laptop in seconds with scripted customers and no model calls.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Make an injected read error come back as an error
  • Add a scenario to the lending-library test world
  • Write a short how-to for running one τ² example

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

World loaderReads the world description andrefuses one that breaks theStand-in systemsPresent the world as the toolsthe agent calls, enforcing eachActorsScripted or model-playedcustomers and approvers.Fault injectorAdds slowness, lost replies andstale reads.Agent under testThe real agent, connectedthrough a binding.JudgesRead the world's state and flagrule breaks, owed actions,Run recordOne file per run with everycall, effect and verdict.world projecteverdicts
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Actors → Agent under test: customer messages · Agent under test → Stand-in systems: tool calls · Fault injector → Stand-in systems: faults · Stand-in systems → Judges: state before and after

6 · Tech stack, per platform

ComponentOpen stackAzureAWS
World loaderPython, JSON Schema ✓——
Stand-in systemsPython, official MCP SDK ✓——
Actorsscripts; LangWatch Scenario for model-played customers ✓——
Fault injectorbuilt-in perturbations ✓; Toxiproxy (planned)——
Agent under test—the motor-claims agent built for Azure, run in-process ✓not built
JudgesPython state checks; pass^k from τ²-bench ✓——
Run recordpytest and JSON ✓——

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Format and simulator 4 of 4 done

TaskStatusSizeSkill
World format, schema and loaderdoneMSpecification design
Stand-in systems served over MCPdoneMPython
Four state judges and the run commanddoneMPython
Release 0.13.0 with 397 passing testsdoneSTesting

Agents we did not build 2 of 3 done

TaskStatusSizeSkill
Judge τ²-bench's airline, retail and telecom agents from their policies alonedoneLAI testing
Explain every disagreement with τ²-bench's own verdictsdoneMAnalysis
Run one more public benchmark agent through AgentTwintodoLAI testing

Adopt, don't build 1 of 5 done

TaskStatusSizeSkill
Adopt PyRIT, AgentDojo, Hypothesis, allpairspy and LangWatch ScenariodoneMPython
Adopt an LLM grader such as DeepEval or Inspect behind a seamtodoMAI evaluation
Adopt Toxiproxy for network faultstodoSTesting
Adopt VCR.py for record and replaytodoSPython
Adopt SDV for realistic data volumetodoMData engineering

Format gaps 0 of 3 done

TaskStatusSizeSkill
Answer an injected read error as an error, not a refusaltodoSPython
Let the format express rules about a change (new versus old values)todoMSpecification design
Describe a customer who acts with their own toolstodoLSpecification design

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Simulator developer23-5 hours/weekBuilds stand-ins, judges and adapters to adopted tools.
World author22-4 hours/weekWrites worlds and scenarios for a new domain.
Benchmark analyst12-4 hours/weekRuns public agents through AgentTwin and explains each disagreement.

9 · What you'll learn

Level-up questions

10 · Links