Home › Learn › AgentTwin, explained
AI engineering · free guide · with a step-through simulator
AgentTwin, explained
How do you know an AI agent is ready to talk to real customers? AgentTwin puts the real agent in a pretend world (pretend customers, pretend systems, pretend data, things going wrong) and then checks what actually changed in that world, not just what the agent said. This page explains it from zero, lets you play it on a tiny agent, scales it up, and then is honest about what is built, what is not, and who else does this.
What problem does it solve?
A flight simulator for AI agents
Pilots don't learn engine failure on a real plane with passengers. They sit in a simulator: the cockpit is real, everything outside it is pretend, and an instructor can make an engine fail, the weather turn, or the radio go silent. Afterwards nobody asks the pilot "did you feel confident?". They look at where the plane ended up.
AgentTwin does the same for an AI agent, such as a customer-support or insurance-claims assistant. The agent is the real one, unchanged. Everything it talks to is pretend: the customer, the order system, the database rows, the manager who approves refunds, the clock, and the faults (a slow system, a lost reply). After each test, judges look at what changed in the pretend world. Did a refund really get issued? Was something promised and never done? Did the agent read another customer's data?
That last point is the heart of it. An agent can say "your refund is on its way" when no refund exists. A judge reading the chat would pass that. A judge reading the world would not.
Is it a digital twin?
AgentTwin vs a digital twin
The name borrows from "digital twin", but it points the other way. A digital twin copies a real thing (a jet engine, a factory) so you can watch it and predict it. AgentTwin keeps the thing real (the agent) and copies the world around it, so you can test it.
| Digital twin | AgentTwin | |
|---|---|---|
| What is copied | The thing itself: an engine, a plant, a city | The world around the agent: customers, systems, data, people, faults, time |
| What stays real | The world. The thing runs for real | The agent. Its code, prompts and model are unchanged |
| Where data flows | From the real thing into the copy (sensors) | From the spec into the world; nothing comes from production |
| When it is used | In operation, for the thing's whole life | Before release and on every change; nightly once live |
| The question | What is this machine doing, and what will it do? | Will this agent do the right thing in every situation it must face? |
| How close to real | As close as possible, everywhere | Exact only where a test needs it; each world states what it is faithful about |
| What comes out | Predictions, alerts, what-ifs | Pass or fail per scenario, evidence, a coverage map, a ship-or-block verdict |
What goes in, what happens, what comes out?
Input, processing, output
Agent's owner
Writes the agent spec (called an AOAS): its tools, the rule behind each tool, the states things move through, and what it owes customers.
Tester
Fills the world with a few rows on each rule's edge and writes or generates scenarios: who says what, what goes wrong, what must be true at the end.
The Assurance Catalog
118 public checks an AI application may owe. The agent's shape picks which ones it owes.
Agent's team
Writes one binding function: how to start the agent and point it at the pretend tools.
What has to exist before any agent is tested?
How the system is set up
Four open repositories, set up once and shared by every agent:
AgentTwin
The library and command line: world format, stand-ins, actors, faults, clock, judges, run, scaffold, coverage.
AI Assurance Catalog
What must be true: 118 checks, 36 owed by every AI app, the rest by shape.
AI Harness Catalog
What must exist around the model call: the harness parts each shape needs.
Clean AI Engineering
The spec formats and the four gates, plus one "yardstick" file per agent that the agent's own team cannot edit.
The yardstick living outside the agent's repository matters: an agent that could bring its own tests could bring easy ones.
I have a new agent. What do I do?
What you do for each agent
- Write the agent's spec (AOAS). Its entities, tools, the rule behind each tool, the states and the promises it owes (for example: a refund is owed when a return is received). The agent's owner writes it; it is the single source of the rules.
- Say what shape it is. One or more of ten archetypes (a tool-using agent is A6; a deterministic router in front of it adds A5). The shape decides which Assurance Catalog checks it owes.
- Run
agenttwin scaffoldon the spec. It writes a world file, a starter scenario for each tool and rule, a binding stub and a gates file. Most of the work after this is data, not code: the electronics version of the shop agent is a 70-line world file. - Fill the world with rows. A few members, orders or claims that sit on each rule's edge: 7 days late and 8 days late, inside and outside a window. The loader refuses impossible rows (a returned book that is still overdue).
- Write the binding. One function that starts the agent and points it at the world's tools:
(live, *, wrap, clock, model) -> Subject. The agent never imports AgentTwin; the build fails if it tries. - Run
agenttwin coverage, then top up. It lists which combinations of intent, state, persona and fault, and which state changes, no scenario covers yet.scaffold --pairwisewrites skeletons for the missing ones. - Run the four gates.
tools/gates.pyruns the scenarios, checks every owed statement is proved or knowingly accepted, checks the build's structure and the harness parts. One command, one verdict, a route for every failure.
For a given agent, how do we know these are the tests it needs?
How we know which tests a given agent needs
This is the most important question on the page. The answer: nobody picks the tests by taste. The agent's test list is whatever it takes to prove everything the agent owes, and what it owes is written down, item by item, each with an id.
Where each owed item comes from:
- The agent's own spec. Every tool gets a test that it works inside its rule and is refused outside it. Every rule gets tests on its edges (7 days late passes, 8 fails; the spec's own numbers say which values matter). Every state change in the spec's state machines gets a path. Every promise ("a refund is owed once the return is received") gets a test that it is kept.
- The Assurance Catalog, by shape. A tool-using agent owes checks a plain chatbot doesn't, such as "hard termination under every condition". The shape picks them.
- The Harness Catalog, by shape. The harness parts the shape needs (approval gates, idempotency, kill switch) each need a test.
- Attack corpora. Injections are generated from PyRIT, AgentDojo and built-in templates into every field the spec marks untrusted.
- Combinations.
agenttwin coveragecrosses intent × state × persona × fault and keeps a small set that covers every pair (pairwise testing), so you don't need thousands of cases. - Production misses. When something slips through in the wild, it becomes a scenario, so it can't slip through twice.
Every test then says which items it proves (discharges: [AAC-0047, op:renew]). The gate builds a map: each owed item is proved, an accepted gap (with a reason, an owner and a review date), not applicable (with a reason), or not exercised. Any "not exercised" item fails the gate, with its name. So the answer to "are these the right tests?" is checkable: the right tests are the ones that leave nothing unaccounted for.
Writing the world and scenarios is a lot for a tester. How do we stop them missing things?
How we help the tester not miss
The tester should never start from a blank page, and should never have to remember the common cases. Six layers of help, from what the spec gives for free to what only production can teach:
| Layer | What it does for the tester | Status |
|---|---|---|
| 1 · From the spec | scaffold writes the world's rows on every rule's edge, and scenarios for each tool (allowed and refused), each refusal, a stranger, and an attack per untrusted field | built |
| 2 · What's missing | coverage lists uncovered pairs of intent, state, customer type (6 built in) and fault (8 kinds), and state changes never walked; --pairwise writes starters for them | built |
| 3 · The common pack | Scenarios every agent needs whatever its business: the same message twice, a long chat that should fetch a person, an approver who never answers, the kill switch, the model provider down, never quoting a figure no tool gave. Sized by the catalogs (118 assurance checks, 118 harness parts), not by a guess; each template fills itself from the spec | to build they exist, hand-written, in two agents |
| 4 · Variety | Many wordings, languages and moods per scenario, from a model-played customer (LangWatch), generated conversations (DeepEval) and attack rewrites (PyRIT) | partly |
| 5 · Search | An adversarial customer: a model given the spec and a goal ("get a refund you aren't owed") that keeps trying, with the judges deciding if it won. Finds what nobody thought to write | to build |
| 6 · Test the tests | Mutation testing: break the agent on purpose (remove a check, loosen 7 days to 8, drop the idempotency key) and see if any scenario notices. A break nobody notices is a named missing test | to build, first |
The gate stays the backstop: anything owed and unproved is named and blocks the release.
Hasn't AI already solved this? The calculator question
"If I want to test a calculator API, can't I generate all the test cases automatically and cover 100%?" Partly. Three different things hide inside "100%":
| "100%" of… | For a calculator |
|---|---|
| Every line of code run (code coverage) | easy Tools like Pynguin (Python) and EvoSuite (Java), and AI coding assistants, generate tests that run every line. But a test can run every line of a buggy divide and still pass. |
| Every possible input | impossible Two 64-bit numbers make about 3.4 × 1038 pairs. At a billion tests a second that is about 1022 years; the universe is about 1010 years old. |
| The right answer for each input | the hard part Generated tests often just record what the code does today, bugs included. Someone must say what is right. Testers call this the oracle problem. |
A calculator is the easy case, because maths gives an oracle for free: compare with a trusted library, or check rules that must always hold (a + b = b + a; (a + b) − b = a). Pick inputs smartly (zero, negatives, huge numbers, the edges) and a machine can test a calculator very well, without trying every input.
An agent has no free oracle. Whether "refund this order" is right depends on the shop's rules, the order's state, who is asking and what was promised. AI can generate endless conversations, but it can't know which outcome is correct unless the rules are written down. That is why AgentTwin starts from the agent's spec: the spec is the oracle, and the four judges apply it to the world's state.
How the software world handles it
Nobody has fully solved "complete tests". Mature teams combine six habits, and each has an AgentTwin counterpart:
| Habit | Software example | In AgentTwin |
|---|---|---|
| Derive cases from the spec mechanically | Equivalence classes, boundary values, decision tables, state-transition testing | built scaffold and the state walker |
| Cover every pair, not every combination | PICT, allpairspy | built |
| A standard checklist per kind of system | OWASP ASVS for web security; ISO 25010 quality traits | partly the Assurance Catalog, not yet turned into templates |
| Trace every requirement to a test | Aviation (DO-178C), medical devices (IEC 62304) | built discharges and the gate |
| Let a machine hunt for failures | Fuzzing, property-based testing (Hypothesis, Schemathesis) | partly Hypothesis for edges; no adversarial customer yet |
| Test the tests | Mutation testing (mutmut for Python, Stryker for JavaScript and C#) | to build |
How LangWatch handles it
In LangWatch Scenario, developers write scenarios in code: a model plays the user, a model judge scores the chat against criteria. An autopilot mode lets the simulated user improvise towards a goal, and LangWatch describes generating multi-step scenarios from a list of business policies, including where two policies overlap. Its answer is breadth: let a model write many scenarios. What it doesn't publish is a count of what's missing. AgentTwin's difference is that it counts the gaps; the idea worth borrowing is drafting scenarios where two rules meet.
What PyRIT-style tools add
PyRIT, Microsoft's red-teaming kit, is built from four parts: seeds (attack goals), converters (rewrite one attack many ways), attacker models (read the reply and try again, turn by turn) and scorers. Two of those ideas reach beyond security. Converters become "one scenario, many wordings, languages and moods", which takes the phrasing work off the tester. The attacker model becomes the adversarial customer in layer 5. PyRIT knows nothing of your business rules, so AgentTwin supplies the goals from the spec and does the scoring with its judges.
Every tool named here, explained in one card each: the open-source tools behind AgentTwin.
How are the pretend systems made?
How the pretend systems are made
Not by writing mocks by hand, and not by asking a model to play the system. AgentTwin reads the agent's spec and projects each system as a stand-in that holds state and enforces the spec's rules. Ask it to cancel a shipped order and it refuses, the way the real system would.
A canned mock that answers "OK" to everything is not enough: it would cancel the shipped order, the agent's test would pass, and it would prove nothing. A model playing the system is not enough either: it can be talked into allowing what the real system refuses.
| Kind of thing the agent talks to | How AgentTwin treats it |
|---|---|
| A data store | The world's state, shown as rows |
| An API with a contract | Answers from the declared contract |
| A tool server (MCP) | Tool results: the cleanest seam there is |
| The customer | An actor (scripted, state machine, or model-played), not a system |
| A person who approves | An actor plus a queue, acting between turns on a clock |
| A plain calculation | Run it for real. Mocking it adds risk and removes nothing |
Each world also says what it is faithful about and what it is not. The clothing shop's world is exact about order status, the return window and which actions can't be undone, and openly not faithful about prices, stock or carriers. A test may only check things inside that list.
Where does each test's data come from?
How test data is made, per case
- A small declared set of rows in the world file, chosen to sit on each rule's edge. Seeded, so every run is the same.
- Edge values are read off the rules: a rule that says "at most 30 days" makes 0, 30 and 31 the interesting values. Hypothesis adds more around them.
- Per-scenario changes: a scenario can plant a value (an injection in an order note) or set the clock forward. Planting is only allowed in fields the spec marks untrusted; the runner refuses otherwise, because it would test a threat nobody faces.
- A population for long runs: Faker generates customers and orders in the AgentTwin Lab, and the loader checks them against the same rules as hand-written rows. SDV for realistic volumes is planned.
- Impossible data is refused. A returned book that is still overdue, or a loan for a member who doesn't exist, stops the load with a reason.
Can I see one run, step by step?
Play it: a tiny library agent
The simplest agent we could pick: a library assistant with three tools (get_loan, renew, note_due_date) and one rule: a loan can be renewed only if it is open, at most 7 days late, and yours. AgentTwin's own tests use this library world. Asha is the simulated member. Pick a scenario, run it against a well-built agent and then a faulty one, and watch which judge catches what.
What happens, step by step
The world's state: the loan table
| loan | member | book | status | days late | note |
|---|
The scenario file for this case
These runs are written for this page to show the idea; they are not recorded from a live model. The scenario files are simplified from the real format.
What changes for a real agent?
Scaling up to a bigger agent
Nothing new in the method; more of everything. The same steps (spec, world, binding, scenarios, gates) and the same four judges, applied to the clothing-shop support agent and the motor-insurance claims assistant.
| Library (this page) | Clothing-shop agent | Motor-claims assistant | |
|---|---|---|---|
| Tools | 3 (get_loan, renew, note_due_date) | 5 (get_order, cancel_order, open_return_request, change_address, issue_refund) | claims tools, payout, document upload |
| Rules | 1 (open, 7 days late or less) | return window, final sale, cancel before shipping, address before shipping, refund needs approval | cover limits, payout limit, consent, injury escalation |
| People in the world | one member | customer + reviewer who approves refunds | policyholder, approver, human desk with a 30-minute hand-off expiry |
| Faults | lost reply | channel errors, stale reads, model provider down | gateway refusing calls (429, 503), slow systems |
| Time | none | days pass for return windows | the clock jumps 1,860 s per turn to expire a hand-off |
| Attacks | 1 planted note | 20 generated notes per run; 180 cases across both agents | injection in a claim note and from the user |
| Scenarios | 5 on this page | 49 | 29 in the shared yardstick + 17 local |
Two things only show up at this size. Time: a hand-off to a person that nobody picks up must expire after 30 minutes, so the world's clock jumps forward between turns. People: a reviewer who approves refunds can say yes, no, never answer, or say yes twice, and each needs its own scenario.
To show the method is not tuned to our own agents, AgentTwin also judged three τ²-bench agents (airline, retail, telecom) from their published policy and tools alone, on 2,472 recorded runs. On airline and retail it flagged exactly the rule-breaking writes τ²-bench flags (188 and 309), and it also checks whether another customer's record was read, which τ²-bench does not.
How do we get 100% coverage of the Assurance Catalog?
Coverage of the Assurance Catalog
The catalog has 118 checks. 36 are owed by every AI application; the rest depend on its shape. The clothing-shop agent is a tool-using agent behind a deterministic router (shapes A6 and A5), so it owes 62.
The 11 not yet proved, and what closes each:
| Check | What it asks | What closes it |
|---|---|---|
AAC-0007 | Latency within budget at expected concurrency | a load test; partly proved only in production |
AAC-0016 | Scheduled re-run against unchanged inputs | a nightly run; designed, not running |
AAC-0044, 0045, 0046, 0048, 0049, 0050 | Six checks for the router-plus-agent shape (A5): each node alone, both sides of each seam, partial failure, determinism, every branch, end-to-end budget | tests exist in spirit; none name these ids yet. The cheapest gap to close |
AAC-0096 | Cached responses never cross a trust boundary | a test once a cache exists; else 'not applicable', written down |
AAC-0097 | Processing region is enforced and recorded | a deployment check, not a simulation |
AAC-0118 | Autonomy widens only on agreement measured for that decision | only provable over time, in production |
So "100% coverage" here means one thing: every owed check is either proved by a passing test or signed off as a known gap. The gate already enforces that for harness parts. Extending it to catalog checks is the next step; until then the catalog number is counted, not gated.
How do we know the tests really ran?
How we know the tests really ran
- The yardstick lives outside the agent. The agent's team can't remove a scenario or bring an easier one.
- Five statuses, never a score. "Unrunnable" and "crashed" are not "passed". A scenario that can't run says so.
- A generator that plants nothing fails. Until 26 Sept 2026 an injection scenario with nothing planted passed quietly, and 12 declared cases claimed an attack check on zero attacks. Now the runner refuses.
- The agent can't see the simulator. If the agent's code imports AgentTwin, the build fails. A system that can see its own simulator produces results that mean nothing.
- Every test names what it proves, and the gate counts only passing ones.
- CI runs it on every push, and a failing gate blocks the release. Scenario coverage may only go up.
Are these tests enough to say "ready for production"?
Enough to call it ready for production?
No test suite proves an agent safe for every future conversation. What it can prove is that the agent keeps every promise it owes, in every situation we can name, under the faults we know about. Here is the list we think "ready" needs, and where we are:
| For "ready" | Status |
|---|---|
| All four gates pass (behaviour, features, structure, harness) | built the reference agent passes all four |
| Every owed check proved or signed off as a gap | partly harness parts gated; catalog 51 of 62 for the shop agent |
| Each scenario passes k times in a row on the live model (pass^k) | partly in the lab; nightly runs designed |
| The stand-ins checked against a real system (shadow mode) | to build |
| Generated attacks hold, judged by state | built 180 of 180 held (scripted model) |
| In production: canary cases, online rules, and every miss turned into a scenario | partly rules and alerts exist for the claims agent |
Our verdict today: enough to start a watched pilot, not enough to claim "proven ready". The missing pieces are named above.
If this were a product, what's in the box?
Built, and still to build
| Feature | Status | Note |
|---|---|---|
| World format (AWD), schema and loader that refuses bad worlds | built | v0.14.0, 29 Oct 2026; 433 passing tests |
| Stand-ins served as an MCP tool server, enforcing the spec's rules | built | the core of AgentTwin |
| Scripted, state-machine and model-played customers | built | model-played via LangWatch Scenario, an optional extra |
| Approver and human desk, acting between turns | built | grant, refuse, never answer, grant twice |
| Faults: slow, lost reply, channel error, stale read, provider down | built | per scenario, at a chosen call |
| Clock control (days pass in seconds) | built | |
| Four judges reading state, with incident records | built | state, owed, truth, ownership |
agenttwin run: five statuses, never a vague score | built | passed, failed, unrunnable, crashed, invalid |
scaffold: a new agent's world and scenarios from its spec | built | |
coverage and scaffold --pairwise | built | intent × state × persona × fault, and state changes |
| Generated attacks from PyRIT, AgentDojo and templates | built | 180 cases held on both reference agents (scripted model) |
| Checked against an outside benchmark | built | 2,472 τ²-bench runs in three domains; every disagreement has a stated cause |
| The four gates, as one command | built | in clean-ai-engineering; the reference agent passes all four |
| Reliability over repeats (pass^k) | partly | in the lab's day report; nightly live-model runs designed, not running |
| Coverage map against the Assurance Catalog | partly | tests name the checks they prove; the 'owed vs proved' gate exists for harness parts, not yet for catalog checks |
| Shadow mode: the stand-in checked against a real system | partly | designed; every world today says verified_against: null |
| Soft graders for tone and completeness (DeepEval or Inspect) | to build | planned |
| Model-driven red team (promptfoo) | to build | planned |
| Realistic data volume (SDV) and network faults on real stores (Toxiproxy) | to build | planned; Faker is in the lab |
| Record and replay of real calls (VCR.py) | to build | planned |
| Agents that do not use MCP, or query a database directly | later | DNS interception and a real database per run are designed |
| A customer who acts with their own tools (τ² dual control) | later | being evaluated |
| Dashboards, a UI, managed cloud runs | later | not planned: a library and a command line |
Can we really reach 100%?
Is 100% coverage achievable?
It depends on what the 100% is of.
| 100% of… | Achievable? | Why |
|---|---|---|
| Owed checks accounted for (proved, or a signed gap) | yes | It is a finite list with ids. This should be the gate, and it is for harness parts already. |
| Owed checks proved by a test before release | nearly | Some checks can only be proved in production: latency at real load, region enforcement, autonomy widened on measured agreement. They are proved at their own stage, by monitoring, not by a simulator. |
| Every rule in the spec (each tool, edge, state change, promise) | yes | Generated from the spec by scaffold and checked by coverage. |
| Every pair of intent, state, persona and fault | yes | Pairwise selection plus a top-up for pairs the library can't reach (it left 187 of 468 for the shop agent; we fill them). |
| Everything that could ever happen | no | No one can promise that, for any software. The honest claim is the four rows above, plus a rule that every miss in production becomes a test. |
Who else is building something like this?
Who else builds this
Five kinds of tool test agents in a simulation. Each is good at one part.
| Kind | Examples | What they simulate | How they judge | Where they lead |
|---|---|---|---|---|
| Conversation simulators | LangWatch Scenario (open source), Azure AI Evaluation Simulator, Maxim, Coval (voice) | the user: a model plays a customer with a persona | mostly a model reading the transcript | realistic customers and voice; dashboards |
| Red-team scanners | Azure AI Red Teaming Agent (wraps PyRIT), promptfoo, garak | an attacker | attack success, scored by evaluators | breadth of attacks |
| Tool simulators | AWS Strands ToolSimulator (open source) | the tools: a model writes each reply, with shared state | goal-success evaluators | covers tools nobody described |
| Simulation sandboxes (commercial) | Veris, Collinear, Patronus | the whole environment, often for training agents (RL) | rubrics, verifiers, model graders | any black-box agent, scale, a managed platform |
| Research worlds | τ²-bench, AgentDojo, AppWorld | one fixed, hand-coded domain | checks on the end state | proven, published benchmarks |
From each tool's public documentation, October 2026. Products change quickly; check before relying on a detail.
How is it different from LangWatch, or anything similar?
How it differs from LangWatch and others
In one line: LangWatch Scenario simulates the customer and asks a model whether the chat went well. AgentTwin simulates the whole world from the agent's own spec and checks what changed in it. They are not rivals: AgentTwin uses LangWatch Scenario to play the customer, and still judges by state.
| AgentTwin | LangWatch Scenario | AWS ToolSimulator | Veris (commercial) | |
|---|---|---|---|---|
| Where the rules come from | the agent's own spec, cited; a world may not restate them | your test script | a model, steered by your prompt | the vendor's mocks and the scenario |
| How a tool behaves | deterministic stand-in that refuses as the real system would | your real tools or your mocks | a model writes each reply | model-powered mocks |
| How a run is judged | four judges reading the world's state | a model judge reading the conversation, plus your checks | goal-success evaluators | verifiers plus model graders |
| Something owed, never done | yes: the 'owed' judge, from the spec | only if the judge notices | no | not stated |
| Approvers and desks on a clock | yes | no | no | simulated users |
| Faults (lost reply, stale read) | yes | no | possible, through the model | not stated |
| Says how faithful it is | yes, per world; shadow mode designed | no | no | seed-pinned, reproducible |
| World generated from the spec | yes (scaffold) | no | no | built by the vendor |
| Realistic customers, personas, voice | basic (it borrows LangWatch for this) | strong; voice too | - | strong (claimed) |
| Any agent, unmodified | MCP agents only, today | via an adapter | Strands agents | yes, by DNS interception |
| UI, dashboards, scale | none | LangWatch platform | AWS | yes |
| Licence | Apache 2.0 | Apache 2.0 | open source | commercial |
Where others are ahead: realistic customers and voice (LangWatch, Azure, Maxim, Coval), breadth of attacks (Azure Red Teaming Agent, promptfoo), testing any agent without changes, scale and dashboards (Veris), and training environments for reinforcement learning (Veris, Collinear, Patronus). AgentTwin borrows the first two rather than competing.
What we found nobody else doing: a world derived from the agent's own spec; a judge for things owed but never done; approvers and desks acting on a clock; and each world stating how faithful it is.
What limits are we facing?
Our limits today
- The tests are only as complete as the spec. If a rule is missing from the spec, no scenario will look for it. Five 'generation runs' (build an agent from the spec alone, see what goes wrong) keep finding spec gaps: run 5 found 7 missing statements and 1 contradiction. A model-played customer found one too: a customer said 'my jacket is lost', the rule only matched 'lost in transit', and nobody was fetched.
- Stand-ins are hand-faithful, not proven faithful. Every world declares what it is faithful about, but none has been checked against a real system yet (shadow mode is designed, not running).
- Scenarios run once, with a scripted model. Live models are random; one pass proves little. pass^k (the same scenario k times, all must pass) runs in the lab only; nightly live runs are designed.
- MCP agents only. An agent that calls REST APIs or a database directly cannot be tested yet. The fixes (DNS interception; a real database per run) are designed.
- Worlds are small and clean. A handful of rows on each rule's edge. Messy, high-volume data (SDV) is planned.
- Soft qualities are not judged. Tone, helpfulness and completeness need model graders (DeepEval or Inspect); not plugged in yet. AgentTwin judges what changed, not how nice it sounded.
- No platform. No UI, dashboards or managed runs. It is a library, a command line and CI. Fine for engineers; a gap for a buyer who wants a product.
- One builder, two reference agents. Proved on a shop agent, a motor-claims agent and three τ²-bench agents we did not build. Not yet used by an outside team.
How does it use open source?
The open-source tools it uses
AgentTwin is an assembler: wherever a proven open-source tool already does a job well, it uses it, behind a seam, and builds only what nobody does.
The full list, opted in and out, with a short card for each tool (what it is, input, processing, output, team effort, why): the open-source tools behind AgentTwin.
| Job | Tool | Status |
|---|---|---|
| Model-played customer | LangWatch Scenario's user simulator (only that part) | in use |
| Attack text and goals | PyRIT (Microsoft), AgentDojo (ETH Zürich) | in use |
| Edge values | Hypothesis, hypothesis-jsonschema | in use |
| Which scenarios to run | allpairspy, plus our own state-machine walker | in use |
| Test runner | pytest | in use |
| Data, schedule, clock for long runs | Faker, SimPy, time-machine (in the Lab) | in use |
| Reliability measure | pass^k, the idea from τ²-bench | in the Lab |
| Soft graders | DeepEval or Inspect | planned |
| Model-driven red team | promptfoo | planned |
| Realistic data volume | SDV | planned |
| Faults and real systems | Toxiproxy, Testcontainers, VCR.py | planned |
| Tools the spec doesn't cover (read-only only) | AWS Strands ToolSimulator | evaluating |
| Agents without MCP | mitmproxy (DNS interception) | later |
FAQ
Questions people ask
Is AgentTwin a digital twin?
Not in the usual sense. A digital twin copies a real machine to watch and predict it. AgentTwin copies the world around an AI agent (customers, systems, data, approvers, faults, time) and keeps the agent itself real, so it can be tested before it meets real people.
Does AgentTwin test the model?
No. It tests the whole agent as built: model, prompts, harness, checks and tools. The model can be fooled in a test and the test can still pass, if nothing harmful reached the world.
Why judge the world's state instead of the conversation?
Because agents can say the right thing and do the wrong thing, or the reverse. 'Your refund is on its way' is only true if a refund exists. The world knows; the transcript does not.
Can I use it on my agent?
If your agent calls its tools over MCP and you can write its spec, yes: scaffold the world from the spec, write one binding function, run the gates. It is open source under Apache 2.0.
Does it replace LangWatch, promptfoo or DeepEval?
No. It uses them. LangWatch Scenario plays the customer, PyRIT and AgentDojo supply attacks, and DeepEval or Inspect are planned for soft grading. AgentTwin adds the spec-driven world and the judges that read it.
Can't AI just generate every test case, like for a calculator?
It can generate inputs and run every line of code. It can't try every input (two 64-bit numbers alone make 3.4 × 10^38 pairs), and it can't know the right answer unless something states it. Maths states it for a calculator; for an agent, the written spec has to, which is why AgentTwin starts there.
Is 100% coverage possible?
100% of what an agent owes can be accounted for: each check proved by a test or signed off as a known gap. 100% of everything that could happen cannot, and nobody can promise that. See the section on coverage.
Next
Where to go next
See where it sits in testing: AgentTwin scenarios and spec coverage in the Agent Harness tutorial.
See an agent work: the Agent Harness Simulator.
The data-engineering sibling: DataTwin, explained, the same idea pointed at a data platform.
The research: the AgentTwin Lab, a world that keeps running for 30 simulated days. Join: the AgentTwin project. Code: github.com/dataagentsai/agenttwin (Apache 2.0).