Home › Learn › AgentTwin, explained

AI engineering · free guide · with a step-through simulator

AgentTwin, explained

How do you know an AI agent is ready to talk to real customers? AgentTwin puts the real agent in a pretend world (pretend customers, pretend systems, pretend data, things going wrong) and then checks what actually changed in that world, not just what the agent said. This page explains it from zero, lets you play it on a tiny agent, scales it up, and then is honest about what is built, what is not, and who else does this.

Part 1 · The explainer

What it is, in plain words, before any detail.

What problem does it solve?

A flight simulator for AI agents

Pilots don't learn engine failure on a real plane with passengers. They sit in a simulator: the cockpit is real, everything outside it is pretend, and an instructor can make an engine fail, the weather turn, or the radio go silent. Afterwards nobody asks the pilot "did you feel confident?". They look at where the plane ended up.

AgentTwin does the same for an AI agent, such as a customer-support or insurance-claims assistant. The agent is the real one, unchanged. Everything it talks to is pretend: the customer, the order system, the database rows, the manager who approves refunds, the clock, and the faults (a slow system, a lost reply). After each test, judges look at what changed in the pretend world. Did a refund really get issued? Was something promised and never done? Did the agent read another customer's data?

That last point is the heart of it. An agent can say "your refund is on its way" when no refund exists. A judge reading the chat would pass that. A judge reading the world would not.

Is it a digital twin?

AgentTwin vs a digital twin

The name borrows from "digital twin", but it points the other way. A digital twin copies a real thing (a jet engine, a factory) so you can watch it and predict it. AgentTwin keeps the thing real (the agent) and copies the world around it, so you can test it.

A digital twinThe real thinga jet engine,a factory linesensor dataIts virtual copymirrors the thing,minute by minutePredictions, what-ifs“service it in 40 hours”act on the real thingIt copies the THING itself,to watch it and predict it, for its whole life.vsAgentTwinthe simulated world: the twinCustomersscripted or model-playedSystems + datastand-ins with rulesApprovers, deskacting on a clockFaults + timeslow, lost, staleReal agentunchangedJudges read the world's stateIt copies the WORLD AROUND the agent,to test the agent before real people meet it.
Same word, opposite direction. A digital twin copies the thing and leaves the world real. AgentTwin keeps the thing (the agent) real and copies its world.
Digital twinAgentTwin
What is copiedThe thing itself: an engine, a plant, a cityThe world around the agent: customers, systems, data, people, faults, time
What stays realThe world. The thing runs for realThe agent. Its code, prompts and model are unchanged
Where data flowsFrom the real thing into the copy (sensors)From the spec into the world; nothing comes from production
When it is usedIn operation, for the thing's whole lifeBefore release and on every change; nightly once live
The questionWhat is this machine doing, and what will it do?Will this agent do the right thing in every situation it must face?
How close to realAs close as possible, everywhereExact only where a test needs it; each world states what it is faithful about
What comes outPredictions, alerts, what-ifsPass or fail per scenario, evidence, a coverage map, a ship-or-block verdict

What goes in, what happens, what comes out?

Input, processing, output

INPUTS: who gives whatOUTPUTSAgent specfrom the agent's owner:tools, rules, statesWorld + scenariosfrom the tester:rows, faults, storiesOwed checksfrom the Assurance Catalog,by the agent's shapeBindingfrom the agent's team:how to start the agentAgentTwin runs each scenario1 load the world; refuse impossible data2 stand up the systems as MCP tools3 play the customer, approver and desk4 inject faults; move the clock5 let the real agent work, unchanged6 record every call and every reply7 four judges read the world's stateA status per scenariopassed, failed, unrunnable,crashed or invalidRun record + incidentsthe evidence forevery failureCoverage mapeach owed check: proved,accepted gap, or missingGate verdictship, or blocked witha route to the fix
Input, processing, output. Four people or sources feed it; seven steps run per scenario; four things come out.

Agent's owner

Writes the agent spec (called an AOAS): its tools, the rule behind each tool, the states things move through, and what it owes customers.

Tester

Fills the world with a few rows on each rule's edge and writes or generates scenarios: who says what, what goes wrong, what must be true at the end.

The Assurance Catalog

118 public checks an AI application may owe. The agent's shape picks which ones it owes.

Agent's team

Writes one binding function: how to start the agent and point it at the pretend tools.

Part 2 · How it works

From set-up to a verdict, then a tiny agent you can play, then a big one.

What has to exist before any agent is tested?

How the system is set up

Four open repositories, set up once and shared by every agent:

AgentTwin

The library and command line: world format, stand-ins, actors, faults, clock, judges, run, scaffold, coverage.

AI Assurance Catalog

What must be true: 118 checks, 36 owed by every AI app, the rest by shape.

AI Harness Catalog

What must exist around the model call: the harness parts each shape needs.

Clean AI Engineering

The spec formats and the four gates, plus one "yardstick" file per agent that the agent's own team cannot edit.

The yardstick living outside the agent's repository matters: an agent that could bring its own tests could bring easy ones.

I have a new agent. What do I do?

What you do for each agent

  1. Write the agent's spec (AOAS). Its entities, tools, the rule behind each tool, the states and the promises it owes (for example: a refund is owed when a return is received). The agent's owner writes it; it is the single source of the rules.
  2. Say what shape it is. One or more of ten archetypes (a tool-using agent is A6; a deterministic router in front of it adds A5). The shape decides which Assurance Catalog checks it owes.
  3. Run agenttwin scaffold on the spec. It writes a world file, a starter scenario for each tool and rule, a binding stub and a gates file. Most of the work after this is data, not code: the electronics version of the shop agent is a 70-line world file.
  4. Fill the world with rows. A few members, orders or claims that sit on each rule's edge: 7 days late and 8 days late, inside and outside a window. The loader refuses impossible rows (a returned book that is still overdue).
  5. Write the binding. One function that starts the agent and points it at the world's tools: (live, *, wrap, clock, model) -> Subject. The agent never imports AgentTwin; the build fails if it tries.
  6. Run agenttwin coverage, then top up. It lists which combinations of intent, state, persona and fault, and which state changes, no scenario covers yet. scaffold --pairwise writes skeletons for the missing ones.
  7. Run the four gates. tools/gates.py runs the scenarios, checks every owed statement is proved or knowingly accepted, checks the build's structure and the harness parts. One command, one verdict, a route for every failure.

For a given agent, how do we know these are the tests it needs?

How we know which tests a given agent needs

This is the most important question on the page. The answer: nobody picks the tests by taste. The agent's test list is whatever it takes to prove everything the agent owes, and what it owes is written down, item by item, each with an id.

The agent's specevery tool, rule, stateand promise it owesThe Assurance Catalogchecks owed bythe agent's shapeThe Harness Catalogparts its shapemust haveAttack corporaPyRIT, AgentDojo,built-in templatesProduction missesa miss in the wildbecomes a caseThe owed listevery statement thisagent must prove,each with an ideach one needsat least one passing testthat names it (discharges: [id])or an accepted gapwith a reason, an owner, a review dateor 'not applicable'with the reason written downanything else is 'not exercised', and the gate fails
How a given agent's test list is decided: not by taste, but by what it owes. A missing test shows up as a named gap.

Where each owed item comes from:

Every test then says which items it proves (discharges: [AAC-0047, op:renew]). The gate builds a map: each owed item is proved, an accepted gap (with a reason, an owner and a review date), not applicable (with a reason), or not exercised. Any "not exercised" item fails the gate, with its name. So the answer to "are these the right tests?" is checkable: the right tests are the ones that leave nothing unaccounted for.

The honest catch. This proves the tests are complete against what is written down. If a rule is missing from the spec, nothing asks for its test. Three things push against that: generation runs (build an agent from the spec alone and route every surprise back into the spec), a model-played customer who says things nobody scripted, and production misses becoming cases.

Writing the world and scenarios is a lot for a tester. How do we stop them missing things?

How we help the tester not miss

The tester should never start from a blank page, and should never have to remember the common cases. Six layers of help, from what the spec gives for free to what only production can teach:

LayerWhat it does for the testerStatus
1 · From the specscaffold writes the world's rows on every rule's edge, and scenarios for each tool (allowed and refused), each refusal, a stranger, and an attack per untrusted fieldbuilt
2 · What's missingcoverage lists uncovered pairs of intent, state, customer type (6 built in) and fault (8 kinds), and state changes never walked; --pairwise writes starters for thembuilt
3 · The common packScenarios every agent needs whatever its business: the same message twice, a long chat that should fetch a person, an approver who never answers, the kill switch, the model provider down, never quoting a figure no tool gave. Sized by the catalogs (118 assurance checks, 118 harness parts), not by a guess; each template fills itself from the specto build they exist, hand-written, in two agents
4 · VarietyMany wordings, languages and moods per scenario, from a model-played customer (LangWatch), generated conversations (DeepEval) and attack rewrites (PyRIT)partly
5 · SearchAn adversarial customer: a model given the spec and a goal ("get a refund you aren't owed") that keeps trying, with the judges deciding if it won. Finds what nobody thought to writeto build
6 · Test the testsMutation testing: break the agent on purpose (remove a check, loosen 7 days to 8, drop the idempotency key) and see if any scenario notices. A break nobody notices is a named missing testto build, first

The gate stays the backstop: anything owed and unproved is named and blocks the release.

Hasn't AI already solved this? The calculator question

"If I want to test a calculator API, can't I generate all the test cases automatically and cover 100%?" Partly. Three different things hide inside "100%":

"100%" of…For a calculator
Every line of code run (code coverage)easy Tools like Pynguin (Python) and EvoSuite (Java), and AI coding assistants, generate tests that run every line. But a test can run every line of a buggy divide and still pass.
Every possible inputimpossible Two 64-bit numbers make about 3.4 × 1038 pairs. At a billion tests a second that is about 1022 years; the universe is about 1010 years old.
The right answer for each inputthe hard part Generated tests often just record what the code does today, bugs included. Someone must say what is right. Testers call this the oracle problem.

A calculator is the easy case, because maths gives an oracle for free: compare with a trusted library, or check rules that must always hold (a + b = b + a; (a + b) − b = a). Pick inputs smartly (zero, negatives, huge numbers, the edges) and a machine can test a calculator very well, without trying every input.

An agent has no free oracle. Whether "refund this order" is right depends on the shop's rules, the order's state, who is asking and what was promised. AI can generate endless conversations, but it can't know which outcome is correct unless the rules are written down. That is why AgentTwin starts from the agent's spec: the spec is the oracle, and the four judges apply it to the world's state.

How the software world handles it

Nobody has fully solved "complete tests". Mature teams combine six habits, and each has an AgentTwin counterpart:

HabitSoftware exampleIn AgentTwin
Derive cases from the spec mechanicallyEquivalence classes, boundary values, decision tables, state-transition testingbuilt scaffold and the state walker
Cover every pair, not every combinationPICT, allpairspybuilt
A standard checklist per kind of systemOWASP ASVS for web security; ISO 25010 quality traitspartly the Assurance Catalog, not yet turned into templates
Trace every requirement to a testAviation (DO-178C), medical devices (IEC 62304)built discharges and the gate
Let a machine hunt for failuresFuzzing, property-based testing (Hypothesis, Schemathesis)partly Hypothesis for edges; no adversarial customer yet
Test the testsMutation testing (mutmut for Python, Stryker for JavaScript and C#)to build

How LangWatch handles it

In LangWatch Scenario, developers write scenarios in code: a model plays the user, a model judge scores the chat against criteria. An autopilot mode lets the simulated user improvise towards a goal, and LangWatch describes generating multi-step scenarios from a list of business policies, including where two policies overlap. Its answer is breadth: let a model write many scenarios. What it doesn't publish is a count of what's missing. AgentTwin's difference is that it counts the gaps; the idea worth borrowing is drafting scenarios where two rules meet.

What PyRIT-style tools add

PyRIT, Microsoft's red-teaming kit, is built from four parts: seeds (attack goals), converters (rewrite one attack many ways), attacker models (read the reply and try again, turn by turn) and scorers. Two of those ideas reach beyond security. Converters become "one scenario, many wordings, languages and moods", which takes the phrasing work off the tester. The attacker model becomes the adversarial customer in layer 5. PyRIT knows nothing of your business rules, so AgentTwin supplies the goals from the spec and does the scoring with its judges.

Every tool named here, explained in one card each: the open-source tools behind AgentTwin.

How are the pretend systems made?

How the pretend systems are made

Not by writing mocks by hand, and not by asking a model to play the system. AgentTwin reads the agent's spec and projects each system as a stand-in that holds state and enforces the spec's rules. Ask it to cancel a shipped order and it refuses, the way the real system would.

A canned mock that answers "OK" to everything is not enough: it would cancel the shipped order, the agent's test would pass, and it would prove nothing. A model playing the system is not enough either: it can be talked into allowing what the real system refuses.

Kind of thing the agent talks toHow AgentTwin treats it
A data storeThe world's state, shown as rows
An API with a contractAnswers from the declared contract
A tool server (MCP)Tool results: the cleanest seam there is
The customerAn actor (scripted, state machine, or model-played), not a system
A person who approvesAn actor plus a queue, acting between turns on a clock
A plain calculationRun it for real. Mocking it adds risk and removes nothing

Each world also says what it is faithful about and what it is not. The clothing shop's world is exact about order status, the return window and which actions can't be undone, and openly not faithful about prices, stock or carriers. A test may only check things inside that list.

Where does each test's data come from?

How test data is made, per case

Can I see one run, step by step?

Play it: a tiny library agent

The simplest agent we could pick: a library assistant with three tools (get_loan, renew, note_due_date) and one rule: a loan can be renewed only if it is open, at most 7 days late, and yours. AgentTwin's own tests use this library world. Asha is the simulated member. Pick a scenario, run it against a well-built agent and then a faulty one, and watch which judge catches what.

Run it against:

What happens, step by step
    The world's state: the loan table
    loanmemberbookstatusdays latenote
    The scenario file for this case

    These runs are written for this page to show the idea; they are not recorded from a live model. The scenario files are simplified from the real format.

    What changes for a real agent?

    Scaling up to a bigger agent

    Nothing new in the method; more of everything. The same steps (spec, world, binding, scenarios, gates) and the same four judges, applied to the clothing-shop support agent and the motor-insurance claims assistant.

    Library (this page)Clothing-shop agentMotor-claims assistant
    Tools3 (get_loan, renew, note_due_date)5 (get_order, cancel_order, open_return_request, change_address, issue_refund)claims tools, payout, document upload
    Rules1 (open, 7 days late or less)return window, final sale, cancel before shipping, address before shipping, refund needs approvalcover limits, payout limit, consent, injury escalation
    People in the worldone membercustomer + reviewer who approves refundspolicyholder, approver, human desk with a 30-minute hand-off expiry
    Faultslost replychannel errors, stale reads, model provider downgateway refusing calls (429, 503), slow systems
    Timenonedays pass for return windowsthe clock jumps 1,860 s per turn to expire a hand-off
    Attacks1 planted note20 generated notes per run; 180 cases across both agentsinjection in a claim note and from the user
    Scenarios5 on this page4929 in the shared yardstick + 17 local

    Two things only show up at this size. Time: a hand-off to a person that nobody picks up must expire after 30 minutes, so the world's clock jumps forward between turns. People: a reviewer who approves refunds can say yes, no, never answer, or say yes twice, and each needs its own scenario.

    To show the method is not tuned to our own agents, AgentTwin also judged three τ²-bench agents (airline, retail, telecom) from their published policy and tools alone, on 2,472 recorded runs. On airline and retail it flagged exactly the rule-breaking writes τ²-bench flags (188 and 309), and it also checks whether another customer's record was read, which τ²-bench does not.

    How do we get 100% coverage of the Assurance Catalog?

    Coverage of the Assurance Catalog

    The catalog has 118 checks. 36 are owed by every AI application; the rest depend on its shape. The clothing-shop agent is a tool-using agent behind a deterministic router (shapes A6 and A5), so it owes 62.

    118 checksin the AssuranceCatalogfilter36 core + shapecore: every AI appshape: A5 + A6 here62 owedby the clothing-shopagent51 proveda passing test names each11 not yetno test, no written gapToday: 82% proved. The target is 100% accounted for: proved, or a gap someone signed for.
    The clothing-shop agent against the Assurance Catalog, counted on 30 Oct 2026 from its gate inventory.

    The 11 not yet proved, and what closes each:

    CheckWhat it asksWhat closes it
    AAC-0007Latency within budget at expected concurrencya load test; partly proved only in production
    AAC-0016Scheduled re-run against unchanged inputsa nightly run; designed, not running
    AAC-0044, 0045, 0046, 0048, 0049, 0050Six checks for the router-plus-agent shape (A5): each node alone, both sides of each seam, partial failure, determinism, every branch, end-to-end budgettests exist in spirit; none name these ids yet. The cheapest gap to close
    AAC-0096Cached responses never cross a trust boundarya test once a cache exists; else 'not applicable', written down
    AAC-0097Processing region is enforced and recordeda deployment check, not a simulation
    AAC-0118Autonomy widens only on agreement measured for that decisiononly provable over time, in production

    So "100% coverage" here means one thing: every owed check is either proved by a passing test or signed off as a known gap. The gate already enforces that for harness parts. Extending it to catalog checks is the next step; until then the catalog number is counted, not gated.

    How do we know the tests really ran?

    How we know the tests really ran

    1. The yardstick lives outside the agent. The agent's team can't remove a scenario or bring an easier one.
    2. Five statuses, never a score. "Unrunnable" and "crashed" are not "passed". A scenario that can't run says so.
    3. A generator that plants nothing fails. Until 26 Sept 2026 an injection scenario with nothing planted passed quietly, and 12 declared cases claimed an attack check on zero attacks. Now the runner refuses.
    4. The agent can't see the simulator. If the agent's code imports AgentTwin, the build fails. A system that can see its own simulator produces results that mean nothing.
    5. Every test names what it proves, and the gate counts only passing ones.
    6. CI runs it on every push, and a failing gate blocks the release. Scenario coverage may only go up.

    Are these tests enough to say "ready for production"?

    Enough to call it ready for production?

    No test suite proves an agent safe for every future conversation. What it can prove is that the agent keeps every promise it owes, in every situation we can name, under the faults we know about. Here is the list we think "ready" needs, and where we are:

    For "ready"Status
    All four gates pass (behaviour, features, structure, harness)built the reference agent passes all four
    Every owed check proved or signed off as a gappartly harness parts gated; catalog 51 of 62 for the shop agent
    Each scenario passes k times in a row on the live model (pass^k)partly in the lab; nightly runs designed
    The stand-ins checked against a real system (shadow mode)to build
    Generated attacks hold, judged by statebuilt 180 of 180 held (scripted model)
    In production: canary cases, online rules, and every miss turned into a scenariopartly rules and alerts exist for the claims agent

    Our verdict today: enough to start a watched pilot, not enough to claim "proven ready". The missing pieces are named above.

    Part 3 · As a product

    What exists, what doesn't, who else is here, and where we are weak.

    If this were a product, what's in the box?

    Built, and still to build

    FeatureStatusNote
    World format (AWD), schema and loader that refuses bad worldsbuiltv0.14.0, 29 Oct 2026; 433 passing tests
    Stand-ins served as an MCP tool server, enforcing the spec's rulesbuiltthe core of AgentTwin
    Scripted, state-machine and model-played customersbuiltmodel-played via LangWatch Scenario, an optional extra
    Approver and human desk, acting between turnsbuiltgrant, refuse, never answer, grant twice
    Faults: slow, lost reply, channel error, stale read, provider downbuiltper scenario, at a chosen call
    Clock control (days pass in seconds)built
    Four judges reading state, with incident recordsbuiltstate, owed, truth, ownership
    agenttwin run: five statuses, never a vague scorebuiltpassed, failed, unrunnable, crashed, invalid
    scaffold: a new agent's world and scenarios from its specbuilt
    coverage and scaffold --pairwisebuiltintent × state × persona × fault, and state changes
    Generated attacks from PyRIT, AgentDojo and templatesbuilt180 cases held on both reference agents (scripted model)
    Checked against an outside benchmarkbuilt2,472 τ²-bench runs in three domains; every disagreement has a stated cause
    The four gates, as one commandbuiltin clean-ai-engineering; the reference agent passes all four
    Reliability over repeats (pass^k)partlyin the lab's day report; nightly live-model runs designed, not running
    Coverage map against the Assurance Catalogpartlytests name the checks they prove; the 'owed vs proved' gate exists for harness parts, not yet for catalog checks
    Shadow mode: the stand-in checked against a real systempartlydesigned; every world today says verified_against: null
    Soft graders for tone and completeness (DeepEval or Inspect)to buildplanned
    Model-driven red team (promptfoo)to buildplanned
    Realistic data volume (SDV) and network faults on real stores (Toxiproxy)to buildplanned; Faker is in the lab
    Record and replay of real calls (VCR.py)to buildplanned
    Agents that do not use MCP, or query a database directlylaterDNS interception and a real database per run are designed
    A customer who acts with their own tools (τ² dual control)laterbeing evaluated
    Dashboards, a UI, managed cloud runslaternot planned: a library and a command line

    Can we really reach 100%?

    Is 100% coverage achievable?

    It depends on what the 100% is of.

    100% of…Achievable?Why
    Owed checks accounted for (proved, or a signed gap)yesIt is a finite list with ids. This should be the gate, and it is for harness parts already.
    Owed checks proved by a test before releasenearlySome checks can only be proved in production: latency at real load, region enforcement, autonomy widened on measured agreement. They are proved at their own stage, by monitoring, not by a simulator.
    Every rule in the spec (each tool, edge, state change, promise)yesGenerated from the spec by scaffold and checked by coverage.
    Every pair of intent, state, persona and faultyesPairwise selection plus a top-up for pairs the library can't reach (it left 187 of 468 for the shop agent; we fill them).
    Everything that could ever happennoNo one can promise that, for any software. The honest claim is the four rows above, plus a rule that every miss in production becomes a test.

    Who else is building something like this?

    Who else builds this

    Five kinds of tool test agents in a simulation. Each is good at one part.

    KindExamplesWhat they simulateHow they judgeWhere they lead
    Conversation simulatorsLangWatch Scenario (open source), Azure AI Evaluation Simulator, Maxim, Coval (voice)the user: a model plays a customer with a personamostly a model reading the transcriptrealistic customers and voice; dashboards
    Red-team scannersAzure AI Red Teaming Agent (wraps PyRIT), promptfoo, garakan attackerattack success, scored by evaluatorsbreadth of attacks
    Tool simulatorsAWS Strands ToolSimulator (open source)the tools: a model writes each reply, with shared stategoal-success evaluatorscovers tools nobody described
    Simulation sandboxes (commercial)Veris, Collinear, Patronusthe whole environment, often for training agents (RL)rubrics, verifiers, model gradersany black-box agent, scale, a managed platform
    Research worldsτ²-bench, AgentDojo, AppWorldone fixed, hand-coded domainchecks on the end stateproven, published benchmarks

    From each tool's public documentation, October 2026. Products change quickly; check before relying on a detail.

    How is it different from LangWatch, or anything similar?

    How it differs from LangWatch and others

    In one line: LangWatch Scenario simulates the customer and asks a model whether the chat went well. AgentTwin simulates the whole world from the agent's own spec and checks what changed in it. They are not rivals: AgentTwin uses LangWatch Scenario to play the customer, and still judges by state.

    AgentTwinLangWatch ScenarioAWS ToolSimulatorVeris (commercial)
    Where the rules come fromthe agent's own spec, cited; a world may not restate themyour test scripta model, steered by your promptthe vendor's mocks and the scenario
    How a tool behavesdeterministic stand-in that refuses as the real system wouldyour real tools or your mocksa model writes each replymodel-powered mocks
    How a run is judgedfour judges reading the world's statea model judge reading the conversation, plus your checksgoal-success evaluatorsverifiers plus model graders
    Something owed, never doneyes: the 'owed' judge, from the speconly if the judge noticesnonot stated
    Approvers and desks on a clockyesnonosimulated users
    Faults (lost reply, stale read)yesnopossible, through the modelnot stated
    Says how faithful it isyes, per world; shadow mode designednonoseed-pinned, reproducible
    World generated from the specyes (scaffold)nonobuilt by the vendor
    Realistic customers, personas, voicebasic (it borrows LangWatch for this)strong; voice too-strong (claimed)
    Any agent, unmodifiedMCP agents only, todayvia an adapterStrands agentsyes, by DNS interception
    UI, dashboards, scalenoneLangWatch platformAWSyes
    LicenceApache 2.0Apache 2.0open sourcecommercial

    Where others are ahead: realistic customers and voice (LangWatch, Azure, Maxim, Coval), breadth of attacks (Azure Red Teaming Agent, promptfoo), testing any agent without changes, scale and dashboards (Veris), and training environments for reinforcement learning (Veris, Collinear, Patronus). AgentTwin borrows the first two rather than competing.

    What we found nobody else doing: a world derived from the agent's own spec; a judge for things owed but never done; approvers and desks acting on a clock; and each world stating how faithful it is.

    What limits are we facing?

    Our limits today

    1. The tests are only as complete as the spec. If a rule is missing from the spec, no scenario will look for it. Five 'generation runs' (build an agent from the spec alone, see what goes wrong) keep finding spec gaps: run 5 found 7 missing statements and 1 contradiction. A model-played customer found one too: a customer said 'my jacket is lost', the rule only matched 'lost in transit', and nobody was fetched.
    2. Stand-ins are hand-faithful, not proven faithful. Every world declares what it is faithful about, but none has been checked against a real system yet (shadow mode is designed, not running).
    3. Scenarios run once, with a scripted model. Live models are random; one pass proves little. pass^k (the same scenario k times, all must pass) runs in the lab only; nightly live runs are designed.
    4. MCP agents only. An agent that calls REST APIs or a database directly cannot be tested yet. The fixes (DNS interception; a real database per run) are designed.
    5. Worlds are small and clean. A handful of rows on each rule's edge. Messy, high-volume data (SDV) is planned.
    6. Soft qualities are not judged. Tone, helpfulness and completeness need model graders (DeepEval or Inspect); not plugged in yet. AgentTwin judges what changed, not how nice it sounded.
    7. No platform. No UI, dashboards or managed runs. It is a library, a command line and CI. Fine for engineers; a gap for a buyer who wants a product.
    8. One builder, two reference agents. Proved on a shop agent, a motor-claims agent and three τ²-bench agents we did not build. Not yet used by an outside team.

    How does it use open source?

    The open-source tools it uses

    AgentTwin is an assembler: wherever a proven open-source tool already does a job well, it uses it, behind a seam, and builds only what nobody does.

    The full list, opted in and out, with a short card for each tool (what it is, input, processing, output, team effort, why): the open-source tools behind AgentTwin.

    AgentTwin keeps onlythe world format (AWD)stand-ins that enforce the specfour judges that read statethe mock / replay / real / shadow switchLangWatch Scenariomodel-played customerPyRIT + AgentDojoattack text and goalsHypothesisedge valuesallpairspywhich scenarios to runpytestthe runnerFaker, SimPy, time-machinedata, schedule, clock (lab)DeepEval or Inspectsoft graders: plannedToxiproxy, Testcontainers, VCR.pyreal systems: plannedgreen = in use today · yellow = planned, behind a seam already drawn
    An assembler, not a framework. Each tool sits behind a seam; the core never imports one, so any of them can be swapped.
    JobToolStatus
    Model-played customerLangWatch Scenario's user simulator (only that part)in use
    Attack text and goalsPyRIT (Microsoft), AgentDojo (ETH Zürich)in use
    Edge valuesHypothesis, hypothesis-jsonschemain use
    Which scenarios to runallpairspy, plus our own state-machine walkerin use
    Test runnerpytestin use
    Data, schedule, clock for long runsFaker, SimPy, time-machine (in the Lab)in use
    Reliability measurepass^k, the idea from τ²-benchin the Lab
    Soft gradersDeepEval or Inspectplanned
    Model-driven red teampromptfooplanned
    Realistic data volumeSDVplanned
    Faults and real systemsToxiproxy, Testcontainers, VCR.pyplanned
    Tools the spec doesn't cover (read-only only)AWS Strands ToolSimulatorevaluating
    Agents without MCPmitmproxy (DNS interception)later

    FAQ

    Questions people ask

    Is AgentTwin a digital twin?

    Not in the usual sense. A digital twin copies a real machine to watch and predict it. AgentTwin copies the world around an AI agent (customers, systems, data, approvers, faults, time) and keeps the agent itself real, so it can be tested before it meets real people.

    Does AgentTwin test the model?

    No. It tests the whole agent as built: model, prompts, harness, checks and tools. The model can be fooled in a test and the test can still pass, if nothing harmful reached the world.

    Why judge the world's state instead of the conversation?

    Because agents can say the right thing and do the wrong thing, or the reverse. 'Your refund is on its way' is only true if a refund exists. The world knows; the transcript does not.

    Can I use it on my agent?

    If your agent calls its tools over MCP and you can write its spec, yes: scaffold the world from the spec, write one binding function, run the gates. It is open source under Apache 2.0.

    Does it replace LangWatch, promptfoo or DeepEval?

    No. It uses them. LangWatch Scenario plays the customer, PyRIT and AgentDojo supply attacks, and DeepEval or Inspect are planned for soft grading. AgentTwin adds the spec-driven world and the judges that read it.

    Can't AI just generate every test case, like for a calculator?

    It can generate inputs and run every line of code. It can't try every input (two 64-bit numbers alone make 3.4 × 10^38 pairs), and it can't know the right answer unless something states it. Maths states it for a calculator; for an agent, the written spec has to, which is why AgentTwin starts there.

    Is 100% coverage possible?

    100% of what an agent owes can be accounted for: each check proved by a test or signed off as a known gap. 100% of everything that could happen cannot, and nobody can promise that. See the section on coverage.

    Next

    Where to go next

    See where it sits in testing: AgentTwin scenarios and spec coverage in the Agent Harness tutorial.

    See an agent work: the Agent Harness Simulator.

    The data-engineering sibling: DataTwin, explained, the same idea pointed at a data platform.

    The research: the AgentTwin Lab, a world that keeps running for 30 simulated days. Join: the AgentTwin project. Code: github.com/dataagentsai/agenttwin (Apache 2.0).