Home › Learn › AgentTwin, explained › Open-source tools
AgentTwin · the full tool list · 38 tools
The open-source tools behind AgentTwin
AgentTwin is an assembler: where a proven open-source tool already does a job, it uses that tool behind a seam and builds only what nothing else does. Here is every tool we use, plan to use, are weighing, or turned down, each in one short card: what it is, what goes in, what comes out, how much work the input is for your team, and why we decided what we decided.
The decisions at a glance
| Decision | Means | Tools |
|---|---|---|
| In use | opted in and working today | 12 |
| Opt in next | chosen; not plugged in yet | 7 |
| Evaluate | promising; must prove itself first | 6 |
| Later | right tool, not needed yet | 5 |
| Cloud binding | the managed version on one cloud, behind the same seam | 2 |
| Opt out | turned down, for the reason given | 6 |
| Tool | Job | Decision | Why |
|---|---|---|---|
| pytest | Run the tests | In use | The default in Python; a second runner would split the evidence. |
| Inspect (UK AI Security Institute) | Run the tests | Evaluate | Out as a runner (pytest already does that); still a candidate for soft grading. |
| pytest-cov | Run the tests | Opt in next | Cheap and useful as a ratchet; misleading as a goal, so it is never the gate. |
| allpairspy | Choose which scenarios to run | In use | Works, but left 187 of 468 possible pairs uncovered for the shop agent; our own code fills the rest. |
| PICT (Microsoft) | Choose which scenarios to run | Evaluate | Worth a test against allpairspy's 187 misses before switching. |
| GraphWalker | Choose which scenarios to run | Opt out | Our state machines are already in the spec; a Java tool for one algorithm would be a heavy dependency. |
| Hypothesis | Find the edge values | In use | Mature, free, and already a dependency. |
| hypothesis-jsonschema | Find the edge values | In use | Test-only dependency; never shipped. |
| Schemathesis | Find the edge values | Later | Needed only when non-MCP agents arrive. |
| Mutation testing (the idea) | Test the tests | Opt in next | We will build a small version that changes rules and checks, because the tools below change program text. |
| mutmut | Test the tests | Opt out | An agent's important rules live in its spec and harness settings, not in arithmetic; mutating Python text would mostly produce noise. |
| Stryker | Test the tests | Opt out | Wrong languages for us; the idea is what we take. |
| LangWatch Scenario (user simulator only) | Play the people | In use | We use only the simulated user. Its judge is out: we judge by state, not by words. No cloud account, nothing uploaded. |
| DeepEval Synthesizer + Conversation Simulator | Play the people | Opt in next | Same family as the soft graders we may adopt; runs on a free hosted model. |
| τ²-bench dual control | Play the people | Evaluate | Needed for telecom-style agents; not for the current ones. |
| Azure AI Evaluation Simulator | Play the people | Cloud binding | Used only when a project runs on Azure. |
| PyRIT (Microsoft) | Attack the agent | In use | Only its fixed, free converters: its model-driven parts need a paid model or a download. |
| AgentDojo (ETH Zürich) | Attack the agent | In use | Copied in as data: installing the package would pull in five model vendors' SDKs. |
| promptfoo red team | Attack the agent | Opt in next | Planned behind the same attack seam; AgentTwin keeps judging by state. |
| Giskard | Attack the agent | Evaluate | Not checked against our needs yet. |
| garak (NVIDIA) | Attack the agent | Opt out | It tests the model, not the agent around it. AgentTwin's question is whether anything harmful reached the world. |
| DeepTeam | Attack the agent | Opt out | Overlaps promptfoo; one red-team generator is enough. |
| Azure AI Red Teaming Agent | Attack the agent | Cloud binding | Used only on Azure projects. |
| DeepEval (or Inspect) graders | Grade soft qualities | Opt in next | One of the two will be chosen; soft scores never override a state judge. |
| Faker | Make test data | In use | Small, free, and already in the Lab. |
| SDV (Synthetic Data Vault) | Make test data | Later | Every new agent starts with no real data, so it comes later. |
| time-machine | Control time | In use | In use in the Lab. |
| SimPy | Control time | In use | A tested engine; no reason to write our own. |
| Toxiproxy (Shopify) | Faults and real systems | Opt in next | Planned for real mode. |
| Testcontainers | Faults and real systems | Evaluate | Per-run namespaces already do this; adopt if CI needs its own. |
| VCR.py / pytest-recording | Faults and real systems | Opt in next | Planned; AgentTwin adds marking stale cassettes. |
| Saleor (and similar) | Faults and real systems | In use | Borrowed, never built; shadow checks are not routine yet. |
| WireMock / Prism | Faults and real systems | Opt out | A canned mock would cancel a shipped order and the test would pass while proving nothing. |
| AWS Strands ToolSimulator | Faults and real systems | Evaluate | Never for a system whose rules matter: a model can be talked into allowing what the real system refuses. |
| PostgreSQL per run + pgaudit + wal2json | Faults and real systems | Later | No such agent in scope yet. |
| mitmproxy (DNS interception) | Reach more agents, and share | Later | Designed; comes after the current agents. |
| OpenEnv (Meta + Hugging Face) | Reach more agents, and share | Later | A distribution route, not a dependency. |
| pass^k (from τ²-bench) | Reach more agents, and share | In use | In the Lab's day report; nightly runs designed. τ²-bench's own 165 tasks are not used as a dependency. |
What AgentTwin builds itself, because no tool does it: the world format, the stand-ins that enforce the agent's spec, the four judges that read the world's state, the switch between mock, replay, real and shadow modes, and (next) mutation testing over the spec's rules.
'Team effort for input' means work for the team testing an agent, beyond writing the agent's spec. Most inputs are generated from the spec, so the effort is often none. Tools that need a model run on a free hosted open-weight model where possible.
Each tool, in one card
Run the tests
pytest In use
The standard Python test runner.
- Job here
- Runs every scenario and test, and reports which passed.
- Input
- Test files, and AgentTwin's scenario files.
- Processing
- Finds the tests, runs them, collects results.
- Output
- Pass or fail per test, as a JUnit report the gates read.
- Team effort for input
- None: the team already writes tests this way.
- Contributes
- One runner for unit tests and scenarios alike, so one report holds everything.
- Why this decision
- The default in Python; a second runner would split the evidence.
Inspect (UK AI Security Institute) Evaluate
An open framework for running AI evaluations.
- Job here
- Considered as a runner (turned down) and as a grader (still open).
- Input
- Evaluation tasks: a dataset, a solver, a scorer.
- Processing
- Runs the tasks against a model or agent and scores them.
- Output
- Scores and logs per sample.
- Team effort for input
- Medium: each grader is a task to write.
- Contributes
- Well-built model graders, if we pick it over DeepEval.
- Why this decision
- Out as a runner (pytest already does that); still a candidate for soft grading.
pytest-cov Opt in next
Measures which lines of code the tests ran (line coverage).
- Job here
- A floor that may only go up, never a target.
- Input
- The agent's code and its test run.
- Processing
- Records each line executed during tests.
- Output
- A percentage and a list of lines never run.
- Team effort for input
- None: one plugin flag.
- Contributes
- Spots code nothing touches. Says nothing about whether the answers were right.
- Why this decision
- Cheap and useful as a ratchet; misleading as a goal, so it is never the gate.
Choose which scenarios to run
allpairspy In use
A small Python library for pairwise testing: choose a few combinations that still cover every pair of values.
- Job here
- Picks scenarios across intent × state × customer type × fault.
- Input
- Lists of values per dimension, plus a rule for which can't go together.
- Processing
- Builds a short list of combinations where every pair appears at least once.
- Output
- A list of combinations, each a scenario to write.
- Team effort for input
- None: AgentTwin reads the dimensions from the spec.
- Contributes
- Hundreds of meetings between two things covered by tens of scenarios, not thousands.
- Why this decision
- Works, but left 187 of 468 possible pairs uncovered for the shop agent; our own code fills the rest.
PICT (Microsoft) Evaluate
Microsoft's pairwise tool, the industry standard, with first-class 'these can't go together' rules.
- Job here
- A possible replacement for allpairspy.
- Input
- A model file: parameters, values, and constraints.
- Processing
- Same job as allpairspy, with stronger constraint handling.
- Output
- A list of combinations.
- Team effort for input
- None: AgentTwin would write the model file from the spec.
- Contributes
- Fewer uncovered pairs, less of our own top-up code.
- Why this decision
- Worth a test against allpairspy's 187 misses before switching.
GraphWalker Opt out
A model-based testing tool (Java): draw a system's states as a graph and it walks every arrow.
- Job here
- Its technique is used; the tool is not.
- Input
- A graph of states and the moves between them.
- Processing
- Finds paths that cross every move at least once.
- Output
- Test paths.
- Team effort for input
- -
- Contributes
- The idea behind AgentTwin's own state-machine walker.
- Why this decision
- Our state machines are already in the spec; a Java tool for one algorithm would be a heavy dependency.
Find the edge values
Hypothesis In use
Property-based testing for Python: you state a rule that must always hold, it invents hundreds of inputs to break it.
- Job here
- Edge values beyond the ones the spec declares.
- Input
- A property, such as 'a refused write changes nothing', and the shape of valid inputs.
- Processing
- Generates inputs, runs the check, and when one fails shrinks it to the smallest failing example.
- Output
- A failing example, or confidence the property held.
- Team effort for input
- Low: the spec's own rules supply most properties.
- Contributes
- Finds odd inputs nobody would think to type.
- Why this decision
- Mature, free, and already a dependency.
hypothesis-jsonschema In use
Turns a JSON schema into Hypothesis inputs.
- Job here
- Proves each stand-in keeps its own promises: every answer matches its declared shape.
- Input
- Each tool's input and output schema.
- Processing
- Generates valid tool inputs and checks every answer.
- Output
- Failures where a stand-in answers outside its schema.
- Team effort for input
- None: the schemas come from the spec.
- Contributes
- Guards the stand-ins themselves; it found a real bug in the claims stand-in.
- Why this decision
- Test-only dependency; never shipped.
Schemathesis Later
Hypothesis for web APIs: reads an OpenAPI description and fires generated requests at the API.
- Job here
- Checking a stand-in against a real REST API's contract.
- Input
- An OpenAPI (or GraphQL) description of the API.
- Processing
- Generates requests from it and checks the responses.
- Output
- Responses that break the contract.
- Team effort for input
- Low: if the API already publishes OpenAPI.
- Contributes
- Keeps stand-ins honest once we test agents that call REST APIs.
- Why this decision
- Needed only when non-MCP agents arrive.
Test the tests
Mutation testing (the idea) Opt in next
Break the system on purpose in small ways and see if any test notices. A break nobody notices is a missing test.
- Job here
- AgentTwin's planned answer to 'are the tests enough?': remove a check, loosen a rule, drop the idempotency key, then run the suite.
- Input
- The agent's spec and harness settings, and its scenario suite.
- Processing
- Makes many slightly broken copies ('mutants') and runs the suite on each.
- Output
- A list of surviving mutants: each one a named test that is missing.
- Team effort for input
- None: the breaks are generated from the spec's own rules.
- Contributes
- The only item on this page that measures how good the suite is, not just how big it is.
- Why this decision
- We will build a small version that changes rules and checks, because the tools below change program text.
mutmut Opt out
A mutation-testing tool for Python. It changes your code (a < becomes <=, a + becomes -) and reruns the tests.
- Job here
- Shown here to explain the idea; not used as the tool.
- Input
- Python source code and its tests.
- Processing
- Mutates the code one change at a time and reruns the tests.
- Output
- Which changes the tests missed.
- Team effort for input
- None
- Contributes
- Measures unit tests well.
- Why this decision
- An agent's important rules live in its spec and harness settings, not in arithmetic; mutating Python text would mostly produce noise.
Stryker Opt out
The same idea for JavaScript, TypeScript, C# and Scala.
- Job here
- None.
- Input
- Source code and tests.
- Processing
- Mutates and reruns.
- Output
- Surviving mutants.
- Team effort for input
- -
- Contributes
- -
- Why this decision
- Wrong languages for us; the idea is what we take.
Play the people
LangWatch Scenario (user simulator only) In use
An open-source agent-testing library. A model plays the user with a persona, and a model judge scores the chat.
- Job here
- Plays the customer when a scripted one is too predictable.
- Input
- A persona and a situation, both taken from the scenario.
- Processing
- A model writes the customer's next message, turn by turn, until done or out of turns.
- Output
- A realistic conversation. AgentTwin then judges it by the world's state.
- Team effort for input
- None: the scenario already has it.
- Contributes
- Customers who say things nobody scripted. It found a real miss: 'my jacket is lost' never matched the 'lost in transit' rule.
- Why this decision
- We use only the simulated user. Its judge is out: we judge by state, not by words. No cloud account, nothing uploaded.
DeepEval Synthesizer + Conversation Simulator Opt in next
Generates test conversations ('goldens') from documents or from scratch, then simulates them.
- Job here
- Variety: many wordings and situations for each scenario draft.
- Input
- Documents (policies, FAQs) or a short description of the users.
- Processing
- A model writes multi-turn situations with an expected outcome.
- Output
- Draft scenarios, which AgentTwin's loader then checks against the spec.
- Team effort for input
- Low: the agent's policy text already exists.
- Contributes
- Cuts the tester's hardest manual job: phrasing every case like a real person.
- Why this decision
- Same family as the soft graders we may adopt; runs on a free hosted model.
τ²-bench dual control Evaluate
A research benchmark where the customer also has tools and acts on their side (for example on their own phone).
- Job here
- A customer kind that acts, not only talks.
- Input
- Tools for the customer side.
- Processing
- Both sides act on one shared world.
- Output
- Runs where success needs both sides to act.
- Team effort for input
- Medium: customer-side tools must be described.
- Contributes
- Covers support flows where the customer must do something.
- Why this decision
- Needed for telecom-style agents; not for the current ones.
Azure AI Evaluation Simulator Cloud binding
Microsoft's managed user simulator.
- Job here
- The Azure binding of the same seam LangWatch fills.
- Input
- A persona and task.
- Processing
- A model plays the user.
- Output
- Conversations.
- Team effort for input
- Low
- Contributes
- Same job on Azure, for teams already there.
- Why this decision
- Used only when a project runs on Azure.
Attack the agent
PyRIT (Microsoft) In use
Microsoft's open-source red-teaming toolkit: attack seeds, 'converters' that rewrite one attack many ways, attacker models, and scorers.
- Job here
- Supplies attack text planted into fields the spec marks untrusted.
- Input
- An attack goal and the untrusted field (both from the spec).
- Processing
- Rewrites the attack: encodings, ciphers, swapped or invisible characters, jailbreak templates.
- Output
- Many attack variants. AgentTwin judges each by what changed in the world.
- Team effort for input
- None: scaffold writes the goals.
- Contributes
- Breadth of attack wording without writing it by hand. 180 generated cases held on both agents.
- Why this decision
- Only its fixed, free converters: its model-driven parts need a paid model or a download.
AgentDojo (ETH Zürich) In use
A research benchmark of prompt-injection attacks planted in emails, files and notes.
- Job here
- Six attack templates and 27 injection tasks, used as data.
- Input
- The templates and tasks.
- Processing
- Each task is mapped to a goal in the world's terms.
- Output
- Planted attacks, judged by state.
- Team effort for input
- None
- Contributes
- Proven attack shapes from published research.
- Why this decision
- Copied in as data: installing the package would pull in five model vendors' SDKs.
promptfoo red team Opt in next
Generates attacks tailored to your app from a short description of its purpose, with plugins per weakness.
- Job here
- A model-driven red team, beyond PyRIT's fixed rewrites.
- Input
- A short 'purpose' paragraph: who the users are, what they may see and do.
- Processing
- Plugins generate attacks per weakness; strategies make them harder.
- Output
- Attacks and a report.
- Team effort for input
- Low: one paragraph, which the spec can supply.
- Contributes
- Attacks aimed at this agent's own rules, not generic ones.
- Why this decision
- Planned behind the same attack seam; AgentTwin keeps judging by state.
Giskard Evaluate
An open-source scanner that probes LLM apps for weaknesses such as injection and made-up answers.
- Job here
- A second opinion scan.
- Input
- The agent wrapped as a function.
- Processing
- Sends generated probes and checks replies.
- Output
- A list of suspected weaknesses.
- Team effort for input
- Low: a small wrapper.
- Contributes
- Quick extra coverage for answer quality.
- Why this decision
- Not checked against our needs yet.
garak (NVIDIA) Opt out
A scanner that probes a language model for weaknesses.
- Job here
- None.
- Input
- A model endpoint.
- Processing
- Fires many probes at the model.
- Output
- A report of model weaknesses.
- Team effort for input
- -
- Contributes
- -
- Why this decision
- It tests the model, not the agent around it. AgentTwin's question is whether anything harmful reached the world.
DeepTeam Opt out
A red-teaming library from the makers of DeepEval.
- Job here
- None.
- Input
- A target and attack types.
- Processing
- Generates and scores attacks.
- Output
- Results.
- Team effort for input
- -
- Contributes
- -
- Why this decision
- Overlaps promptfoo; one red-team generator is enough.
Azure AI Red Teaming Agent Cloud binding
Microsoft's managed red team, built on PyRIT.
- Job here
- The Azure binding of the attack seam.
- Input
- A target.
- Processing
- PyRIT attacks plus Azure scorers.
- Output
- Attack results.
- Team effort for input
- Low
- Contributes
- Same job on Azure.
- Why this decision
- Used only on Azure projects.
Grade soft qualities
DeepEval (or Inspect) graders Opt in next
Model judges with a written rubric, for qualities a rule can't check.
- Job here
- Grades tone, clarity and completeness, which state judges can't see.
- Input
- The conversation and a rubric.
- Processing
- A model scores it against the rubric.
- Output
- A score and a reason.
- Team effort for input
- Low to medium: a rubric per quality.
- Contributes
- Adds 'was it well said?' to 'was it right?'.
- Why this decision
- One of the two will be chosen; soft scores never override a state judge.
Make test data
Faker In use
Generates fake but realistic names, addresses, emails and dates.
- Job here
- A population of customers and orders for long runs (in the Lab).
- Input
- Which fields to fill and how many rows.
- Processing
- Generates values; AgentTwin checks every row against the spec's rules.
- Output
- Rows the world can load.
- Team effort for input
- None
- Contributes
- Volume and variety without real customer data.
- Why this decision
- Small, free, and already in the Lab.
SDV (Synthetic Data Vault) Later
Learns the patterns in real tables and generates new, fake rows with the same patterns.
- Job here
- Realistic messy data at volume.
- Input
- A sample of real data.
- Processing
- Fits a model to the sample and generates look-alike rows.
- Output
- Synthetic tables.
- Team effort for input
- High: needs real data, which a new agent doesn't have, plus privacy review.
- Contributes
- Realistic skew and dirt, once an agent has real history.
- Why this decision
- Every new agent starts with no real data, so it comes later.
Control time
time-machine In use
Moves a Python program's clock forward instantly.
- Job here
- 'Days have passed' without waiting.
- Input
- A target time.
- Processing
- Fakes the clock the code reads.
- Output
- The agent sees the new time.
- Team effort for input
- None
- Contributes
- Tests return windows and expiry in seconds.
- Why this decision
- In use in the Lab.
SimPy In use
A discrete-event simulation library: schedules events on a simulated timeline.
- Job here
- The Lab's world that keeps running: arrivals, shifts, incidents.
- Input
- Rates and a calendar of events.
- Processing
- Fires events in simulated time.
- Output
- A simulated day, week or month.
- Team effort for input
- Low
- Contributes
- Long-running tests of an agent over 30 simulated days.
- Why this decision
- A tested engine; no reason to write our own.
Faults and real systems
Toxiproxy (Shopify) Opt in next
A proxy that adds slowness, drops and errors to network traffic, on command.
- Job here
- Network faults against a real system, not just the stand-in.
- Input
- A real store running behind it.
- Processing
- Sits between agent and store and injects faults.
- Output
- The agent experiences real network trouble.
- Team effort for input
- Medium: needs the real system running.
- Contributes
- Checks fault handling holds outside the simulation.
- Why this decision
- Planned for real mode.
Testcontainers Evaluate
Starts throwaway real services (databases, shops) in Docker for each test.
- Job here
- A clean real system per run, in CI.
- Input
- A Docker image of the system.
- Processing
- Starts it, seeds it, tears it down.
- Output
- A fresh real system.
- Team effort for input
- Medium: Docker images and seeding.
- Contributes
- Real-system tests without shared state.
- Why this decision
- Per-run namespaces already do this; adopt if CI needs its own.
VCR.py / pytest-recording Opt in next
Records real HTTP calls once, then replays them in later tests.
- Job here
- Replay mode: real answers without calling the real system again.
- Input
- One recorded session against the real system.
- Processing
- Saves calls to 'cassettes' and serves them back.
- Output
- Repeatable real responses.
- Team effort for input
- Medium: one recording run, kept fresh.
- Contributes
- Realism at stand-in speed.
- Why this decision
- Planned; AgentTwin adds marking stale cassettes.
Saleor (and similar) In use
An open-source online shop.
- Job here
- The real system the shop agent's stand-in is checked against (shadow mode).
- Input
- A seeded Saleor store.
- Processing
- The same calls go to stand-in and Saleor; answers are compared.
- Output
- Where the stand-in differs from reality.
- Team effort for input
- Medium: run and seed it.
- Contributes
- Turns 'faithful' from a claim into a check.
- Why this decision
- Borrowed, never built; shadow checks are not routine yet.
WireMock / Prism Opt out
Mock servers that return canned answers (Prism builds them from OpenAPI).
- Job here
- None.
- Input
- Request-to-response rules.
- Processing
- Returns the canned answer.
- Output
- Fixed replies.
- Team effort for input
- -
- Contributes
- -
- Why this decision
- A canned mock would cancel a shipped order and the test would pass while proving nothing.
AWS Strands ToolSimulator Evaluate
A model pretends to be a tool and writes its replies.
- Job here
- Only for read-only, low-stakes tools the spec doesn't describe.
- Input
- A tool's schema and a prompt.
- Processing
- A model invents each reply, keeping shared state.
- Output
- Plausible tool replies.
- Team effort for input
- Low
- Contributes
- Covers the long tail cheaply.
- Why this decision
- Never for a system whose rules matter: a model can be talked into allowing what the real system refuses.
PostgreSQL per run + pgaudit + wal2json Later
A real database per run, with logs of everything read and changed.
- Job here
- Testing agents that query a database directly.
- Input
- The app's own database migrations and a mapping to the spec.
- Processing
- Seeds a copy, runs the agent, captures reads and writes.
- Output
- Exactly what the agent read and changed.
- Team effort for input
- Medium to high: the mapping to the spec.
- Contributes
- Reaches agents that bypass tools.
- Why this decision
- No such agent in scope yet.
Reach more agents, and share
mitmproxy (DNS interception) Later
An HTTPS proxy. Pointing an API's web address at it lets tests catch an unmodified agent's calls.
- Job here
- Testing agents that don't use MCP, without changing them.
- Input
- A translator per API, from its calls to world actions.
- Processing
- Intercepts each call and answers from the world.
- Output
- Any agent, unmodified, in the simulation.
- Team effort for input
- High: one translator per API.
- Contributes
- The biggest reach gain; today AgentTwin tests MCP agents only.
- Why this decision
- Designed; comes after the current agents.
OpenEnv (Meta + Hugging Face) Later
A standard way to package environments for training agents.
- Job here
- Export AgentTwin worlds so the research community can use them.
- Input
- An AgentTwin world.
- Processing
- An adapter wraps it in OpenEnv's interface.
- Output
- A shareable environment.
- Team effort for input
- Medium: the adapter is ours.
- Contributes
- Reach, not testing power.
- Why this decision
- A distribution route, not a dependency.
pass^k (from τ²-bench) In use
A measure, not a tool: run the same scenario k times; it passes only if all k pass.
- Job here
- Reliability with live, random models.
- Input
- A scenario and k.
- Processing
- Runs it k times.
- Output
- Pass only if every run passed.
- Team effort for input
- None
- Contributes
- Separates 'worked once' from 'works reliably'.
- Why this decision
- In the Lab's day report; nightly runs designed. τ²-bench's own 165 tasks are not used as a dependency.