Home › Learn › AgentTwin, explained › Open-source tools

AgentTwin · the full tool list · 38 tools

The open-source tools behind AgentTwin

AgentTwin is an assembler: where a proven open-source tool already does a job, it uses that tool behind a seam and builds only what nothing else does. Here is every tool we use, plan to use, are weighing, or turned down, each in one short card: what it is, what goes in, what comes out, how much work the input is for your team, and why we decided what we decided.

Choosewhich scenariosMake datarows, edgesPlay peoplecustomer, approverAttackplanted textFaults, timeslow, lost, daysRunpytestJudgestate + softTest the testsmutationeach job has its own small group of tools below; AgentTwin builds only what none of them does
The jobs a test run needs, in order. Every tool below does one of them.

The decisions at a glance

DecisionMeansTools
In useopted in and working today12
Opt in nextchosen; not plugged in yet7
Evaluatepromising; must prove itself first6
Laterright tool, not needed yet5
Cloud bindingthe managed version on one cloud, behind the same seam2
Opt outturned down, for the reason given6
ToolJobDecisionWhy
pytestRun the testsIn useThe default in Python; a second runner would split the evidence.
Inspect (UK AI Security Institute)Run the testsEvaluateOut as a runner (pytest already does that); still a candidate for soft grading.
pytest-covRun the testsOpt in nextCheap and useful as a ratchet; misleading as a goal, so it is never the gate.
allpairspyChoose which scenarios to runIn useWorks, but left 187 of 468 possible pairs uncovered for the shop agent; our own code fills the rest.
PICT (Microsoft)Choose which scenarios to runEvaluateWorth a test against allpairspy's 187 misses before switching.
GraphWalkerChoose which scenarios to runOpt outOur state machines are already in the spec; a Java tool for one algorithm would be a heavy dependency.
HypothesisFind the edge valuesIn useMature, free, and already a dependency.
hypothesis-jsonschemaFind the edge valuesIn useTest-only dependency; never shipped.
SchemathesisFind the edge valuesLaterNeeded only when non-MCP agents arrive.
Mutation testing (the idea)Test the testsOpt in nextWe will build a small version that changes rules and checks, because the tools below change program text.
mutmutTest the testsOpt outAn agent's important rules live in its spec and harness settings, not in arithmetic; mutating Python text would mostly produce noise.
StrykerTest the testsOpt outWrong languages for us; the idea is what we take.
LangWatch Scenario (user simulator only)Play the peopleIn useWe use only the simulated user. Its judge is out: we judge by state, not by words. No cloud account, nothing uploaded.
DeepEval Synthesizer + Conversation SimulatorPlay the peopleOpt in nextSame family as the soft graders we may adopt; runs on a free hosted model.
τ²-bench dual controlPlay the peopleEvaluateNeeded for telecom-style agents; not for the current ones.
Azure AI Evaluation SimulatorPlay the peopleCloud bindingUsed only when a project runs on Azure.
PyRIT (Microsoft)Attack the agentIn useOnly its fixed, free converters: its model-driven parts need a paid model or a download.
AgentDojo (ETH Zürich)Attack the agentIn useCopied in as data: installing the package would pull in five model vendors' SDKs.
promptfoo red teamAttack the agentOpt in nextPlanned behind the same attack seam; AgentTwin keeps judging by state.
GiskardAttack the agentEvaluateNot checked against our needs yet.
garak (NVIDIA)Attack the agentOpt outIt tests the model, not the agent around it. AgentTwin's question is whether anything harmful reached the world.
DeepTeamAttack the agentOpt outOverlaps promptfoo; one red-team generator is enough.
Azure AI Red Teaming AgentAttack the agentCloud bindingUsed only on Azure projects.
DeepEval (or Inspect) gradersGrade soft qualitiesOpt in nextOne of the two will be chosen; soft scores never override a state judge.
FakerMake test dataIn useSmall, free, and already in the Lab.
SDV (Synthetic Data Vault)Make test dataLaterEvery new agent starts with no real data, so it comes later.
time-machineControl timeIn useIn use in the Lab.
SimPyControl timeIn useA tested engine; no reason to write our own.
Toxiproxy (Shopify)Faults and real systemsOpt in nextPlanned for real mode.
TestcontainersFaults and real systemsEvaluatePer-run namespaces already do this; adopt if CI needs its own.
VCR.py / pytest-recordingFaults and real systemsOpt in nextPlanned; AgentTwin adds marking stale cassettes.
Saleor (and similar)Faults and real systemsIn useBorrowed, never built; shadow checks are not routine yet.
WireMock / PrismFaults and real systemsOpt outA canned mock would cancel a shipped order and the test would pass while proving nothing.
AWS Strands ToolSimulatorFaults and real systemsEvaluateNever for a system whose rules matter: a model can be talked into allowing what the real system refuses.
PostgreSQL per run + pgaudit + wal2jsonFaults and real systemsLaterNo such agent in scope yet.
mitmproxy (DNS interception)Reach more agents, and shareLaterDesigned; comes after the current agents.
OpenEnv (Meta + Hugging Face)Reach more agents, and shareLaterA distribution route, not a dependency.
pass^k (from τ²-bench)Reach more agents, and shareIn useIn the Lab's day report; nightly runs designed. τ²-bench's own 165 tasks are not used as a dependency.

What AgentTwin builds itself, because no tool does it: the world format, the stand-ins that enforce the agent's spec, the four judges that read the world's state, the switch between mock, replay, real and shadow modes, and (next) mutation testing over the spec's rules.

'Team effort for input' means work for the team testing an agent, beyond writing the agent's spec. Most inputs are generated from the spec, so the effort is often none. Tools that need a model run on a free hosted open-weight model where possible.

Each tool, in one card

Run the tests

pytest In use

The standard Python test runner.

Job here
Runs every scenario and test, and reports which passed.
Input
Test files, and AgentTwin's scenario files.
Processing
Finds the tests, runs them, collects results.
Output
Pass or fail per test, as a JUnit report the gates read.
Team effort for input
None: the team already writes tests this way.
Contributes
One runner for unit tests and scenarios alike, so one report holds everything.
Why this decision
The default in Python; a second runner would split the evidence.

Inspect (UK AI Security Institute) Evaluate

An open framework for running AI evaluations.

Job here
Considered as a runner (turned down) and as a grader (still open).
Input
Evaluation tasks: a dataset, a solver, a scorer.
Processing
Runs the tasks against a model or agent and scores them.
Output
Scores and logs per sample.
Team effort for input
Medium: each grader is a task to write.
Contributes
Well-built model graders, if we pick it over DeepEval.
Why this decision
Out as a runner (pytest already does that); still a candidate for soft grading.

pytest-cov Opt in next

Measures which lines of code the tests ran (line coverage).

Job here
A floor that may only go up, never a target.
Input
The agent's code and its test run.
Processing
Records each line executed during tests.
Output
A percentage and a list of lines never run.
Team effort for input
None: one plugin flag.
Contributes
Spots code nothing touches. Says nothing about whether the answers were right.
Why this decision
Cheap and useful as a ratchet; misleading as a goal, so it is never the gate.

Choose which scenarios to run

allpairspy In use

A small Python library for pairwise testing: choose a few combinations that still cover every pair of values.

Job here
Picks scenarios across intent × state × customer type × fault.
Input
Lists of values per dimension, plus a rule for which can't go together.
Processing
Builds a short list of combinations where every pair appears at least once.
Output
A list of combinations, each a scenario to write.
Team effort for input
None: AgentTwin reads the dimensions from the spec.
Contributes
Hundreds of meetings between two things covered by tens of scenarios, not thousands.
Why this decision
Works, but left 187 of 468 possible pairs uncovered for the shop agent; our own code fills the rest.

PICT (Microsoft) Evaluate

Microsoft's pairwise tool, the industry standard, with first-class 'these can't go together' rules.

Job here
A possible replacement for allpairspy.
Input
A model file: parameters, values, and constraints.
Processing
Same job as allpairspy, with stronger constraint handling.
Output
A list of combinations.
Team effort for input
None: AgentTwin would write the model file from the spec.
Contributes
Fewer uncovered pairs, less of our own top-up code.
Why this decision
Worth a test against allpairspy's 187 misses before switching.

GraphWalker Opt out

A model-based testing tool (Java): draw a system's states as a graph and it walks every arrow.

Job here
Its technique is used; the tool is not.
Input
A graph of states and the moves between them.
Processing
Finds paths that cross every move at least once.
Output
Test paths.
Team effort for input
-
Contributes
The idea behind AgentTwin's own state-machine walker.
Why this decision
Our state machines are already in the spec; a Java tool for one algorithm would be a heavy dependency.

Find the edge values

Hypothesis In use

Property-based testing for Python: you state a rule that must always hold, it invents hundreds of inputs to break it.

Job here
Edge values beyond the ones the spec declares.
Input
A property, such as 'a refused write changes nothing', and the shape of valid inputs.
Processing
Generates inputs, runs the check, and when one fails shrinks it to the smallest failing example.
Output
A failing example, or confidence the property held.
Team effort for input
Low: the spec's own rules supply most properties.
Contributes
Finds odd inputs nobody would think to type.
Why this decision
Mature, free, and already a dependency.

hypothesis-jsonschema In use

Turns a JSON schema into Hypothesis inputs.

Job here
Proves each stand-in keeps its own promises: every answer matches its declared shape.
Input
Each tool's input and output schema.
Processing
Generates valid tool inputs and checks every answer.
Output
Failures where a stand-in answers outside its schema.
Team effort for input
None: the schemas come from the spec.
Contributes
Guards the stand-ins themselves; it found a real bug in the claims stand-in.
Why this decision
Test-only dependency; never shipped.

Schemathesis Later

Hypothesis for web APIs: reads an OpenAPI description and fires generated requests at the API.

Job here
Checking a stand-in against a real REST API's contract.
Input
An OpenAPI (or GraphQL) description of the API.
Processing
Generates requests from it and checks the responses.
Output
Responses that break the contract.
Team effort for input
Low: if the API already publishes OpenAPI.
Contributes
Keeps stand-ins honest once we test agents that call REST APIs.
Why this decision
Needed only when non-MCP agents arrive.

Test the tests

Mutation testing (the idea) Opt in next

Break the system on purpose in small ways and see if any test notices. A break nobody notices is a missing test.

Job here
AgentTwin's planned answer to 'are the tests enough?': remove a check, loosen a rule, drop the idempotency key, then run the suite.
Input
The agent's spec and harness settings, and its scenario suite.
Processing
Makes many slightly broken copies ('mutants') and runs the suite on each.
Output
A list of surviving mutants: each one a named test that is missing.
Team effort for input
None: the breaks are generated from the spec's own rules.
Contributes
The only item on this page that measures how good the suite is, not just how big it is.
Why this decision
We will build a small version that changes rules and checks, because the tools below change program text.

mutmut Opt out

A mutation-testing tool for Python. It changes your code (a < becomes <=, a + becomes -) and reruns the tests.

Job here
Shown here to explain the idea; not used as the tool.
Input
Python source code and its tests.
Processing
Mutates the code one change at a time and reruns the tests.
Output
Which changes the tests missed.
Team effort for input
None
Contributes
Measures unit tests well.
Why this decision
An agent's important rules live in its spec and harness settings, not in arithmetic; mutating Python text would mostly produce noise.

Stryker Opt out

The same idea for JavaScript, TypeScript, C# and Scala.

Job here
None.
Input
Source code and tests.
Processing
Mutates and reruns.
Output
Surviving mutants.
Team effort for input
-
Contributes
-
Why this decision
Wrong languages for us; the idea is what we take.

Play the people

LangWatch Scenario (user simulator only) In use

An open-source agent-testing library. A model plays the user with a persona, and a model judge scores the chat.

Job here
Plays the customer when a scripted one is too predictable.
Input
A persona and a situation, both taken from the scenario.
Processing
A model writes the customer's next message, turn by turn, until done or out of turns.
Output
A realistic conversation. AgentTwin then judges it by the world's state.
Team effort for input
None: the scenario already has it.
Contributes
Customers who say things nobody scripted. It found a real miss: 'my jacket is lost' never matched the 'lost in transit' rule.
Why this decision
We use only the simulated user. Its judge is out: we judge by state, not by words. No cloud account, nothing uploaded.

DeepEval Synthesizer + Conversation Simulator Opt in next

Generates test conversations ('goldens') from documents or from scratch, then simulates them.

Job here
Variety: many wordings and situations for each scenario draft.
Input
Documents (policies, FAQs) or a short description of the users.
Processing
A model writes multi-turn situations with an expected outcome.
Output
Draft scenarios, which AgentTwin's loader then checks against the spec.
Team effort for input
Low: the agent's policy text already exists.
Contributes
Cuts the tester's hardest manual job: phrasing every case like a real person.
Why this decision
Same family as the soft graders we may adopt; runs on a free hosted model.

τ²-bench dual control Evaluate

A research benchmark where the customer also has tools and acts on their side (for example on their own phone).

Job here
A customer kind that acts, not only talks.
Input
Tools for the customer side.
Processing
Both sides act on one shared world.
Output
Runs where success needs both sides to act.
Team effort for input
Medium: customer-side tools must be described.
Contributes
Covers support flows where the customer must do something.
Why this decision
Needed for telecom-style agents; not for the current ones.

Azure AI Evaluation Simulator Cloud binding

Microsoft's managed user simulator.

Job here
The Azure binding of the same seam LangWatch fills.
Input
A persona and task.
Processing
A model plays the user.
Output
Conversations.
Team effort for input
Low
Contributes
Same job on Azure, for teams already there.
Why this decision
Used only when a project runs on Azure.

Attack the agent

PyRIT (Microsoft) In use

Microsoft's open-source red-teaming toolkit: attack seeds, 'converters' that rewrite one attack many ways, attacker models, and scorers.

Job here
Supplies attack text planted into fields the spec marks untrusted.
Input
An attack goal and the untrusted field (both from the spec).
Processing
Rewrites the attack: encodings, ciphers, swapped or invisible characters, jailbreak templates.
Output
Many attack variants. AgentTwin judges each by what changed in the world.
Team effort for input
None: scaffold writes the goals.
Contributes
Breadth of attack wording without writing it by hand. 180 generated cases held on both agents.
Why this decision
Only its fixed, free converters: its model-driven parts need a paid model or a download.

AgentDojo (ETH Zürich) In use

A research benchmark of prompt-injection attacks planted in emails, files and notes.

Job here
Six attack templates and 27 injection tasks, used as data.
Input
The templates and tasks.
Processing
Each task is mapped to a goal in the world's terms.
Output
Planted attacks, judged by state.
Team effort for input
None
Contributes
Proven attack shapes from published research.
Why this decision
Copied in as data: installing the package would pull in five model vendors' SDKs.

promptfoo red team Opt in next

Generates attacks tailored to your app from a short description of its purpose, with plugins per weakness.

Job here
A model-driven red team, beyond PyRIT's fixed rewrites.
Input
A short 'purpose' paragraph: who the users are, what they may see and do.
Processing
Plugins generate attacks per weakness; strategies make them harder.
Output
Attacks and a report.
Team effort for input
Low: one paragraph, which the spec can supply.
Contributes
Attacks aimed at this agent's own rules, not generic ones.
Why this decision
Planned behind the same attack seam; AgentTwin keeps judging by state.

Giskard Evaluate

An open-source scanner that probes LLM apps for weaknesses such as injection and made-up answers.

Job here
A second opinion scan.
Input
The agent wrapped as a function.
Processing
Sends generated probes and checks replies.
Output
A list of suspected weaknesses.
Team effort for input
Low: a small wrapper.
Contributes
Quick extra coverage for answer quality.
Why this decision
Not checked against our needs yet.

garak (NVIDIA) Opt out

A scanner that probes a language model for weaknesses.

Job here
None.
Input
A model endpoint.
Processing
Fires many probes at the model.
Output
A report of model weaknesses.
Team effort for input
-
Contributes
-
Why this decision
It tests the model, not the agent around it. AgentTwin's question is whether anything harmful reached the world.

DeepTeam Opt out

A red-teaming library from the makers of DeepEval.

Job here
None.
Input
A target and attack types.
Processing
Generates and scores attacks.
Output
Results.
Team effort for input
-
Contributes
-
Why this decision
Overlaps promptfoo; one red-team generator is enough.

Azure AI Red Teaming Agent Cloud binding

Microsoft's managed red team, built on PyRIT.

Job here
The Azure binding of the attack seam.
Input
A target.
Processing
PyRIT attacks plus Azure scorers.
Output
Attack results.
Team effort for input
Low
Contributes
Same job on Azure.
Why this decision
Used only on Azure projects.

Grade soft qualities

DeepEval (or Inspect) graders Opt in next

Model judges with a written rubric, for qualities a rule can't check.

Job here
Grades tone, clarity and completeness, which state judges can't see.
Input
The conversation and a rubric.
Processing
A model scores it against the rubric.
Output
A score and a reason.
Team effort for input
Low to medium: a rubric per quality.
Contributes
Adds 'was it well said?' to 'was it right?'.
Why this decision
One of the two will be chosen; soft scores never override a state judge.

Make test data

Faker In use

Generates fake but realistic names, addresses, emails and dates.

Job here
A population of customers and orders for long runs (in the Lab).
Input
Which fields to fill and how many rows.
Processing
Generates values; AgentTwin checks every row against the spec's rules.
Output
Rows the world can load.
Team effort for input
None
Contributes
Volume and variety without real customer data.
Why this decision
Small, free, and already in the Lab.

SDV (Synthetic Data Vault) Later

Learns the patterns in real tables and generates new, fake rows with the same patterns.

Job here
Realistic messy data at volume.
Input
A sample of real data.
Processing
Fits a model to the sample and generates look-alike rows.
Output
Synthetic tables.
Team effort for input
High: needs real data, which a new agent doesn't have, plus privacy review.
Contributes
Realistic skew and dirt, once an agent has real history.
Why this decision
Every new agent starts with no real data, so it comes later.

Control time

time-machine In use

Moves a Python program's clock forward instantly.

Job here
'Days have passed' without waiting.
Input
A target time.
Processing
Fakes the clock the code reads.
Output
The agent sees the new time.
Team effort for input
None
Contributes
Tests return windows and expiry in seconds.
Why this decision
In use in the Lab.

SimPy In use

A discrete-event simulation library: schedules events on a simulated timeline.

Job here
The Lab's world that keeps running: arrivals, shifts, incidents.
Input
Rates and a calendar of events.
Processing
Fires events in simulated time.
Output
A simulated day, week or month.
Team effort for input
Low
Contributes
Long-running tests of an agent over 30 simulated days.
Why this decision
A tested engine; no reason to write our own.

Faults and real systems

Toxiproxy (Shopify) Opt in next

A proxy that adds slowness, drops and errors to network traffic, on command.

Job here
Network faults against a real system, not just the stand-in.
Input
A real store running behind it.
Processing
Sits between agent and store and injects faults.
Output
The agent experiences real network trouble.
Team effort for input
Medium: needs the real system running.
Contributes
Checks fault handling holds outside the simulation.
Why this decision
Planned for real mode.

Testcontainers Evaluate

Starts throwaway real services (databases, shops) in Docker for each test.

Job here
A clean real system per run, in CI.
Input
A Docker image of the system.
Processing
Starts it, seeds it, tears it down.
Output
A fresh real system.
Team effort for input
Medium: Docker images and seeding.
Contributes
Real-system tests without shared state.
Why this decision
Per-run namespaces already do this; adopt if CI needs its own.

VCR.py / pytest-recording Opt in next

Records real HTTP calls once, then replays them in later tests.

Job here
Replay mode: real answers without calling the real system again.
Input
One recorded session against the real system.
Processing
Saves calls to 'cassettes' and serves them back.
Output
Repeatable real responses.
Team effort for input
Medium: one recording run, kept fresh.
Contributes
Realism at stand-in speed.
Why this decision
Planned; AgentTwin adds marking stale cassettes.

Saleor (and similar) In use

An open-source online shop.

Job here
The real system the shop agent's stand-in is checked against (shadow mode).
Input
A seeded Saleor store.
Processing
The same calls go to stand-in and Saleor; answers are compared.
Output
Where the stand-in differs from reality.
Team effort for input
Medium: run and seed it.
Contributes
Turns 'faithful' from a claim into a check.
Why this decision
Borrowed, never built; shadow checks are not routine yet.

WireMock / Prism Opt out

Mock servers that return canned answers (Prism builds them from OpenAPI).

Job here
None.
Input
Request-to-response rules.
Processing
Returns the canned answer.
Output
Fixed replies.
Team effort for input
-
Contributes
-
Why this decision
A canned mock would cancel a shipped order and the test would pass while proving nothing.

AWS Strands ToolSimulator Evaluate

A model pretends to be a tool and writes its replies.

Job here
Only for read-only, low-stakes tools the spec doesn't describe.
Input
A tool's schema and a prompt.
Processing
A model invents each reply, keeping shared state.
Output
Plausible tool replies.
Team effort for input
Low
Contributes
Covers the long tail cheaply.
Why this decision
Never for a system whose rules matter: a model can be talked into allowing what the real system refuses.

PostgreSQL per run + pgaudit + wal2json Later

A real database per run, with logs of everything read and changed.

Job here
Testing agents that query a database directly.
Input
The app's own database migrations and a mapping to the spec.
Processing
Seeds a copy, runs the agent, captures reads and writes.
Output
Exactly what the agent read and changed.
Team effort for input
Medium to high: the mapping to the spec.
Contributes
Reaches agents that bypass tools.
Why this decision
No such agent in scope yet.

Reach more agents, and share

mitmproxy (DNS interception) Later

An HTTPS proxy. Pointing an API's web address at it lets tests catch an unmodified agent's calls.

Job here
Testing agents that don't use MCP, without changing them.
Input
A translator per API, from its calls to world actions.
Processing
Intercepts each call and answers from the world.
Output
Any agent, unmodified, in the simulation.
Team effort for input
High: one translator per API.
Contributes
The biggest reach gain; today AgentTwin tests MCP agents only.
Why this decision
Designed; comes after the current agents.

OpenEnv (Meta + Hugging Face) Later

A standard way to package environments for training agents.

Job here
Export AgentTwin worlds so the research community can use them.
Input
An AgentTwin world.
Processing
An adapter wraps it in OpenEnv's interface.
Output
A shareable environment.
Team effort for input
Medium: the adapter is ours.
Contributes
Reach, not testing power.
Why this decision
A distribution route, not a dependency.

pass^k (from τ²-bench) In use

A measure, not a tool: run the same scenario k times; it passes only if all k pass.

Job here
Reliability with live, random models.
Input
A scenario and k.
Processing
Runs it k times.
Output
Pass only if every run passed.
Team effort for input
None
Contributes
Separates 'worked once' from 'works reliably'.
Why this decision
In the Lab's day report; nightly runs designed. τ²-bench's own 165 tasks are not used as a dependency.

← Back to AgentTwin, explained