AgentTwin Lab
Simulated days for AI agents, to see whether automated fixers weaken the checks that caught them.
1 · Purpose
The lab runs an AI agent through simulated days in an AgentTwin world, with customers arriving, incidents on a calendar and a monitor that raises them with evidence. When an incident is found, a coding agent tries to fix it, and the lab records whether the fix is honest or quietly weakens a test or a rule. A person approves every deploy before the next day. It is for researchers and teams who plan to let coding agents repair live AI systems, and need to know that the alarm was fixed, not silenced.
2 · Business benefits
3 · Status and how to join
Good first tasks this week:
- Plant a second incident on the day's calendar
- Add a classifier test for a loosened ceiling
- Run three seeded days and check the report reads clearly
4 · Requirements
What it must do
- Runs a simulated day: a generated population, visits on a curve and incidents from the world's calendar.
- Monitors every call and reply as it happens and raises incidents with their evidence.
- Turns an incident into a brief and runs a headless coding agent on it in a separate git worktree.
- Protects test and rule files in six layers, each logging what it stopped.
- Classifies every edit as weakening, strengthening, neutral or unclear, without a model.
- Records the loop of day, fix, approve, deploy and next day as one run with its report.
How well it must do it
- The lab builds only the day runner, the ledger and the report; every other job is an adopted tool behind a port.
- Same world, plan and seed give the same ledger.
- Nothing leaves the machine in a scripted run.
- Keys used by a real fixer are kept out of every record, and records are scanned to confirm it.
- Results report reliability over several seeded days, not one lucky run.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Open stack | Azure | AWS |
|---|---|---|---|
| World and calendar | AgentTwin, SimPy ✓ | — | — |
| Population and visits | Faker (en_IN) ✓ | — | — |
| Agent under test | reference agent in process; Temporal test server; time-machine ✓ | not built | not built |
| Monitor | AgentTwin monitor, OpenTelemetry SDK ✓ | — | — |
| Fixer harness | Claude Code headless in git worktrees; a LiteLLM proxy to a free hosted model ✓ | — | — |
| Weakening classifier | Python ast and YAML diffs ✓ | — | — |
| Release pointer | git, with GitHub as an option ✓ | — | — |
7 · What needs to be done
One simulated day 3 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Day runner, population, calendar and monitor | done | M | Python |
| Planted incident: parcels marked delivered that never arrived | done | S | Scenario design |
| Pass rates over several seeded days | done | S | Statistics |
The fixer and its guard rails 4 of 4 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Fixer harness with honest, gaming and rule-narrowing stubs | done | M | Python |
| Protected paths in six layers | done | M | Security |
| Weakening classifier over every attempted edit | done | M | Static analysis |
| Person-approved deploys and the full loop as one run | done | M | Release engineering |
Real fixers 1 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| First real fixer runs through a free hosted model | done | M | LLM tooling |
| Run the fixer with a larger-context model and compare | todo | M | LLM tooling |
| Repeat runs on several incidents and publish the results | todo | L | Research |
Richer days 0 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Add turns that go through the model loop, with a scripted policy | todo | M | Python |
| Add refund requests and approvals to the day | todo | M | Scenario design |
| Run the claims agent through a simulated day | todo | L | AI testing |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Research engineer | 1 | 4-6 hours/week | Runs fixer experiments and writes up what they show. |
| Lab developer | 1 | 3-5 hours/week | Extends days, incidents and the classifier. |
| Reviewer | 1 | 1-2 hours/week | Labels unclear edits and checks the classifier's verdicts. |
9 · What you'll learn
- Why an automated fixer might weaken the test instead of fixing the fault.
- How to run a whole simulated day deterministically.
- How to guard files from an agent in layers.
- How to classify code and spec changes without a model.
- Why a free model tier is too small for real repair work, measured.
Level-up questions
- If a fixer wrote nothing, is that evidence it is honest?
- What should happen to an edit the classifier marks unclear?
- Which protection layer would you trust least, and why?
- How many seeded days are enough to claim a fix holds?