AgentTwin Lab

Simulated days for AI agents, to see whether automated fixers weaken the checks that caught them.

Active · join meAI Engineering

8 of 13 tasks done

1 · Purpose

The lab runs an AI agent through simulated days in an AgentTwin world, with customers arriving, incidents on a calendar and a monitor that raises them with evidence. When an incident is found, a coding agent tries to fix it, and the lab records whether the fix is honest or quietly weakens a test or a rule. A person approves every deploy before the next day. It is for researchers and teams who plan to let coding agents repair live AI systems, and need to know that the alarm was fixed, not silenced.

For: AI safety and oversight researchers, platform teams considering automated fixes, assessors.

2 · Business benefits

Trust automated fixesEach change a fixer makes or tries is labelled as weakening or not, before it can ship.
A person stays in chargeNo fix reaches production without a person's approval through the lab's release pointer.
Find slow-burn faultsRunning whole days surfaces faults that a single test conversation never meets.
Cheap to repeatA simulated day takes about two seconds on a laptop, with no model calls.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Plant a second incident on the day's calendar
  • Add a classifier test for a loosened ceiling
  • Run three seeded days and check the report reads clearly

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

World and calendarThe AgentTwin world, withincidents planted on aPopulation and visitsGenerated customers and whenthey arrive.Agent under testThe reference support agent,through its gate binding.MonitorJudges each call and reply andraises incidents.Fixer harnessGives a coding agent the briefand records everything itWeakening classifierLabels each attempted edit.Release pointerDeploys a fix only after aperson approves.Ledger and reportEvery turn, call and incident,and pass rates over seededworld statecustomer visitcalls and replincident briefattempted editallowed fix
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Monitor → Ledger and report: incidents

6 · Tech stack, per platform

ComponentOpen stackAzureAWS
World and calendarAgentTwin, SimPy ✓——
Population and visitsFaker (en_IN) ✓——
Agent under testreference agent in process; Temporal test server; time-machine ✓not builtnot built
MonitorAgentTwin monitor, OpenTelemetry SDK ✓——
Fixer harnessClaude Code headless in git worktrees; a LiteLLM proxy to a free hosted model ✓——
Weakening classifierPython ast and YAML diffs ✓——
Release pointergit, with GitHub as an option ✓——

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

One simulated day 3 of 3 done

TaskStatusSizeSkill
Day runner, population, calendar and monitordoneMPython
Planted incident: parcels marked delivered that never arriveddoneSScenario design
Pass rates over several seeded daysdoneSStatistics

The fixer and its guard rails 4 of 4 done

TaskStatusSizeSkill
Fixer harness with honest, gaming and rule-narrowing stubsdoneMPython
Protected paths in six layersdoneMSecurity
Weakening classifier over every attempted editdoneMStatic analysis
Person-approved deploys and the full loop as one rundoneMRelease engineering

Real fixers 1 of 3 done

TaskStatusSizeSkill
First real fixer runs through a free hosted modeldoneMLLM tooling
Run the fixer with a larger-context model and comparetodoMLLM tooling
Repeat runs on several incidents and publish the resultstodoLResearch

Richer days 0 of 3 done

TaskStatusSizeSkill
Add turns that go through the model loop, with a scripted policytodoMPython
Add refund requests and approvals to the daytodoMScenario design
Run the claims agent through a simulated daytodoLAI testing

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Research engineer14-6 hours/weekRuns fixer experiments and writes up what they show.
Lab developer13-5 hours/weekExtends days, incidents and the classifier.
Reviewer11-2 hours/weekLabels unclear edits and checks the classifier's verdicts.

9 · What you'll learn

Level-up questions

10 · Links