Home › Learn › DataTwin, explained

Data engineering · free guide · with a step-through simulator

DataTwin, explained

How do you know a data platform shows the right number, every day, whatever its sources do? DataTwin keeps the platform real and puts it in a pretend business: a shop that keeps selling, source systems that send its data, and a calendar of things going wrong. Every simulated night it asks one question: do the dashboard, the conversation UI and the world agree? This is the data-engineering sibling of AgentTwin.

Where it stands (31 Oct 2026). The DataTwin Studio, where you learn by watching real Spark and Delta runs, is built and live. DataTwin as a testing tool for your own platform is designed, not built: it is phases 6 to 8 of the plan, early 2027. This page explains the design and marks every feature as built, partly built, or to build.

Part 1 · The explainer

What it is, in plain words, before any detail.

What problem does it solve?

A fire drill for a data platform

Most data platforms are tested the same way: run the jobs, see them finish green, look at the dashboard and decide the numbers "look about right". Then one morning a source team renames a column, the pipeline carries on happily, and revenue shows zero. Or a file arrives twice and revenue doubles. Every job was green both times.

The trouble is that nobody knows what the right number is. Production can't tell you; it only has the data it has.

DataTwin fixes that by inventing the business. A pretend clothing shop sells, takes returns and pays refunds on its own clock, and it remembers every sale. Its source systems send that data to your platform exactly as real ones would: database changes, daily files, an API. The platform is the real one, unchanged. Then, like a fire drill, DataTwin sets things off on purpose from a calendar: day 3 arrives twice, day 4 has a renamed column, day 7 the source goes dark for six hours.

Because the shop made every sale, it knows the true total. So at the end of each simulated day, the check is simple and strict: does the number on the dashboard equal the number the world knows? Not "does it look normal", but "is it right".

Is it a digital twin?

DataTwin vs a digital twin

The name borrows from "digital twin", but it points the other way. A digital twin copies a real thing so you can watch and predict it. DataTwin keeps the thing real (your platform) and copies the world that feeds it, so you can test it.

A digital twinThe real thinga jet engine,a factory linesensor dataIts virtual copymirrors the thing,minute by minutePredictions, what-ifs“service it in 40 hours”act on the real thingIt copies the THING itself,to watch it and predict it, for its whole life.vsDataTwinthe simulated world: the twinA pretend shopsells, refunds, on a clockIts source systemsdatabase, files, APIIncident calendarrenames, late daysFaults + timedark source, slow APIReal dataplatformunchangedEach day: dashboard = world's truth?It copies the WORLD AROUND the platform,to break it on purpose before real data does.
Same word, opposite direction. A digital twin copies the thing. DataTwin keeps the thing (your data platform) real and copies the shop and source systems that feed it.
Digital twinAgentTwinDataTwin
What is copiedThe thing itself: an engine, a plant, a cityThe world around the agent: customers, systems, faultsThe world around the platform: the business, its source systems, their faults, the clock
What stays realThe worldThe agent, unchangedThe platform, unchanged: the same pipelines, Spark, Delta, checks and dashboard
Where data flowsFrom the real thing into the copyFrom the agent's spec into the worldFrom the world into the platform, as rows, files and a change feed
The questionWhat is this machine doing, and what will it do?Will this agent do the right thing in every situation?Will this platform show the right number every day, whatever the sources do?
How it judgesPrediction accuracyFour judges read the world's stateEvery number the platform shows is compared with the world's truth
How long a run isThe thing's whole lifeOne conversation, a few secondsDays to weeks of simulated time, in minutes

I already know AgentTwin. What maps to what?

If you know AgentTwin

DataTwin asks a data platform the same five questions AgentTwin's family asks an agent. Each spec has a data-side sibling; we copy the shape, not the content.

Agent side (built)Data side (designed)
The thing under testan AI agenta data platform
What it must DO (one per thing)AOAS: the agent's specDPM: the platform manifest (sources, contracts, metrics, SLOs)
What must EXISTAHC: AI Harness CatalogDPC: Data Platform Catalog (spine stages, 22 concerns, ports)
What must be TRUEAAC: AI Assurance CatalogDAC: Data Assurance Catalog
What it must FACEAWD: the agent's worldDWD: the data world, extending AWD (change feed, incident calendar)
Which productharness profileDBS: binding files, databricks.yaml first
The pretend 'other side'customers, approvers, tool systemsthe shop, its source systems, people who rename columns
What goes wronglost reply, stale read, injectionduplicate day, rename, late day, dark source, bad value, real spike, cost jump, metric change
One test isa scenario: one conversationa simulated day with its planted incidents
The long runAgentTwin Lab: 30 days of customers30 simulated days, unattended

What goes in, what happens, what comes out?

Input, processing, output

INPUTS: who gives whatOUTPUTSPlatform manifestfrom the platform team:sources, contracts, metricsWorld + incidentsfrom the tester:the shop, its clock, what breaksOwed checksfrom the Data AssuranceCatalog, by pipeline kindBindingwhich stack runs it:databricks.yaml firstDataTwin runs each simulated day1 the shop trades; the world keeps the truth2 sources emit rows, files, change feed3 plant the day's incidents; move the clock4 the real platform runs, unchanged5 detectors fire; fixes go through gates6 record run log, lineage, cost7 compare every number with the worldA verdict per simulated daydashboard, conversation UIand world agree, or notIncident recordscaught by what, how fast,fixed how, did it holdCoverage mapevery incident kind planted;every owed check provedEvidence pagethe page a client reads,generated, never hand-written
Input, processing, output. Four sources feed it; seven steps run for every simulated day; four things come out. This is the design; see Part 3 for what is built.

Platform team

Writes the platform manifest: its sources, tables, each table's contract, metrics and freshness promises. The same file the generators turn into pipelines.

Tester

Describes the world: the business, its source systems, its clock, and the incident calendar of what goes wrong on which day.

Data Assurance Catalog

The checks a platform owes, by the kinds of pipeline it has, such as "every silver row traces back to a bronze row". Planned: about 30 core checks first.

Binding

Which stack runs the platform: Databricks first. The only place a product name appears.

Part 2 · How it works

From set-up to a verdict, then a tiny platform you can play, then a real one.

What has to exist before any platform is tested?

How the system is set up

Five specs, shared by every platform. Three name no product and no business, so a bank and a clothing shop read the same document:

DPC · must exist

The Data Platform Catalog: each stage of the spine (sources, landing, bronze, silver, gold, semantic, experiences), the 22 cross-cutting concerns and the port each is reached through.

DAC · must be true

The Data Assurance Catalog: checkable obligations, citing DAMA and ODCS rather than restating them.

DWD · must be faced

The Data World Description: source systems, the change feed, the clock and the incident calendar. Extends AgentTwin's world format.

DPM · must do

The Platform Manifest: one per company, universal format. What this platform does.

DBS · which product

The Binding: for each port, the adapter on one stack. databricks.yaml first.

DataTwin itself is not a separate repository. It is AgentTwin extended with a data world, because the clock, the faults, the run modes and the "nothing passes by absence" runner are the same.

I have a data platform. What do I do?

What you do for each platform

  1. Write the platform manifest (DPM). Every source, entity and table, each table's contract (written in ODCS, the open data contract standard), each metric, each freshness promise. The platform team writes it; it is the single source of what the platform does, and the generators read it to build the pipelines.
  2. Say which kinds of pipeline it has. Ten pipeline kinds are planned: file drop, database replication (CDC), API pull, event stream, analytical mart, ML features, documents for AI, reverse ETL, data sharing, regulatory report. The kinds decide which checks it owes.
  3. Describe the world (DWD). A business that trades on its own clock (our example is a clothing shop: orders, returns, refunds), its source systems, and the incident calendar: which day a column is renamed, which day arrives twice. The plan is to scaffold most of this from the manifest, as AgentTwin scaffolds a world from an agent's spec.
  4. Choose the binding (DBS). Which stack the platform runs on: Databricks first, then Microsoft, AWS and GCP. The world never changes when the stack does.
  5. Point the platform at the world's sources. Each source has the mode switch from AgentTwin: mock (the world makes the rows), replay (recorded real extracts), real (a real store started for the run), shadow (real and simulated side by side, compared). The platform doesn't know which one it is reading.
  6. Run N simulated days. One day, seven days, or thirty, unattended. Every night the world's truth is compared with the dashboard and the conversation UI.
  7. Read the verdicts and the evidence. Per day: pass or fail, and why. Per incident: caught by which detector, how fast, fixed how, did it hold, did a person have to decide. Per platform: which incident kinds and owed checks are still unproved.

For a given platform, how do we know these are the tests it needs?

How we know which tests a platform needs

The same answer as for agents: nobody picks the tests by taste. The platform's test list is whatever it takes to prove what it declares and what its kinds of pipeline owe.

The platform manifestevery source, table,contract rule, metric, SLOData Assurance Catalogchecks owed by eachpipeline kind (planned)Data Platform Catalog22 concerns, on bydefault; off needs a waiverThe Problem Lab19 problems everyplatform meetsProduction incidentsa miss in the wildbecomes a planted incidentThe owed listevery incident kindand check this platformmust survive, with an ideach one needsa planted incident that firedand the check that it held (rule R9)or a waiverwith a reason, an approver, an expiry (R5)or 'not applicable'e.g. no streaming source, written downanything else is a detector nobody has seen fire: a named gap
How a given platform's test list is decided: by what it declares and what its pipeline kinds owe, not by taste.

Then rule R9 joins the two lists: every incident kind in the world's calendar must be matched to a detector that has been seen to fire on it. An unmatched kind is a named gap. A detector nobody has ever seen fire is treated as untested, however good it looks.

The honest catch. As with agents, this proves the tests are complete against what is written down. A source missing from the manifest gets no incidents. Replay of real extracts, and every production miss becoming a planted incident, push against that.

The world and its incidents are a lot for a tester. How do we stop them missing things?

How we help the tester not miss

LayerWhat it does for the testerStatus
1 · The common packThe 19 Problem Lab problems and the 8 incident kinds in the plan, as templates that fill themselves from the manifestpartly the problems are built and recorded; not yet templates
2 · From the manifestA starter world and calendar written from the manifest: each source's kind picks its incidents, each rule its edge rowsto build AgentTwin's scaffold is the model
3 · What's missingR9's join lists incident kinds with no detector seen firing, and concerns switched off without a waiverto build
4 · CombinationsPairs of incident × source kind × layer × day type (normal, sale, month end), chosen pairwise, as AgentTwin's coverage doesto build allpairspy already adopted
5 · True eventsPlanted things that are not faults (a real sale-day spike, a real price change), so a platform that raises false alarms failsto build
6 · Test the testsBreak the platform on purpose (remove a dedup, loosen a rule, switch off the freshness check) and see if any planted incident noticesto build

How are the pretend sources made?

How the pretend sources are made

From the world's state, not by hand. The shop keeps its own tables; each source system shows those tables the way it really would.

Kind of source or partHow DataTwin treats it
An operational databaseA change feed (inserts, updates, deletes) from the world's tables, on the world's clock; or a real database started for the run
File dropsFiles written to landing with a batch manifest, including the planted ones: twice, late, half-written, wrong columns
An APIAnswers from its declared contract, with throttling and timeouts from Toxiproxy
An event streamEvents with their own timestamps, some late (later, with the streaming lane)
The people upstreamThe incident calendar: the team that renames a column, the system that goes dark
The engine (Spark, Delta) and your pipelinesRun for real. Mocking them would test nothing

Each source also has AgentTwin's mode switch: mock (the world makes the rows), replay (recorded real extracts), real (a real store per run) and shadow (both, compared). The platform can't tell which one it is reading.

Where does each day's data come from, and how is the right answer known?

The data, and how the truth is known

The worldgenerated every order,so it KNOWS day 3sold ₹240,000rows onlyYour platform (real)landing, bronze, silver, gold,semantic layer, dashboard,conversation UIWhat it showstile: ₹480,000“sales were ₹480,000”the check: does what it shows equal what the world knows?₹480,000 ≠ ₹240,000, so the day fails, even though every job finished green
Why a simulated world fixes the oracle problem for data: the right answer is known because the world made the data.

On the AgentTwin page we met the oracle problem: generating tests is easy; knowing the right answer for each is hard. Data platforms have the same problem, and usually dodge it. Quality tools check rows against rules ("amount is never negative"); observability tools check today against history ("volume is 3× normal"). Neither knows what the revenue should be.

A simulated world does. It generated every order, so the true total for every day, every product and every customer is known before the platform sees a row. That is DataTwin's oracle.

Can I see one day, step by step?

Play it: a tiny platform

The smallest platform we could pick: one source (the shop's orders, plus refunds), bronze, silver, one gold total, one dashboard tile and one question to the conversation UI. Pick a day, run it against a well-built platform and then a faulty one, and watch which check catches what. The green row is the world's truth.

Run it against:

What happens, step by step
    The numbers, layer by layer
    wherewhat it holds now
    The incident file for this day

    These runs are written for this page to show the idea; DataTwin's checks are not built yet. Each day has a counterpart recorded for real, on Spark and Delta, in the Problem Lab: re-runs (2), schema drift (7), poison records and bad data (6, 11), source goes dark and late data (13, 4), self-healing (19).

    What changes for a real platform?

    Scaling up to a real platform

    Tiny (this page)Clothing-shop platformSecond domain
    Sources1 (orders, as daily files)orders by CDC, returns as files, refunds by API, customersa second domain to prove the specs are domain-free
    Tablesbronze, silver, one gold totalbronze and silver per source, SCD2 customers, fact_sales star, a semantic layer-
    Readersa tile and one questiondashboard and conversation UI reading one semantic layer-
    Incidents5 on this pagethe 8 kinds in the plan, on a calendar and at random-
    Timeone day each30 simulated days, unattended-
    Fixesshown as stepsself-healing agents open PRs against the manifest; contracts are protected paths-
    Statusillustrative, on this pageplanned: phase 8, Feb 2027planned: phase 10, mid 2027

    The eight incident kinds the plan plants, and what "fixed" means for each:

    Planted incidentDetector (concern)Expected fixCheck that it held
    Source renames a columnSchema drift (X3)Mapping PR in the manifestSilver row count back to the world's count
    A day arrives twiceReconciliation (X6), idempotency (X15)Dedup on the batch key, replayGold total equals the world's total
    A day arrives lateFreshness (X9)Backfill that windowLate rows land in the right day
    Source dark for six hoursFreshness, volume (X9, X4)Wait, then catch upGap closed; SLO breach recorded honestly; no false alarm afterwards
    Negative refund amountContract rule (X4)Quarantine; never loosen the ruleQuarantine count equals planted count
    Volume triples on sale dayVolume anomaly, cost (X4, X10)None: it is realNo false incident raised
    Job cost jumpsCost (X10)Cost-advisor PRCost back within budget
    Metric definition changesSemantic drift (X13)A person decidesDashboard and conversation UI agree

    Two things only show up at this size. Time: a late day must be backfilled into the right day, and a freshness breach must stay on record after the gap closes. Agents: self-healing agents may only open a pull request against the manifest; contract and metric files are protected, so a fix that loosens one is refused.

    How do we get to full coverage?

    Coverage

    Coverage is counted against three finite lists:

    ListSizeCounted howToday
    Incident kinds the world can plant8 in the plan, 19 Problem Lab problems to fold inR9: each kind joined to a detector seen firingto build
    Concerns switched on22 (X1 to X22)on by default; each off needs a waiverto build
    Owed checks in the Data Assurance Catalogabout 30 core, firsteach proved by a passing day that names it, or a signed gapto build not written yet

    So, unlike the AgentTwin page, there is no coverage number to show yet: the catalog it would be counted against is the next spec to write. When it exists, "full coverage" will mean what it means for agents: every owed item proved by a passing day, or signed off as a known gap.

    How will we know the tests really ran?

    How we know the tests really ran

    1. Nothing passes by absence (R7). A check that ran on zero rows is an error, never a pass. Already built in Ops Gate.
    2. Every incident kind has a planted test (R9). A detector that has never been seen to fire counts as untested.
    3. The platform can't see the simulator. Inherited from AgentTwin: if the platform's code imports it, the build fails.
    4. Generated code is regenerated in CI (R3). A hand-edited pipeline, which the manifest no longer describes, is caught.
    5. Everything deployed is declared (R2). A table or check that no manifest entry asked for is a finding.
    6. The evidence page is generated from the run log, never written by hand.

    Will these tests be enough to call a platform ready for production?

    Enough to call it ready?

    For "ready"Status
    Every declared incident kind planted, caught and heldto build
    Every owed check proved or signed off as a gapto build catalog not written
    30 simulated days unattended, the dashboard agreeing with the world every dayto build phase 8
    Replay with real extracts, so real-world mess is faced tooto build
    In production: observability on, and every miss turned into a planted incidentpartly detectors exist; the loop does not

    Our verdict today: the method is sound and the pieces it reuses are proven on the agent side, but there is nothing yet to run against your platform. Use the Studio to learn the problems now; DataTwin the tool follows the plan.

    Part 3 · As a product

    What exists, what doesn't, who else is here, and where we are weak.

    If this were a product, what's in the box?

    Built, and still to build

    FeatureStatusNote
    DataTwin Studio: the simulator for learning, with real recordingsbuiltone batch end to end, Delta internals, Spark internals, 19 Problem Lab problems, 13 performance stories, data quality; all recorded on Spark 3.5.1 and Delta 3.2.0 on a laptop
    The Problem Lab as the first 'common pack'built19 problems, each with symptom, naive fix and right fix, run for real; each is meant to become an incident kind
    Data quality as metadata, one port, two enginesbuiltSpark SQL and PyDeequ adapters, profiling, anomaly checks (29 Oct)
    The AgentTwin core DataTwin will extendbuiltworld loader, clock, faults, mock / replay / real / shadow switch, runner with five statuses; built for agents
    'Nothing passes by absence' (R7)builtbuilt in Ops Gate (DataPlatformSuite): unreachable evidence is an error, never a pass
    Cost and performance detectorsbuiltCostIntelligence1: skew, small files, Python UDFs, too many partitions, collect(), shuffle, cartesian joins
    Metadata store design (D10)builtadopted 27 Oct; already in LakeIngestionFramework
    The design: five specs, rules R1 to R9, eight incident kinds, the daily agreement questionpartlywritten in the plan; spec text not written yet
    DWD format: the data world (change feed, incident calendar)to buildplanned as an extension of AgentTwin's world format, not a new repo
    A shop world that trades on a clock and keeps the truthto buildplanned (T-085); Faker first, then SDV
    Pretend sources: CDC feed, file drops, API, streamto buildplanned; Toxiproxy for faults, Testcontainers for real stores
    The four checks: agreement, caught, held, accountedto buildplanned; this page's simulator shows the idea
    Data Assurance Catalog: about 30 core checksto buildnext (DE-021)
    R9 linter: every incident kind has a planted testto buildplanned
    Self-healing loop with protected contract filesto buildphase 7, early 2027; the agent-side version exists
    30 simulated days and the client evidence pageto buildphase 8, Feb 2027
    Streaming sources, late events, Kafkalaterphase 6; the streaming design is still your call (D5)
    Other clouds and a second domainlaterphase 10, mid 2027

    Can we really reach 100%?

    Is 100% coverage achievable?

    100% of…Achievable?Why
    Declared incident kinds planted and caughtyesA finite list; R9 checks it.
    Contract rules tested on both edgesyesRead straight off the manifest.
    Owed checks accounted foryes, once the catalog existsProved, waived with a reason, or not applicable.
    Owed checks proved before releasenearlyCost at real scale and real source behaviour are only fully proved in production, by monitoring.
    Everything real data can donoNo one can promise that. The honest claim is the rows above, plus every production miss becoming a planted incident.

    Who else is building something like this?

    Who else builds this

    Each kind of tool covers one part. None we found plants incidents in a simulated business and checks the dashboard against the truth.

    KindExamplesWhat they needThe question they answerWhere they lead
    Data quality and test frameworksGreat Expectations, Soda Core, dbt tests and unit tests, Deequ, Databricks DQX and pipeline expectationsrules on real or sample datado the rows meet the rules I wrote?mature, widely used; easy to adopt
    Data observabilityMonte Carlo, Bigeye, Anomalo, Elementary (open-source core)nothing: they watch productionis today unusual compared with history?catching surprises in production nobody wrote a rule for
    Data diff and branches for CIDatafold, lakeFS, Nessie, zero-copy clones (Databricks, Snowflake)a copy of production to test a change ondid my change alter the output?safe changes against real data
    Synthetic and test dataSDV (DataCebo), MOSTLY AI, Tonic, Gretel (now NVIDIA), Fakerrealistic rows, often privacy-safenothing: they make data, they don't judgerealism and privacy
    Benchmarks with generated sourcesTPC-DI (data integration), TPC-DSa fixed company with history and incremental loadsvalidation queries, speeda published yardstick for ETL

    From each tool's public documentation, October 2026. Products change quickly; check before relying on a detail.

    How is it different?

    How it differs

    In one line: quality tools ask "does the data meet my rules?", observability tools ask "is today unusual?", diff tools ask "did my change alter the output?". DataTwin asks "is the number right?", because its world knows the answer, and then "did your rules and monitors fire when we broke things on purpose?". It doesn't replace them; they are among the things it tests.

    DataTwinQuality tools (GE, Soda, dbt)Observability (Monte Carlo)Diff and branches (Datafold, lakeFS)Synthetic data (SDV, Tonic)
    Where the right answer comes fromthe world: it made the data, so it knows the true totalsthe rules you wrotehistory: what is normalthe version before your change-
    Can say 'this number is wrong'yes, against the truthonly if a rule covers itonly 'unusual'only 'different'no
    Planted incidents on a calendaryes: the core ideanononono
    Tests the detectors themselvesyes: a planted incident nobody caught fails (R9)nononono
    Checks a fix didn't loosen a contractyes: contracts are protected pathsnononono
    Simulated time (days in minutes)yesnononono
    Realistic databasic until SDV is inyour datayour datayour datastrong
    Works on productionno: before production, and nightlyyesyesyes, on a copy-
    Product, UI, supportno: a library and a courseyesyesyesyes
    Statusdesigned; Studio builtmaturematurematuremature

    Where others are ahead: all of them are real products in use today; observability catches production surprises nobody planted; diff tools test changes on real data; synthetic-data vendors make far more realistic data. DataTwin plans to use SDV rather than compete.

    What limits are we facing?

    Our limits today

    1. It isn't built yet. The biggest limit. The design is written and the learning studio is real, but the data world, the pretend sources, the checks and the 30-day run are phases 6 to 8. Everything in Part 2 past the Studio is how it will work, not how it works today.
    2. Pretend data is cleaner than real data. Real sources send odd encodings, half-written files, duplicate keys with different values, time zones that change in March. Faker and even SDV only make what they are told to. Replay mode (recorded real extracts) and shadow mode are the answer, and both are planned.
    3. The tests are only as complete as the manifest. If a source or rule is not declared, no incident will be planted for it. The same is true of AgentTwin and its spec, and the same answer applies: every production incident becomes a planted one.
    4. Laptop scale hides some problems. A lesson from building the Studio: at small volumes, some before-and-after numbers don't move. Skew, spill and cost only show at size. Those stories must be rescaled or labelled illustrative, never passed off as measured.
    5. Some managed-platform behaviour can't run locally. Serverless compute, Photon and Unity Catalog grants are Databricks-only. The engine internals are recorded on open-source Spark and Delta; the Databricks binding has to be tested on Databricks Free Edition, which has quotas.
    6. Dashboards are judged by their numbers. DataTwin checks that the tile and the conversation UI give the world's number. It does not judge whether a chart is clear, or whether the metric is the right one for the business.
    7. One domain so far. The clothing shop. A second domain is planned to prove the specs don't secretly assume retail.

    Which open-source tools does it use?

    The open-source tools

    Like AgentTwin, DataTwin is an assembler: it uses proven open-source tools behind ports and builds only what nobody does.

    DataTwin builds onlythe world format (DWD, extends AWD)a shop that keeps the truththe incident calendar + R9 joinchecks: dashboard vs worldSpark + Delta (OSS)the engine, run for realPyDeequdata quality engineAgentTwin coreclock, faults, modes, runnerpytest, Hypothesisrunner, edge valuesFaker, then SDVrows, then realistic rowstime-machine, SimPythe world's clockToxiproxy, Testcontainersfaults, real storesOpenLineage, Great Expectationslineage, 2nd DQ enginegreen = in use today (in the Studio or AgentTwin) · yellow = planned, behind a port
    An assembler, like AgentTwin. Each tool sits behind a port, so it can be swapped.
    JobToolStatusWhy
    The engineApache Spark 3.5 and Delta Lake 3.2 (open source)in useRun for real, never mocked: internals are visible locally and free
    Data qualityPyDeequ; Spark SQL rulesin useBoth behind one port in the Studio's data quality scene
    World, clock, faults, runnerAgentTwin corein useExtended, not copied: one world format for agents and data
    Test runner, edge valuespytest, Hypothesisin useAlready used by AgentTwin
    Second data quality engineGreat ExpectationsplannedProves the port: same rules, other engine
    Rows for the worldFaker, then SDVplannedFaker for structure; SDV for realistic distributions and volume
    The world's clocktime-machine, SimPyplannedBoth in use in the AgentTwin Lab
    Faults on the wireToxiproxyplannedSlow, dropped or cut connections to an API or database
    Real stores per runTestcontainersplannedA real Postgres or Kafka started for one run, then thrown away
    LineageOpenLineage, MarquezplannedRule R8 (data flows one way) is checked against lineage
    Profilingydata-profilingevaluatingProfiles that change when the data does
    Change data capture, streamsDebezium, RedpandalaterWith the streaming lane (phase 6)
    Branch per runlakeFSevaluatingEach simulated day on its own branch; cheap rollback

    Several of these (Faker, SDV, time-machine, SimPy, Toxiproxy, Testcontainers, Hypothesis) have a one-card explainer on the AgentTwin tools page.

    FAQ

    Questions people ask

    Is DataTwin a digital twin?

    Not in the usual sense. A digital twin copies a real machine to watch and predict it. DataTwin keeps your data platform real and copies the business and source systems that feed it, so you can break the platform on purpose before real data does.

    How is it different from AgentTwin?

    Same idea, different subject. AgentTwin keeps an AI agent real and simulates its customers and tools. DataTwin keeps a data platform real and simulates its sources and their incidents. DataTwin extends AgentTwin's world format rather than starting a new one, so the clock, faults and run modes are shared.

    Is it built?

    Partly. The DataTwin Studio, where you learn by watching real Spark and Delta runs, is built and live. DataTwin as a testing tool for your platform is designed and planned for early 2027. This page marks every feature as built, partly built, or to build.

    Why not just use Great Expectations or Monte Carlo?

    Keep them: they are what DataTwin tests. Rule tools check rows against rules you wrote; observability tools check today against history. Neither knows the right answer. DataTwin's world made the data, so it knows the true total, and it checks that your rules and monitors actually fire when an incident is planted.

    Can't I test on a copy of production instead?

    Yes, and you should, for changes. But production only contains the incidents that already happened. DataTwin lets you plant the ones that haven't happened yet (a renamed column, a duplicate day, a dark source), on a calendar, as often as you like.

    Is 100% coverage possible?

    100% of what a platform owes can be accounted for: every declared incident kind planted and caught, every owed check proved or signed off. 100% of everything real data can do cannot. See the section on coverage.

    Does it work only on Databricks?

    No. The world and the checks name no product. Databricks is the first binding; Microsoft, AWS, GCP and an open-source binding are planned.

    Next

    Where to go next

    Learn the problems DataTwin plants, recorded for real: the DataTwin Studio, the Problem Lab, performance stories, data quality.

    The architecture it tests: the data platform architecture and the medallion, explained.

    The sibling: AgentTwin, explained. Code it extends: github.com/dataagentsai/agenttwin (Apache 2.0).