Home › Learn › DataTwin, explained
Data engineering · free guide · with a step-through simulator
DataTwin, explained
How do you know a data platform shows the right number, every day, whatever its sources do? DataTwin keeps the platform real and puts it in a pretend business: a shop that keeps selling, source systems that send its data, and a calendar of things going wrong. Every simulated night it asks one question: do the dashboard, the conversation UI and the world agree? This is the data-engineering sibling of AgentTwin.
What problem does it solve?
A fire drill for a data platform
Most data platforms are tested the same way: run the jobs, see them finish green, look at the dashboard and decide the numbers "look about right". Then one morning a source team renames a column, the pipeline carries on happily, and revenue shows zero. Or a file arrives twice and revenue doubles. Every job was green both times.
The trouble is that nobody knows what the right number is. Production can't tell you; it only has the data it has.
DataTwin fixes that by inventing the business. A pretend clothing shop sells, takes returns and pays refunds on its own clock, and it remembers every sale. Its source systems send that data to your platform exactly as real ones would: database changes, daily files, an API. The platform is the real one, unchanged. Then, like a fire drill, DataTwin sets things off on purpose from a calendar: day 3 arrives twice, day 4 has a renamed column, day 7 the source goes dark for six hours.
Because the shop made every sale, it knows the true total. So at the end of each simulated day, the check is simple and strict: does the number on the dashboard equal the number the world knows? Not "does it look normal", but "is it right".
Is it a digital twin?
DataTwin vs a digital twin
The name borrows from "digital twin", but it points the other way. A digital twin copies a real thing so you can watch and predict it. DataTwin keeps the thing real (your platform) and copies the world that feeds it, so you can test it.
| Digital twin | AgentTwin | DataTwin | |
|---|---|---|---|
| What is copied | The thing itself: an engine, a plant, a city | The world around the agent: customers, systems, faults | The world around the platform: the business, its source systems, their faults, the clock |
| What stays real | The world | The agent, unchanged | The platform, unchanged: the same pipelines, Spark, Delta, checks and dashboard |
| Where data flows | From the real thing into the copy | From the agent's spec into the world | From the world into the platform, as rows, files and a change feed |
| The question | What is this machine doing, and what will it do? | Will this agent do the right thing in every situation? | Will this platform show the right number every day, whatever the sources do? |
| How it judges | Prediction accuracy | Four judges read the world's state | Every number the platform shows is compared with the world's truth |
| How long a run is | The thing's whole life | One conversation, a few seconds | Days to weeks of simulated time, in minutes |
I already know AgentTwin. What maps to what?
If you know AgentTwin
DataTwin asks a data platform the same five questions AgentTwin's family asks an agent. Each spec has a data-side sibling; we copy the shape, not the content.
| Agent side (built) | Data side (designed) | |
|---|---|---|
| The thing under test | an AI agent | a data platform |
| What it must DO (one per thing) | AOAS: the agent's spec | DPM: the platform manifest (sources, contracts, metrics, SLOs) |
| What must EXIST | AHC: AI Harness Catalog | DPC: Data Platform Catalog (spine stages, 22 concerns, ports) |
| What must be TRUE | AAC: AI Assurance Catalog | DAC: Data Assurance Catalog |
| What it must FACE | AWD: the agent's world | DWD: the data world, extending AWD (change feed, incident calendar) |
| Which product | harness profile | DBS: binding files, databricks.yaml first |
| The pretend 'other side' | customers, approvers, tool systems | the shop, its source systems, people who rename columns |
| What goes wrong | lost reply, stale read, injection | duplicate day, rename, late day, dark source, bad value, real spike, cost jump, metric change |
| One test is | a scenario: one conversation | a simulated day with its planted incidents |
| The long run | AgentTwin Lab: 30 days of customers | 30 simulated days, unattended |
What goes in, what happens, what comes out?
Input, processing, output
Platform team
Writes the platform manifest: its sources, tables, each table's contract, metrics and freshness promises. The same file the generators turn into pipelines.
Tester
Describes the world: the business, its source systems, its clock, and the incident calendar of what goes wrong on which day.
Data Assurance Catalog
The checks a platform owes, by the kinds of pipeline it has, such as "every silver row traces back to a bronze row". Planned: about 30 core checks first.
Binding
Which stack runs the platform: Databricks first. The only place a product name appears.
What has to exist before any platform is tested?
How the system is set up
Five specs, shared by every platform. Three name no product and no business, so a bank and a clothing shop read the same document:
DPC · must exist
The Data Platform Catalog: each stage of the spine (sources, landing, bronze, silver, gold, semantic, experiences), the 22 cross-cutting concerns and the port each is reached through.
DAC · must be true
The Data Assurance Catalog: checkable obligations, citing DAMA and ODCS rather than restating them.
DWD · must be faced
The Data World Description: source systems, the change feed, the clock and the incident calendar. Extends AgentTwin's world format.
DPM · must do
The Platform Manifest: one per company, universal format. What this platform does.
DBS · which product
The Binding: for each port, the adapter on one stack. databricks.yaml first.
DataTwin itself is not a separate repository. It is AgentTwin extended with a data world, because the clock, the faults, the run modes and the "nothing passes by absence" runner are the same.
I have a data platform. What do I do?
What you do for each platform
- Write the platform manifest (DPM). Every source, entity and table, each table's contract (written in ODCS, the open data contract standard), each metric, each freshness promise. The platform team writes it; it is the single source of what the platform does, and the generators read it to build the pipelines.
- Say which kinds of pipeline it has. Ten pipeline kinds are planned: file drop, database replication (CDC), API pull, event stream, analytical mart, ML features, documents for AI, reverse ETL, data sharing, regulatory report. The kinds decide which checks it owes.
- Describe the world (DWD). A business that trades on its own clock (our example is a clothing shop: orders, returns, refunds), its source systems, and the incident calendar: which day a column is renamed, which day arrives twice. The plan is to scaffold most of this from the manifest, as AgentTwin scaffolds a world from an agent's spec.
- Choose the binding (DBS). Which stack the platform runs on: Databricks first, then Microsoft, AWS and GCP. The world never changes when the stack does.
- Point the platform at the world's sources. Each source has the mode switch from AgentTwin: mock (the world makes the rows), replay (recorded real extracts), real (a real store started for the run), shadow (real and simulated side by side, compared). The platform doesn't know which one it is reading.
- Run N simulated days. One day, seven days, or thirty, unattended. Every night the world's truth is compared with the dashboard and the conversation UI.
- Read the verdicts and the evidence. Per day: pass or fail, and why. Per incident: caught by which detector, how fast, fixed how, did it hold, did a person have to decide. Per platform: which incident kinds and owed checks are still unproved.
For a given platform, how do we know these are the tests it needs?
How we know which tests a platform needs
The same answer as for agents: nobody picks the tests by taste. The platform's test list is whatever it takes to prove what it declares and what its kinds of pipeline owe.
- The manifest. Every source gets the incidents its kind can suffer: a database replication source can replay a day, a file source can send a file twice or half a file, an API can throttle. Every contract rule gets an edge row on each side. Every metric gets "the dashboard equals the world". Every freshness promise gets a dark source that breaks it.
- The Data Assurance Catalog, by pipeline kind. A regulatory report owes checks a marketing mart doesn't.
- The Data Platform Catalog's 22 concerns. Quality, reconciliation, lineage, freshness, cost, privacy and the rest are on by default (rule R5). Turning one off needs a waiver with a reason, an approver and an expiry, so a missing check is a decision someone signed, never an accident.
- The Problem Lab. 19 problems every platform meets sooner or later: duplicates, re-runs that double a day, history that can't be true, late data, failure halfway, a poison row, schema drift, deletes nobody sees. Each one becomes an incident kind.
- Production incidents. Anything that slips through in the wild becomes a planted incident, so it can't slip through twice.
Then rule R9 joins the two lists: every incident kind in the world's calendar must be matched to a detector that has been seen to fire on it. An unmatched kind is a named gap. A detector nobody has ever seen fire is treated as untested, however good it looks.
The world and its incidents are a lot for a tester. How do we stop them missing things?
How we help the tester not miss
| Layer | What it does for the tester | Status |
|---|---|---|
| 1 · The common pack | The 19 Problem Lab problems and the 8 incident kinds in the plan, as templates that fill themselves from the manifest | partly the problems are built and recorded; not yet templates |
| 2 · From the manifest | A starter world and calendar written from the manifest: each source's kind picks its incidents, each rule its edge rows | to build AgentTwin's scaffold is the model |
| 3 · What's missing | R9's join lists incident kinds with no detector seen firing, and concerns switched off without a waiver | to build |
| 4 · Combinations | Pairs of incident × source kind × layer × day type (normal, sale, month end), chosen pairwise, as AgentTwin's coverage does | to build allpairspy already adopted |
| 5 · True events | Planted things that are not faults (a real sale-day spike, a real price change), so a platform that raises false alarms fails | to build |
| 6 · Test the tests | Break the platform on purpose (remove a dedup, loosen a rule, switch off the freshness check) and see if any planted incident notices | to build |
How are the pretend sources made?
How the pretend sources are made
From the world's state, not by hand. The shop keeps its own tables; each source system shows those tables the way it really would.
| Kind of source or part | How DataTwin treats it |
|---|---|
| An operational database | A change feed (inserts, updates, deletes) from the world's tables, on the world's clock; or a real database started for the run |
| File drops | Files written to landing with a batch manifest, including the planted ones: twice, late, half-written, wrong columns |
| An API | Answers from its declared contract, with throttling and timeouts from Toxiproxy |
| An event stream | Events with their own timestamps, some late (later, with the streaming lane) |
| The people upstream | The incident calendar: the team that renames a column, the system that goes dark |
| The engine (Spark, Delta) and your pipelines | Run for real. Mocking them would test nothing |
Each source also has AgentTwin's mode switch: mock (the world makes the rows), replay (recorded real extracts), real (a real store per run) and shadow (both, compared). The platform can't tell which one it is reading.
Where does each day's data come from, and how is the right answer known?
The data, and how the truth is known
On the AgentTwin page we met the oracle problem: generating tests is easy; knowing the right answer for each is hard. Data platforms have the same problem, and usually dodge it. Quality tools check rows against rules ("amount is never negative"); observability tools check today against history ("volume is 3× normal"). Neither knows what the revenue should be.
A simulated world does. It generated every order, so the true total for every day, every product and every customer is known before the platform sees a row. That is DataTwin's oracle.
- Rows that obey the contract by default, from Faker, then SDV for realistic shapes and volumes. Seeded, so every run is the same.
- Bad rows only where planted. A negative refund appears because the calendar says so, and the world counts it, so "quarantine count equals planted count" can be checked exactly.
- Edge rows on each rule: amount 0 and −1, a timestamp one second before midnight.
- Volume on demand for performance stories: the same day at 1,000 rows or 10 million.
- Impossible worlds are refused. A refund for an order that never existed stops the load, unless it is a planted incident.
Can I see one day, step by step?
Play it: a tiny platform
The smallest platform we could pick: one source (the shop's orders, plus refunds), bronze, silver, one gold total, one dashboard tile and one question to the conversation UI. Pick a day, run it against a well-built platform and then a faulty one, and watch which check catches what. The green row is the world's truth.
What happens, step by step
The numbers, layer by layer
| where | what it holds now |
|---|
The incident file for this day
These runs are written for this page to show the idea; DataTwin's checks are not built yet. Each day has a counterpart recorded for real, on Spark and Delta, in the Problem Lab: re-runs (2), schema drift (7), poison records and bad data (6, 11), source goes dark and late data (13, 4), self-healing (19).
What changes for a real platform?
Scaling up to a real platform
| Tiny (this page) | Clothing-shop platform | Second domain | |
|---|---|---|---|
| Sources | 1 (orders, as daily files) | orders by CDC, returns as files, refunds by API, customers | a second domain to prove the specs are domain-free |
| Tables | bronze, silver, one gold total | bronze and silver per source, SCD2 customers, fact_sales star, a semantic layer | - |
| Readers | a tile and one question | dashboard and conversation UI reading one semantic layer | - |
| Incidents | 5 on this page | the 8 kinds in the plan, on a calendar and at random | - |
| Time | one day each | 30 simulated days, unattended | - |
| Fixes | shown as steps | self-healing agents open PRs against the manifest; contracts are protected paths | - |
| Status | illustrative, on this page | planned: phase 8, Feb 2027 | planned: phase 10, mid 2027 |
The eight incident kinds the plan plants, and what "fixed" means for each:
| Planted incident | Detector (concern) | Expected fix | Check that it held |
|---|---|---|---|
| Source renames a column | Schema drift (X3) | Mapping PR in the manifest | Silver row count back to the world's count |
| A day arrives twice | Reconciliation (X6), idempotency (X15) | Dedup on the batch key, replay | Gold total equals the world's total |
| A day arrives late | Freshness (X9) | Backfill that window | Late rows land in the right day |
| Source dark for six hours | Freshness, volume (X9, X4) | Wait, then catch up | Gap closed; SLO breach recorded honestly; no false alarm afterwards |
| Negative refund amount | Contract rule (X4) | Quarantine; never loosen the rule | Quarantine count equals planted count |
| Volume triples on sale day | Volume anomaly, cost (X4, X10) | None: it is real | No false incident raised |
| Job cost jumps | Cost (X10) | Cost-advisor PR | Cost back within budget |
| Metric definition changes | Semantic drift (X13) | A person decides | Dashboard and conversation UI agree |
Two things only show up at this size. Time: a late day must be backfilled into the right day, and a freshness breach must stay on record after the gap closes. Agents: self-healing agents may only open a pull request against the manifest; contract and metric files are protected, so a fix that loosens one is refused.
How do we get to full coverage?
Coverage
Coverage is counted against three finite lists:
| List | Size | Counted how | Today |
|---|---|---|---|
| Incident kinds the world can plant | 8 in the plan, 19 Problem Lab problems to fold in | R9: each kind joined to a detector seen firing | to build |
| Concerns switched on | 22 (X1 to X22) | on by default; each off needs a waiver | to build |
| Owed checks in the Data Assurance Catalog | about 30 core, first | each proved by a passing day that names it, or a signed gap | to build not written yet |
So, unlike the AgentTwin page, there is no coverage number to show yet: the catalog it would be counted against is the next spec to write. When it exists, "full coverage" will mean what it means for agents: every owed item proved by a passing day, or signed off as a known gap.
How will we know the tests really ran?
How we know the tests really ran
- Nothing passes by absence (R7). A check that ran on zero rows is an error, never a pass. Already built in Ops Gate.
- Every incident kind has a planted test (R9). A detector that has never been seen to fire counts as untested.
- The platform can't see the simulator. Inherited from AgentTwin: if the platform's code imports it, the build fails.
- Generated code is regenerated in CI (R3). A hand-edited pipeline, which the manifest no longer describes, is caught.
- Everything deployed is declared (R2). A table or check that no manifest entry asked for is a finding.
- The evidence page is generated from the run log, never written by hand.
Will these tests be enough to call a platform ready for production?
Enough to call it ready?
| For "ready" | Status |
|---|---|
| Every declared incident kind planted, caught and held | to build |
| Every owed check proved or signed off as a gap | to build catalog not written |
| 30 simulated days unattended, the dashboard agreeing with the world every day | to build phase 8 |
| Replay with real extracts, so real-world mess is faced too | to build |
| In production: observability on, and every miss turned into a planted incident | partly detectors exist; the loop does not |
Our verdict today: the method is sound and the pieces it reuses are proven on the agent side, but there is nothing yet to run against your platform. Use the Studio to learn the problems now; DataTwin the tool follows the plan.
If this were a product, what's in the box?
Built, and still to build
| Feature | Status | Note |
|---|---|---|
| DataTwin Studio: the simulator for learning, with real recordings | built | one batch end to end, Delta internals, Spark internals, 19 Problem Lab problems, 13 performance stories, data quality; all recorded on Spark 3.5.1 and Delta 3.2.0 on a laptop |
| The Problem Lab as the first 'common pack' | built | 19 problems, each with symptom, naive fix and right fix, run for real; each is meant to become an incident kind |
| Data quality as metadata, one port, two engines | built | Spark SQL and PyDeequ adapters, profiling, anomaly checks (29 Oct) |
| The AgentTwin core DataTwin will extend | built | world loader, clock, faults, mock / replay / real / shadow switch, runner with five statuses; built for agents |
| 'Nothing passes by absence' (R7) | built | built in Ops Gate (DataPlatformSuite): unreachable evidence is an error, never a pass |
| Cost and performance detectors | built | CostIntelligence1: skew, small files, Python UDFs, too many partitions, collect(), shuffle, cartesian joins |
| Metadata store design (D10) | built | adopted 27 Oct; already in LakeIngestionFramework |
| The design: five specs, rules R1 to R9, eight incident kinds, the daily agreement question | partly | written in the plan; spec text not written yet |
| DWD format: the data world (change feed, incident calendar) | to build | planned as an extension of AgentTwin's world format, not a new repo |
| A shop world that trades on a clock and keeps the truth | to build | planned (T-085); Faker first, then SDV |
| Pretend sources: CDC feed, file drops, API, stream | to build | planned; Toxiproxy for faults, Testcontainers for real stores |
| The four checks: agreement, caught, held, accounted | to build | planned; this page's simulator shows the idea |
| Data Assurance Catalog: about 30 core checks | to build | next (DE-021) |
| R9 linter: every incident kind has a planted test | to build | planned |
| Self-healing loop with protected contract files | to build | phase 7, early 2027; the agent-side version exists |
| 30 simulated days and the client evidence page | to build | phase 8, Feb 2027 |
| Streaming sources, late events, Kafka | later | phase 6; the streaming design is still your call (D5) |
| Other clouds and a second domain | later | phase 10, mid 2027 |
Can we really reach 100%?
Is 100% coverage achievable?
| 100% of… | Achievable? | Why |
|---|---|---|
| Declared incident kinds planted and caught | yes | A finite list; R9 checks it. |
| Contract rules tested on both edges | yes | Read straight off the manifest. |
| Owed checks accounted for | yes, once the catalog exists | Proved, waived with a reason, or not applicable. |
| Owed checks proved before release | nearly | Cost at real scale and real source behaviour are only fully proved in production, by monitoring. |
| Everything real data can do | no | No one can promise that. The honest claim is the rows above, plus every production miss becoming a planted incident. |
Who else is building something like this?
Who else builds this
Each kind of tool covers one part. None we found plants incidents in a simulated business and checks the dashboard against the truth.
| Kind | Examples | What they need | The question they answer | Where they lead |
|---|---|---|---|---|
| Data quality and test frameworks | Great Expectations, Soda Core, dbt tests and unit tests, Deequ, Databricks DQX and pipeline expectations | rules on real or sample data | do the rows meet the rules I wrote? | mature, widely used; easy to adopt |
| Data observability | Monte Carlo, Bigeye, Anomalo, Elementary (open-source core) | nothing: they watch production | is today unusual compared with history? | catching surprises in production nobody wrote a rule for |
| Data diff and branches for CI | Datafold, lakeFS, Nessie, zero-copy clones (Databricks, Snowflake) | a copy of production to test a change on | did my change alter the output? | safe changes against real data |
| Synthetic and test data | SDV (DataCebo), MOSTLY AI, Tonic, Gretel (now NVIDIA), Faker | realistic rows, often privacy-safe | nothing: they make data, they don't judge | realism and privacy |
| Benchmarks with generated sources | TPC-DI (data integration), TPC-DS | a fixed company with history and incremental loads | validation queries, speed | a published yardstick for ETL |
From each tool's public documentation, October 2026. Products change quickly; check before relying on a detail.
How is it different?
How it differs
In one line: quality tools ask "does the data meet my rules?", observability tools ask "is today unusual?", diff tools ask "did my change alter the output?". DataTwin asks "is the number right?", because its world knows the answer, and then "did your rules and monitors fire when we broke things on purpose?". It doesn't replace them; they are among the things it tests.
| DataTwin | Quality tools (GE, Soda, dbt) | Observability (Monte Carlo) | Diff and branches (Datafold, lakeFS) | Synthetic data (SDV, Tonic) | |
|---|---|---|---|---|---|
| Where the right answer comes from | the world: it made the data, so it knows the true totals | the rules you wrote | history: what is normal | the version before your change | - |
| Can say 'this number is wrong' | yes, against the truth | only if a rule covers it | only 'unusual' | only 'different' | no |
| Planted incidents on a calendar | yes: the core idea | no | no | no | no |
| Tests the detectors themselves | yes: a planted incident nobody caught fails (R9) | no | no | no | no |
| Checks a fix didn't loosen a contract | yes: contracts are protected paths | no | no | no | no |
| Simulated time (days in minutes) | yes | no | no | no | no |
| Realistic data | basic until SDV is in | your data | your data | your data | strong |
| Works on production | no: before production, and nightly | yes | yes | yes, on a copy | - |
| Product, UI, support | no: a library and a course | yes | yes | yes | yes |
| Status | designed; Studio built | mature | mature | mature | mature |
Where others are ahead: all of them are real products in use today; observability catches production surprises nobody planted; diff tools test changes on real data; synthetic-data vendors make far more realistic data. DataTwin plans to use SDV rather than compete.
What limits are we facing?
Our limits today
- It isn't built yet. The biggest limit. The design is written and the learning studio is real, but the data world, the pretend sources, the checks and the 30-day run are phases 6 to 8. Everything in Part 2 past the Studio is how it will work, not how it works today.
- Pretend data is cleaner than real data. Real sources send odd encodings, half-written files, duplicate keys with different values, time zones that change in March. Faker and even SDV only make what they are told to. Replay mode (recorded real extracts) and shadow mode are the answer, and both are planned.
- The tests are only as complete as the manifest. If a source or rule is not declared, no incident will be planted for it. The same is true of AgentTwin and its spec, and the same answer applies: every production incident becomes a planted one.
- Laptop scale hides some problems. A lesson from building the Studio: at small volumes, some before-and-after numbers don't move. Skew, spill and cost only show at size. Those stories must be rescaled or labelled illustrative, never passed off as measured.
- Some managed-platform behaviour can't run locally. Serverless compute, Photon and Unity Catalog grants are Databricks-only. The engine internals are recorded on open-source Spark and Delta; the Databricks binding has to be tested on Databricks Free Edition, which has quotas.
- Dashboards are judged by their numbers. DataTwin checks that the tile and the conversation UI give the world's number. It does not judge whether a chart is clear, or whether the metric is the right one for the business.
- One domain so far. The clothing shop. A second domain is planned to prove the specs don't secretly assume retail.
Which open-source tools does it use?
The open-source tools
Like AgentTwin, DataTwin is an assembler: it uses proven open-source tools behind ports and builds only what nobody does.
| Job | Tool | Status | Why |
|---|---|---|---|
| The engine | Apache Spark 3.5 and Delta Lake 3.2 (open source) | in use | Run for real, never mocked: internals are visible locally and free |
| Data quality | PyDeequ; Spark SQL rules | in use | Both behind one port in the Studio's data quality scene |
| World, clock, faults, runner | AgentTwin core | in use | Extended, not copied: one world format for agents and data |
| Test runner, edge values | pytest, Hypothesis | in use | Already used by AgentTwin |
| Second data quality engine | Great Expectations | planned | Proves the port: same rules, other engine |
| Rows for the world | Faker, then SDV | planned | Faker for structure; SDV for realistic distributions and volume |
| The world's clock | time-machine, SimPy | planned | Both in use in the AgentTwin Lab |
| Faults on the wire | Toxiproxy | planned | Slow, dropped or cut connections to an API or database |
| Real stores per run | Testcontainers | planned | A real Postgres or Kafka started for one run, then thrown away |
| Lineage | OpenLineage, Marquez | planned | Rule R8 (data flows one way) is checked against lineage |
| Profiling | ydata-profiling | evaluating | Profiles that change when the data does |
| Change data capture, streams | Debezium, Redpanda | later | With the streaming lane (phase 6) |
| Branch per run | lakeFS | evaluating | Each simulated day on its own branch; cheap rollback |
Several of these (Faker, SDV, time-machine, SimPy, Toxiproxy, Testcontainers, Hypothesis) have a one-card explainer on the AgentTwin tools page.
FAQ
Questions people ask
Is DataTwin a digital twin?
Not in the usual sense. A digital twin copies a real machine to watch and predict it. DataTwin keeps your data platform real and copies the business and source systems that feed it, so you can break the platform on purpose before real data does.
How is it different from AgentTwin?
Same idea, different subject. AgentTwin keeps an AI agent real and simulates its customers and tools. DataTwin keeps a data platform real and simulates its sources and their incidents. DataTwin extends AgentTwin's world format rather than starting a new one, so the clock, faults and run modes are shared.
Is it built?
Partly. The DataTwin Studio, where you learn by watching real Spark and Delta runs, is built and live. DataTwin as a testing tool for your platform is designed and planned for early 2027. This page marks every feature as built, partly built, or to build.
Why not just use Great Expectations or Monte Carlo?
Keep them: they are what DataTwin tests. Rule tools check rows against rules you wrote; observability tools check today against history. Neither knows the right answer. DataTwin's world made the data, so it knows the true total, and it checks that your rules and monitors actually fire when an incident is planted.
Can't I test on a copy of production instead?
Yes, and you should, for changes. But production only contains the incidents that already happened. DataTwin lets you plant the ones that haven't happened yet (a renamed column, a duplicate day, a dark source), on a calendar, as often as you like.
Is 100% coverage possible?
100% of what a platform owes can be accounted for: every declared incident kind planted and caught, every owed check proved or signed off. 100% of everything real data can do cannot. See the section on coverage.
Does it work only on Databricks?
No. The world and the checks name no product. Databricks is the first binding; Microsoft, AWS, GCP and an open-source binding are planned.
Next
Where to go next
Learn the problems DataTwin plants, recorded for real: the DataTwin Studio, the Problem Lab, performance stories, data quality.
The architecture it tests: the data platform architecture and the medallion, explained.
The sibling: AgentTwin, explained. Code it extends: github.com/dataagentsai/agenttwin (Apache 2.0).