Home › Learn › Data platform architecture › Test architecture
Data platform architecture · testing
Test architecture for a data platform
How to know the numbers are right before anyone sees them: nine kinds of test, from reading the code to comparing a new version with production on real data, and one release report that decides.
The one-minute version
Cheap tests catch most problems; real-data tests catch the rest
Push logic into pure functions. Then fast unit tests on a laptop cover most of the logic, and slow tests only need to prove the plumbing.
Test each hop on its own. Source to bronze, bronze to silver, silver to gold, gold to metrics: a failure points at one place.
Use simulation and real data, not one or the other. DataTwin plants known faults; a pre-production run on real data, compared with production, catches what nobody imagined.
Release on evidence. Every test writes its result to one record; the release report is generated from it, never ticked by hand.
Design for testing
Logic in the middle, input and output at the edges
A pure function gives the same output for the same input and touches nothing else: it reads no table, writes no file, calls no service and does not look at the clock. For data code that means DataFrame in, DataFrame out. Everything else is impure.
Static analysis enforces it: the shift-left bundle marks every function pure, impure or unknown, shows where an impure function could be pure, and fails a pull request that adds impure functions to a transformation module.
Nine kinds of test
From reading the code to comparing with production
| Kind | What it proves | Data used | Where and when | Built with |
|---|---|---|---|---|
| 1 · Static analysis | Problems found by reading the code: bad PySpark patterns, collect() on big data, cartesian joins, MERGE without keys, SQL injection, secrets, costly job settings, metric view mistakes; and whether each function is pure | none | Laptop, pull request | Shift-left bundle: 131 checks; purity classifier (pure, impure, unknown, and 'could be pure' hints) |
| 2 · Unit tests | Each pure function returns the right rows, case by case: new, changed, unchanged, deleted, duplicated, late, across a clock change | Small tables written inside the test (table-driven) | Laptop, pull request; seconds | pytest + local Spark and Delta; the shift-left test kit for SCD2, idempotency, duplicates and dropped rows |
| 3 · Coverage and mutation | The tests really exercise the logic, and fail when the logic is broken on purpose | — | Pull request | coverage.py (branch coverage) on the pure modules; mutation testing on critical logic |
| 4 · Contract tests | Each data contract is valid, and a change does not break the teams that use the data | — | Pull request | ODCS schema validation + breaking-change comparison with the last released version |
| 5 · Integration, one hop at a time | One hop works on the real platform: source → bronze, bronze → silver, silver → gold, gold → metrics (table below) | Fixed fixtures with known answers | Dev environment, after merge, on a fresh schema | Deployment bundle + the platform's job runner |
| 6 · End to end, synthetic | All hops together give the expected numbers from source to metric | A fixed synthetic dataset with known answers | Dev | Same as above, run as one job |
| 7 · DataTwin | The platform survives bad days: 30 simulated days with faults planted on purpose; it catches each one, holds bad data back, and every number matches the simulated world's truth | Generated by DataTwin | Dev, nightly; pre-production, before each release | DataTwin |
| 8 · End to end on real data | The new version works on real data, with mess nobody imagined | Zero-copy clones of production tables; personal data already tokens, recipes and the token vault left out | Pre-production, before release | Table clones (Delta shallow clone) + the same jobs |
| 9 · Parallel run and data compare | The new version gives the same numbers as production, or differs exactly where we meant it to | Same real input to the old and the new version | Pre-production, for any change to existing logic | Our compare job: full outer join on keys, counts and sums per key, every metric's result, with tolerances |
Integration
One integration test per hop
Each hop between layers has its own integration test, with a fixed input and a known expected output, run on the real platform in the dev environment on a fresh schema. When one fails, you know which hop broke.
| Hop | Fixed input | The test passes when |
|---|---|---|
| Source → bronze | Sample files and change records in the landing folder | Every row lands once; arrival details added; the same file again changes nothing; unexpected new columns kept; run log and watermark right |
| Bronze → silver | A bronze table with duplicates, late rows, a schema change, a delete, and a record changed twice in a day | Types and standardisation right; the contract check stops the schema change; duplicates removed; SCD2 versions and dates right; bad rows quarantined with a reason; a re-run changes nothing; rows in = rows out + quarantined |
| Silver → gold | Silver tables including a missing dimension row and a late-arriving contributing fact | The fact's level of detail is right; no row dropped (reconciliation); an inferred member is created, then filled when the dimension arrives; changed windows recomputed; totals match silver |
| Gold → metrics | Gold tables with known answers | Each metric returns the expected number (for example scrap rate for plant A on day 3 = 2.4%); joins and filters right; masked columns stay masked |
Simulation and real data
DataTwin is not the whole story
DataTwin runs the platform in a simulated business and plants faults on purpose, so the bad days happen on schedule. But a simulation only contains what someone thought of. Real data has formats, values and volumes nobody wrote down. So before release the new version also runs end to end on real data in pre-production, using zero-copy clones of production tables: instant, cheap, and production is never touched. Personal data stays masked and restricted data is left out.
| Test | Proves | Catches | Cannot catch |
|---|---|---|---|
| DataTwin (simulation) | Faults we know about, on purpose, every night | Late files, duplicates, schema changes, a job dying halfway, clock changes | Cannot know this company's real formats and odd values |
| End to end on real data | This company's real data, before release | Odd values, real volumes, real edge cases nobody wrote down | Only the faults that happen to be in the data that week |
| Parallel run | That nothing changed that should not have | A fix that also moves a number it should not | Needs a production version to compare with |
Old version against new
Parallel run, canary, and what A/B testing means here
For any change to logic that already runs in production, the old and new versions run side by side on the same real input and the outputs are compared: row by row on keys, counts and sums per key, and the result of every metric. "Scrap rate per plant per day matches to 0.01%, except plant B, where the bug fix moved it as intended" is release evidence.
| Name | What it is | Use it for | Verdict |
|---|---|---|---|
| Parallel run (shadow run, data diff) | Old and new pipeline on the same data; outputs compared | Pipelines: every change to existing logic | Pursue |
| Canary release | The new version switched on for one plant or one business unit first, watched, then rolled out | Risky pipeline changes | Pursue for risky changes |
| A/B testing (true sense) | Users split between two versions; their behaviour measured | The conversation UI and Genie: two ways of answering, measured on accuracy and ratings | Only for user-facing AI, not pipelines |
Evidence
Profiles, checks and one release report
Every test run in the dev and pre-production environments also profiles its outputs and compares them with the last good run, and runs every quality rule and reconciliation. All results go to one test-run record, and the release report is generated from it.
| Evidence | Passes when |
|---|---|
| Static analysis | No must-fix finding; impure functions in transformation modules did not grow |
| Unit tests and coverage | All pass; branch coverage on pure modules at or above target; mutation score on critical logic |
| Contract tests | All contracts valid; no unapproved breaking change |
| Integration, per hop | All four hops pass, including the re-run test |
| DataTwin | Every planted fault caught and held; dashboard matches the simulated truth |
| Real-data run and compare | Differences from production are zero, or each is explained and approved |
| Profile comparison | Row counts, empty rates, distinct counts and ranges within tolerance of the last good run |
| Quality rules and reconciliation | All stop-level rules pass; source totals = gold totals; rows in = rows out + quarantined |
Targets
What "good enough to release" means
| Target | Rule | Why |
|---|---|---|
| Purity | Transformation modules are mostly pure; the number of impure functions in them may not grow in a pull request | Keeps logic unit-testable |
| Branch coverage | 90% on pure modules; no target on the thin read and write layer | Integration tests cover the edges |
| Mutation testing | On critical logic only: SCD2, deduplication, inferred members, key metrics | Proves the tests would notice a wrong answer |
| Integration | One test per hop, every merge | A failure points at one hop |
| Real data | Pre-production run + parallel run for every change to existing logic | Catches what simulation cannot |
| Canary | One plant or unit first for risky changes | Limits the blast radius |
When each test runs
Six gates from laptop to production
Tools
What does each job
| Job | Any platform | On Databricks | Comes from |
|---|---|---|---|
| Static analysis | Shift-left bundle (Ruff, sqlfluff, dbt-bouncer underneath) | Same, plus Databricks job and SQL checks | Ours (built) |
| Unit tests | pytest + local Spark and Delta | Same; no cluster needed | Open source |
| Coverage and mutation | coverage.py, mutmut | Same | Open source |
| Contract tests | ODCS + datacontract CLI | Same | Open standard |
| Integration and end to end | Deployment bundle per environment, fresh schema per change | Databricks Asset Bundles, Lakeflow Jobs | Platform |
| Real-data clones | Zero-copy table clones | Delta shallow clone in Unity Catalog | Platform |
| Simulation | DataTwin | DataTwin on Databricks | Ours (designed; Studio built) |
| Profiles and quality | A profiler and a quality engine behind one port | DQX profiler and checks | Databricks Labs |
| Compare | Key-based compare job + metric comparison | Same, in SQL | Ours |
Product names change often; check the current docs before you build.
Questions
Questions people ask
Why not just test on production data and look at the dashboard?
Because by then the wrong number has already been seen. Most problems are cheaper to catch in seconds on a laptop than in hours on a cluster, and the ones that only show on real data are caught in pre-production, before release.
Is DataTwin enough for end-to-end testing?
No. DataTwin proves the platform survives the faults we know about. A run on real data in pre-production, compared with production, proves it works on this company's actual data. We need both.
Which one is A/B testing?
In data pipelines, what people usually mean is a parallel run: old and new versions on the same data, outputs compared. True A/B testing splits users between two versions and belongs to user-facing AI like a conversation UI.
Why insist on pure functions?
A pure function can be tested with a small table in seconds, without a cluster. If most logic is pure, unit tests cover most of it, and the slow integration tests only need to prove the plumbing.
Who owns the test strategy?
The test strategy lead in the centre of excellence writes it into the specs; data engineers write the unit and integration tests; the release gate reads the evidence.
Next
Where to go next
The strategy in practice: Test strategy for a data platform, with seventeen kinds of test on a reference build, what blocks a release and what is proven so far.
The checks themselves: Shift left: fail fast on data code, with pure functions, table-driven tests, SCD2 tests, coverage and mutation testing.
The faults to test for: the Problem Lab, nineteen data problems run for real, each with the naive fix and the right one.
Simulation: DataTwin, explained. On a real stack: testing on AWS + Databricks.