Home › Learn › Data platform architecture › Test architecture

Data platform architecture · testing

Test architecture for a data platform

How to know the numbers are right before anyone sees them: nine kinds of test, from reading the code to comparing a new version with production on real data, and one release report that decides.

↑ Data platform architectureShift leftDataTwinOn AWS + DatabricksTest strategy in practice

The one-minute version

Cheap tests catch most problems; real-data tests catch the rest

Push logic into pure functions. Then fast unit tests on a laptop cover most of the logic, and slow tests only need to prove the plumbing.

Test each hop on its own. Source to bronze, bronze to silver, silver to gold, gold to metrics: a failure points at one place.

Use simulation and real data, not one or the other. DataTwin plants known faults; a pre-production run on real data, compared with production, catches what nobody imagined.

Release on evidence. Every test writes its result to one record; the release report is generated from it, never ticked by hand.

Design for testing

Logic in the middle, input and output at the edges

A pure function gives the same output for the same input and touches nothing else: it reads no table, writes no file, calls no service and does not look at the clock. For data code that means DataFrame in, DataFrame out. Everything else is impure.

Read (impure)tables, files, APIs,metadata, the clockthin · tested by integrationPure core: all the business logic· dedupe and latest-wins· SCD2 versions and dates· inferred members· OEE and scrap formulas· changed-window recomputeDataFrame in, DataFrame out · tested by unit tests, in secondsWrite (impure)MERGE, run_log,watermark, alertsthin · tested by integration
Put all business logic in the pure core. The read and write layers stay thin, so there is little in them to go wrong.

Static analysis enforces it: the shift-left bundle marks every function pure, impure or unknown, shows where an impure function could be pure, and fails a pull request that adds impure functions to a transformation module.

Nine kinds of test

From reading the code to comparing with production

9 · Parallel run vs productionreal data, old vs new8 · End to end on real datapre-production, clones7 · DataTwinsimulated bad days6 · End to end, syntheticknown answers5 · Integration, one hop at a timefour hops4 · Contract testspromises between teams3 · Coverage and mutationdo the tests bite?2 · Unit tests on pure functionsmost of the logic1 · Static analysis (shift left)read the code, run nothingmore realslower, costliercheaper, fasterrun on every change
Many cheap tests at the bottom, few costly ones at the top. Each kind catches something the others cannot.
KindWhat it provesData usedWhere and whenBuilt with
1 · Static analysisProblems found by reading the code: bad PySpark patterns, collect() on big data, cartesian joins, MERGE without keys, SQL injection, secrets, costly job settings, metric view mistakes; and whether each function is purenoneLaptop, pull requestShift-left bundle: 131 checks; purity classifier (pure, impure, unknown, and 'could be pure' hints)
2 · Unit testsEach pure function returns the right rows, case by case: new, changed, unchanged, deleted, duplicated, late, across a clock changeSmall tables written inside the test (table-driven)Laptop, pull request; secondspytest + local Spark and Delta; the shift-left test kit for SCD2, idempotency, duplicates and dropped rows
3 · Coverage and mutationThe tests really exercise the logic, and fail when the logic is broken on purpose—Pull requestcoverage.py (branch coverage) on the pure modules; mutation testing on critical logic
4 · Contract testsEach data contract is valid, and a change does not break the teams that use the data—Pull requestODCS schema validation + breaking-change comparison with the last released version
5 · Integration, one hop at a timeOne hop works on the real platform: source → bronze, bronze → silver, silver → gold, gold → metrics (table below)Fixed fixtures with known answersDev environment, after merge, on a fresh schemaDeployment bundle + the platform's job runner
6 · End to end, syntheticAll hops together give the expected numbers from source to metricA fixed synthetic dataset with known answersDevSame as above, run as one job
7 · DataTwinThe platform survives bad days: 30 simulated days with faults planted on purpose; it catches each one, holds bad data back, and every number matches the simulated world's truthGenerated by DataTwinDev, nightly; pre-production, before each releaseDataTwin
8 · End to end on real dataThe new version works on real data, with mess nobody imaginedZero-copy clones of production tables; personal data already tokens, recipes and the token vault left outPre-production, before releaseTable clones (Delta shallow clone) + the same jobs
9 · Parallel run and data compareThe new version gives the same numbers as production, or differs exactly where we meant it toSame real input to the old and the new versionPre-production, for any change to existing logicOur compare job: full outer join on keys, counts and sums per key, every metric's result, with tolerances

Integration

One integration test per hop

Each hop between layers has its own integration test, with a fixed input and a known expected output, run on the real platform in the dev environment on a fresh schema. When one fails, you know which hop broke.

SourceTest 1ingestionBronzeTest 2bronze → silverSilverTest 3silver → goldGoldTest 4gold → metricsSemanticEnd to end: all hops together, synthetic → DataTwin → real data in pre-production → parallel run
Four hop tests, then end-to-end runs over all hops together.
HopFixed inputThe test passes when
Source → bronzeSample files and change records in the landing folderEvery row lands once; arrival details added; the same file again changes nothing; unexpected new columns kept; run log and watermark right
Bronze → silverA bronze table with duplicates, late rows, a schema change, a delete, and a record changed twice in a dayTypes and standardisation right; the contract check stops the schema change; duplicates removed; SCD2 versions and dates right; bad rows quarantined with a reason; a re-run changes nothing; rows in = rows out + quarantined
Silver → goldSilver tables including a missing dimension row and a late-arriving contributing factThe fact's level of detail is right; no row dropped (reconciliation); an inferred member is created, then filled when the dimension arrives; changed windows recomputed; totals match silver
Gold → metricsGold tables with known answersEach metric returns the expected number (for example scrap rate for plant A on day 3 = 2.4%); joins and filters right; masked columns stay masked

Simulation and real data

DataTwin is not the whole story

DataTwin runs the platform in a simulated business and plants faults on purpose, so the bad days happen on schedule. But a simulation only contains what someone thought of. Real data has formats, values and volumes nobody wrote down. So before release the new version also runs end to end on real data in pre-production, using zero-copy clones of production tables: instant, cheap, and production is never touched. Personal data stays masked and restricted data is left out.

TestProvesCatchesCannot catch
DataTwin (simulation)Faults we know about, on purpose, every nightLate files, duplicates, schema changes, a job dying halfway, clock changesCannot know this company's real formats and odd values
End to end on real dataThis company's real data, before releaseOdd values, real volumes, real edge cases nobody wrote downOnly the faults that happen to be in the data that week
Parallel runThat nothing changed that should not haveA fix that also moves a number it should notNeeds a production version to compare with

Old version against new

Parallel run, canary, and what A/B testing means here

For any change to logic that already runs in production, the old and new versions run side by side on the same real input and the outputs are compared: row by row on keys, counts and sums per key, and the result of every metric. "Scrap rate per plant per day matches to 0.01%, except plant B, where the bug fix moved it as intended" is release evidence.

NameWhat it isUse it forVerdict
Parallel run (shadow run, data diff)Old and new pipeline on the same data; outputs comparedPipelines: every change to existing logicPursue
Canary releaseThe new version switched on for one plant or one business unit first, watched, then rolled outRisky pipeline changesPursue for risky changes
A/B testing (true sense)Users split between two versions; their behaviour measuredThe conversation UI and Genie: two ways of answering, measured on accuracy and ratingsOnly for user-facing AI, not pipelines

Evidence

Profiles, checks and one release report

Every test run in the dev and pre-production environments also profiles its outputs and compares them with the last good run, and runs every quality rule and reconciliation. All results go to one test-run record, and the release report is generated from it.

static analysis + purityunit + coverageintegration, per hopDataTwinreal-data run + compareprofile comparisonTest-run recordone run id per releaseRelease reportevery line from a test,never ticked by handRelease gateall pass, or stop
Nobody ticks the release report by hand; every line comes from a test.
EvidencePasses when
Static analysisNo must-fix finding; impure functions in transformation modules did not grow
Unit tests and coverageAll pass; branch coverage on pure modules at or above target; mutation score on critical logic
Contract testsAll contracts valid; no unapproved breaking change
Integration, per hopAll four hops pass, including the re-run test
DataTwinEvery planted fault caught and held; dashboard matches the simulated truth
Real-data run and compareDifferences from production are zero, or each is explained and approved
Profile comparisonRow counts, empty rates, distinct counts and ranges within tolerance of the last good run
Quality rules and reconciliationAll stop-level rules pass; source totals = gold totals; rows in = rows out + quarantined

Targets

What "good enough to release" means

TargetRuleWhy
PurityTransformation modules are mostly pure; the number of impure functions in them may not grow in a pull requestKeeps logic unit-testable
Branch coverage90% on pure modules; no target on the thin read and write layerIntegration tests cover the edges
Mutation testingOn critical logic only: SCD2, deduplication, inferred members, key metricsProves the tests would notice a wrong answer
IntegrationOne test per hop, every mergeA failure points at one hop
Real dataPre-production run + parallel run for every change to existing logicCatches what simulation cannot
CanaryOne plant or unit first for risky changesLimits the blast radius

When each test runs

Six gates from laptop to production

Laptoppre-commit· shift-left checks· purity check· unit testssecondsPull requestCI· full shift-left scan· unit + coverage· contract checksminutesDevafter merge· one test per hop· end to end, synthetic· fresh schema15–30 minDataTwinnightly· pretend plant, 30 days· planted faults· caught? held?an hourPre-productionbefore release· real data (clones)· parallel run vs prod· profile comparean hour or twoProductionevery run· contract check· quality rules· freshness alertsalways oncheapest and fastest on the left · a problem caught one gate earlier costs a fraction to fix
The earlier a problem is caught, the cheaper it is to fix.

Tools

What does each job

JobAny platformOn DatabricksComes from
Static analysisShift-left bundle (Ruff, sqlfluff, dbt-bouncer underneath)Same, plus Databricks job and SQL checksOurs (built)
Unit testspytest + local Spark and DeltaSame; no cluster neededOpen source
Coverage and mutationcoverage.py, mutmutSameOpen source
Contract testsODCS + datacontract CLISameOpen standard
Integration and end to endDeployment bundle per environment, fresh schema per changeDatabricks Asset Bundles, Lakeflow JobsPlatform
Real-data clonesZero-copy table clonesDelta shallow clone in Unity CatalogPlatform
SimulationDataTwinDataTwin on DatabricksOurs (designed; Studio built)
Profiles and qualityA profiler and a quality engine behind one portDQX profiler and checksDatabricks Labs
CompareKey-based compare job + metric comparisonSame, in SQLOurs

Product names change often; check the current docs before you build.

Questions

Questions people ask

Why not just test on production data and look at the dashboard?

Because by then the wrong number has already been seen. Most problems are cheaper to catch in seconds on a laptop than in hours on a cluster, and the ones that only show on real data are caught in pre-production, before release.

Is DataTwin enough for end-to-end testing?

No. DataTwin proves the platform survives the faults we know about. A run on real data in pre-production, compared with production, proves it works on this company's actual data. We need both.

Which one is A/B testing?

In data pipelines, what people usually mean is a parallel run: old and new versions on the same data, outputs compared. True A/B testing splits users between two versions and belongs to user-facing AI like a conversation UI.

Why insist on pure functions?

A pure function can be tested with a small table in seconds, without a cluster. If most logic is pure, unit tests cover most of it, and the slow integration tests only need to prove the plumbing.

Who owns the test strategy?

The test strategy lead in the centre of excellence writes it into the specs; data engineers write the unit and integration tests; the release gate reads the evidence.

Next

Where to go next

The strategy in practice: Test strategy for a data platform, with seventeen kinds of test on a reference build, what blocks a release and what is proven so far.

The checks themselves: Shift left: fail fast on data code, with pure functions, table-driven tests, SCD2 tests, coverage and mutation testing.

The faults to test for: the Problem Lab, nineteen data problems run for real, each with the naive fix and the right one.

Simulation: DataTwin, explained. On a real stack: testing on AWS + Databricks.