Home › Learn › Data platform architecture › Test strategy

Data platform architecture · testing

Test strategy for a data platform

What we test, when each test runs, and what blocks a release: seventeen kinds of test on a reference build, with what is proven so far and how each one works.

↑ Data platform architectureTest architecture (the design)Shift leftDataTwin

Part one · crisp

The strategy on one page

Aim. A wrong number never reaches a dashboard without someone being told, and a bad change never reaches production. Tests run before the merge, at release, and inside every production run.

Five principles

Test against known answers. A simulated company produces the source data and also knows the correct total at every layer for every day. The platform's output must equal those totals.

Plant every fault you claim to handle. Duplicates, late files, renamed columns and mass deletes are put into the data on purpose. Each one must be caught at the layer that is supposed to catch it.

Compare versions on the same input. A change runs beside the current version on copies of production tables. Any difference nobody declared blocks the release.

Production runs test themselves. Contract rules, reconciliation and row counts run on every load, so a problem in real data is caught the same way as a planted one.

Every rule has a test or a written waiver. A rule catalog lists what must be true of the platform. Coverage is counted, not assumed.

Four questions the tests answer

QuestionTest kinds that answer itWhen
A. Is the code right?Static checks, unit tests, contract tests, engine conformance, spec and infrastructure testsEvery commit and pull request
B. Is the data right?World against known answers, planted faults, metric known answers, checks inside every runEvery merge, every deploy, every production run
C. Is this change safe?Release gate with a data comparison, version comparisons, integration run in devBefore every release to production
D. Does it hold up in operation?Scenario tests, recovery tests, performance and scale, security and privacy, a long unattended runOn a schedule and before go-live

The test architecture page gives the design behind this: why logic lives in pure functions, and the nine kinds of test in general terms. This page shows the same ideas running on one build.

When each test runs

Six steps a change climbs

Each step is slower and closer to real data than the one before, so cheap tests catch most problems first.

1. CommitStatic checks onchanged codeseconds2. Pull requestUnit and contracttests, two engines,specs, infrastructureabout 3 minutes3. MergeThree-day simulatedworld: every layerequals known answers,all faults handledabout 3 minutes4. Deploy to devEvery generated jobruns once on the realplatform, then theworld is checked thereabout 15 minutes5. ReleaseThe change on copiesof production tables,compared per window;a person approvesminutes + approval6. Every runContract rules,quarantine, sourceagainst silver, rowledger, metrics,freshness, alertsinside the load
On the reference build, steps 1 to 3 run today. Steps 4 and 5 are written and wait for a paid cloud workspace. Step 6 runs inside every proof run.

The test kinds

Seventeen kinds of test, by the question they answer

#KindThe question it answersRuns at stepState on the reference build
A. Is the code right?
1Static checksDoes the code break a known rule before it runs?1, 2partly no secret scan yet
2Unit testsDoes each function give the right output for each listed case?2done 1,269 pass
3Contract testsDoes every adapter behind an interface behave the same?2done
4Engine conformanceDoes the same logic give the same result on two engines?2done 143 cases
5Spec and infrastructure testsAre the manifest, contracts, rules and infrastructure valid?2done
B. Is the data right?
6World against known answersDoes every layer hold exactly what it should, every day?3, 4done laptop and a free Databricks workspace
7Planted faultsIs each failure caught at the right layer, with none silent?3, 4done 11 of 11
8Metric known answersIs the business number the same in the metric, gold and the world?4, 6done 8 of 8
9Checks inside every runIs today's real load complete, valid and reconciled?6done
C. Is this change safe?
10Release gate with data comparisonDoes the change alter any number nobody declared?5partly built; not yet run against a production environment
11Version comparisonsDo two ways of doing the same job agree?2, 3partly four built; format benchmark not built
12Integration run in devDoes every deployed job run once and pass its checks?4not yet written; waits for a paid cloud workspace
D. Does it hold up in operation?
13Scenario testsDoes the right alert fire once, and none on clean days?2done laptop
14Recovery testsCan a bad load be withdrawn, rerun and rebuilt with nothing changed?2, 3partly withdraw not yet proven on the real platform
15Performance and scaleDoes it finish in time at full volume?schedulednot yet
16Security and privacyAre secrets, personal data and access kept as declared?2, 4partly scan and access check not running
17Long unattended runDoes it stay right for 30 days with nobody watching?before go-livenot yet

Across all of them: rule coverage. The rule catalog holds 89 rules. At the last count 47 were named in code or tests. The target is 89 of 89, each with a passing test or a written waiver.

Gates

What blocks a release

GateBlocks whenWho can override
Pull requestAny static check, unit, contract, conformance, spec or infrastructure test fails; a new shift-left finding appears; the generated jobs are out of dateNobody. A shift-left exception needs a written reason on the line
MergeThe three-day world differs from the known answers at any layer, or a planted fault is not handledNobody
Deploy to devA generated job fails its checks on the real platformNobody
Release to productionThe change differs from production in a measure, a key set or a window, and the difference is not declared and explained; or the result misses the known answers beyond the agreed toleranceThe data owner, by writing the explanation. The release also needs a named reviewer's approval
Production runA critical contract rule is broken (the batch is held and nothing is written); a full copy would delete more than the allowed share; a delivery arrives twiceThe data owner, through a recorded rerun or correction plan

Honest state

What is not proven yet, in the order we would close it

#Not proven yetWhy it matters
1Integration run in dev: every generated job once, checks must passIt proves the deployed jobs work, not only the proof run
2The three-day world in CI on every merge, in both run modesToday it is run by hand
3Both silver engines in CIOnly the job engine runs there; the declarative engine is checked by hand
4Release gate run from CI against a production environmentThe gate blocks planted bad changes, but only in tests
5Secret scan in CIRedaction is tested, but nothing scans commits for secrets
6Personal-data scan after every runTokens are applied at landing, but no scan proves that no clear value got through
7Hop times and gold-ready-by-morning measured on the real platformTimeliness is measured on the laptop only
8Full-scale run, 14 daysThe laptop ran out of memory at full scale
9Access check against the catalogNeeds a workspace with account-level groups
10Rule coverage from 47 to 8942 rules have no named test or waiver
11Table format benchmarkThe format is chosen by decision, not by measurement
1230 days unattendedThe last proof before calling it production-ready

Part two · how

How each kind works

A. Is the code right?

1. Static checks partly no secret scan yet

What. Checks that read the code without running it.

How. A linter for style and common mistakes, strict type checks on the cloud adapters, and the shift-left checks for PySpark and SQL anti-patterns. The shift-left scan compares with a baseline: old findings are listed there, and any new finding blocks the pull request.

Left to do. Add a secret scan. Widen strict typing from the cloud adapters to the whole core.

2. Unit tests done 1,269 pass

What. Each pure function is tested with a list of cases: a name, the input and the expected output.

How. The cases sit in a table at the top of the test file and one test runs them all, so a new case is one new row. Logic is kept in pure functions so most of it is tested with no cluster and no database. The whole set runs in under a minute.

3. Contract tests done

What. Every outside system sits behind an interface (a port). A contract test is one table of cases per port that every implementation (adapter) must pass.

How. The same cases run against the in-memory adapter, the real one, and a simulator of the cloud service. Examples: 64 cases for the metadata store on memory and PostgreSQL; 23 for the table store on Delta and Iceberg; 8 history cases that both silver engines pass. Moving to another cloud means writing new adapters and passing the same tables.

4. Engine conformance done 143 cases

What. The silver steps and the gold builds give identical results on Spark and on DuckDB.

How. The same test tables run once per engine in separate processes, and hashed keys must be equal on both. The first run found 20 differences, 6 of them silent wrong values.

5. Spec and infrastructure tests done

What. The things that drive the platform are valid before anything runs.

How. Every rule, capability and decision record is linted. Data contracts are validated against the open data contract standard. The generated jobs must equal what the manifest would generate now. Infrastructure code is formatted, validated and planned against mocked providers for each environment.

B. Is the data right?

6. World against known answers done laptop and a free Databricks workspace

What. The end-to-end test. A simulated company writes a day of data into mock source systems, the platform loads it, and each layer is compared with the totals the world knows are right.

How. For each day: bronze row counts per dataset, silver measures per dataset (some per hour) and gold measures per table. A day passes only when every number is equal. It runs with every hop in one process, and also with ingestion, silver and gold as separate processes, the shape deployed jobs have.

Left to do. Run it in CI on every merge. Run it at full scale for 14 days.

7. Planted faults done 11 of 11

What. Eleven kinds of fault are put into the world's data on set days. Each names the check that must catch it.

How. After each day the run gives a verdict per fault: handled, flagged or missed. A missed fault fails the run. A compressed calendar puts all eleven on days 2 and 3, so a three-day run covers them. The faults: a file sent twice, a file sent late, rows deleted at the source, a column renamed, readings out of order, a full copy that is 40% short, a correction to past data, a new dealer whose record arrives after its orders, a change feed that drops rows, a stream replayed, and a promised date changed out of order. A 14-day fixture also checks that nothing fires on a clean day.

8. Metric known answers done 8 of 8

What. The number a business user sees equals gold and equals the world's truth.

How. After each run the certified metrics are recreated and asked fixed questions, for example order lines in November. Each answer is compared with gold and with the world. The plain-language question interface is checked the same way: its answer must equal the metric.

9. Checks inside every run done

What. Tests that run on real data in production, as part of the load.

How. Contract rules by severity: a broken critical rule holds the whole batch, a major one sends the row to quarantine, a minor one flags it. A second copy of a delivery is refused, and a full copy that would delete too much is held. The live source is compared with silver every day, naming the exact missing keys and the hop that lost them. In gold, rows in must equal rows out at every join, and the join that changes the count is named. Every check writes a step in the run log, which feeds the dashboards and the alerts.

C. Is this change safe?

10. Release gate with data comparison partly built; not yet run against a production environment

What. The data form of A/B testing. The new version and the current version run on the same data, and the outputs are compared.

How. Each production table the change touches is shallow-cloned, which copies only the table's file list and so costs almost nothing. The change runs against the clones. Row counts, measure totals and key sets are compared with production, in total and per window, for example per day. A difference blocks unless the change declared it and someone wrote the explanation. The clones are dropped whatever happens.

Left to do. Run it from CI against a real production environment. Today it blocks two planted bad changes in tests.

11. Version comparisons partly four built; format benchmark not built

What. Where the platform offers two ways to do one job, both must agree.

How. Spark against DuckDB. The job silver engine against the declarative one: equal rows on six datasets. Delta against Iceberg: the same 23 cases. One process against separate processes: every layer equal, with the same fault verdicts.

Left to do. A format benchmark: Iceberg, Delta and Hudi on the same world, measured for speed and cost.

12. Integration run in dev not yet written; waits for a paid cloud workspace

What. After a deploy to dev, one world day goes through cloud storage and every generated job runs once. Any failed check fails the pipeline.

How. The deploy pipeline calls it straight after the deploy, so a job that deploys but does not run is caught the same day.

Left to do. Run it. It needs the job generator updated for the current engine and a cloud account.

D. Does it hold up in operation?

13. Scenario tests done laptop

What. Eleven written stories of something going wrong in operation, each with the alert and the state expected afterwards.

How. Each scenario is a small file: the situation, the fault, and what must be true after. One engine plays it against the platform with a simulated clock. Noise rules are tested too: one alert per incident, none on clean days, and snoozes expire.

Left to do. Send alerts to a live channel and play the scenarios against the deployed platform.

14. Recovery tests partly withdraw not yet proven on the real platform

What. Proof that the platform can get back from a bad state.

How. Loading the same day twice adds no rows and changes no window. Withdrawing a batch voids it, rebuilds silver from bronze, and recomputes gold, replacing only the changed windows and logging each as a restatement. After old bronze batches are removed, silver rebuilt from what is left has the same current rows. A second run on a table already being written skips with a reason. A past window rebuilt from its run log equals what was published.

Left to do. Prove the withdraw path on the real platform.

15. Performance and scale not yet

What. It finishes in time at real volume.

How. Every hop stamps its time, and a report gives the delay per hop and names any breach. One day at one tenth scale takes about 5.8 minutes against a target of 6.

Left to do. Full scale for 14 days. Hop times and gold-ready-by-morning measured on the real platform. Cost per daily load, which needs a billed workspace.

16. Security and privacy partly scan and access check not running

What. Secrets, personal data and access stay as declared.

How. Error messages are redacted before they reach the run log. Personal values are replaced by tokens at landing. Column masks and grants are generated from one access model, with a check that compares the model with what the catalog holds.

Left to do. A scan of every layer for clear personal values, with planted ones to prove the scan works. A secret scan in CI. The access check on a workspace with account-level groups. Right to be forgotten.

17. Long unattended run not yet

What. Thirty days on a schedule with nobody watching, and at the end the dashboard equals the world.

How. It comes last because it needs everything else in place: deployed jobs, live alerts and a billed workspace.

Test data

Where the test data comes from

Source of test dataUsed byWhy
Small case tables written in the test fileUnit, contract and conformance testsThe reader sees the input and the expected output together
The simulated world, with a fixed seedWorld, fault, metric, recovery and performance testsThe same run gives the same data every time, and the world knows the right answer at every layer
Shallow clones of production tablesThe release gateReal data and real volume, with no copy and no risk to production
Real production dataChecks inside every runProblems that no simulator thought of

The simulated world runs at any scale: one tenth for the daily loop on a laptop, full scale on the real platform. Personal data in it is made up, so no real personal data is used in a test.

Environments

What runs where

EnvironmentWhat runs thereState
LaptopKinds 1 to 7, 11, 13 and 14: three days at one tenth scale, on Spark or DuckDBdone
CIKinds 1 to 5 on every pull request and pushdone the world run is not in CI yet
Dev on the real platformKinds 6 to 9 through the proof run; kind 12 after each deploypartly a free workspace stands in; kind 12 not running
ProductionKind 9 in every run; kind 10 before every releasenot yet not deployed

Adding a dataset

What a new dataset gets without anyone writing a test

Most tests come from the dataset's entry in the manifest and its data contract, so adding a dataset adds its tests.

Where the tests come fromWhat the dataset gets
From the data contractColumn, type and row rules run in every load, with quarantine by severity
From the manifest entryDuplicate-delivery refusal, the mass-delete guard for full copies, freshness, and the live source comparison for database sources
From the gold definitionThe row ledger at each join, completeness per window, and the per-window comparison at the release gate
What the author addsThe dataset in the simulated world with its known answers, and a planted fault if it has a failure mode the existing eleven do not cover

Words used

Words used on this page

WordMeaning
Known answersThe correct totals the simulated world works out for itself at every layer, used to judge the platform's output. Often called the oracle.
Table-driven testOne test that runs a list of cases, each a name, an input and an expected output.
Port and adapterA port is the interface the platform's logic talks to. An adapter is one implementation of it, for example PostgreSQL or object storage.
Shallow cloneA copy of a table that shares the original's data files, so it is quick and nearly free.
WindowThe slice of a table a run rebuilds, usually one business day.
RestatementA logged change to a number that was already published.
HopOne move of data between layers: landing to bronze, bronze to silver, silver to gold.
Shift leftCatching a problem earlier, at commit or pull request, instead of in production.

Questions

Questions people ask

How do you do A/B testing on a data platform?

Not in the classic sense of splitting live users between two versions, because a pipeline has no user behaviour to measure. The equivalent is to run the new version and the current version on the same data and compare the outputs. That is the release gate: the change runs on shallow clones of production tables, and any undeclared difference in a measure, a key set or a window blocks the release. True A/B testing belongs to user-facing AI, where two versions answer real questions and the answers are scored.

How is this page different from the test architecture page?

The test architecture page is the design: nine kinds of test and why each exists, with no platform named. This page is the strategy as it is practised on a reference build: seventeen kinds grouped by the question they answer, when each runs, what blocks a release, and what is proven so far.

Why simulate a company instead of using sample data?

Sample data has no right answer attached. A simulated company with a fixed seed produces the same data every time and also knows the correct total at every layer, so the test is a comparison of numbers and not a judgement.

Which test would you add first to an existing platform?

The checks inside every run: contract rules with quarantine, and a daily comparison of the live source with silver. They need no simulator and they catch real problems from the first day.

What does a new dataset need before it can go live?

Its contract and manifest entry give it most of its tests. The author adds the dataset to the simulated world with its known answers, and a planted fault if it has a failure mode the existing ones do not cover.

Next

Where to go next

The design behind it: Test architecture for a data platform.

The static checks: Shift left: fail fast on data code. The simulated company: DataTwin, explained.

The faults to test for: the Problem Lab, nineteen data problems run for real.