Data PlatformZoom level 2
The daily end-to-end test: every layer against the truth, day by day
This is the test to run before every change is trusted. It plays the simulated company for three days, sends each day through ingestion, silver and gold, and then asks one question per layer: did it equal the true answer on every day? A failure names the layer and the day.
What it does
- Run the loop. A fixture starts
scripts/run_platform.pywith DuckDB, SQLite sources, the in-memory metadata store and the compressed calendar, which puts every planted fault in days 2 and 3. It takes about two minutes on a laptop. - Read the report. The run writes
report.json: per day, each layer's measures next to the oracle's, every incident's verdict, and every run's status. - Judge it with tables. One table says which part of the report holds each layer. One says which layers' incidents must all be handled. One says which runs must have succeeded.
- Name the failure. A failing test says which layer and which day, and lists each measure that differs.
tests/dataplatform/test_end_to_end.py:62The layers and where their verdicts are in the report.LAYERS = [
("bronze", "per_day", bronze_differences),
("silver", "silver", lambda d: differences(d["measures"], "silver")),
("gold", "gold", lambda d: differences(d["measures"], "gold")),
]tests/dataplatform/test_end_to_end.py:69The check that answers the question: a layer's measures equal the oracle's on every day.@pytest.mark.parametrize(("layer", "key", "mismatches"), LAYERS, ids=[x[0] for x in LAYERS])
def test_the_layer_equals_the_oracle_every_day(report, layer, key, mismatches):
days = report.get(key) or []
assert days, f"the report has no {layer} days: that layer did not run"
bad = {f"day {d['day']}": m for d in days if (m := mismatches(d))}
assert bad == {}, f"{layer} differs from the oracle"tests/dataplatform/test_end_to_end.py:88A green run that planted nothing proves nothing, so the test counts the kinds of fault judged.def test_incidents_were_actually_planted(report):
"""A green run that planted nothing proves nothing: every kind of fault must be judged in the run."""
judged = {v["incident"] for key in ("per_day", "silver", "gold") for d in report.get(key) or []
for v in d.get("incidents", [])}
assert len(judged) >= 11, f"only {len(judged)} kinds of incident were judged: {sorted(judged)}"What it checks
| Test | What it asserts |
|---|---|
| The layer equals the oracle every day (bronze, silver, gold) | Every measure of every dataset or table equals the oracle's value on every day |
| Every incident the layer judges is handled | No incident's verdict is missed at ingestion, silver or gold |
| Incidents were actually planted | At least eleven kinds of fault were judged in the run |
| Every run succeeded | Ingestion, silver and gold runs ended succeeded, or partial where a batch was held on purpose |
| The metrics were asked of built gold | The known questions were asked and answered by the gold tables the run built |
| The known answers match the specification (full scale only) | Skipped at three days and scale 0.1: the specification's values hold at scale 1.0 after 14 days |
Run it on another run's report
The test does not need to start its own run. Set DP_E2E_REPORT to the path of a report.json from any run, and the same tables judge that run. The 14-day proof on Databricks writes the same report, so one set of checks judges both the laptop loop and the cluster. The full-scale known-answer check runs only when the report is for 14 days at scale 1.0.
A recorded result
On 19 November 2026 the loop ran for three days at scale 0.1 on a laptop in 119 seconds. Bronze, silver and gold equalled the oracle on all three days, 11 of 11 planted faults were handled at their layer, and the test passed (11 passed, 1 skipped). Silver took 87 seconds of the 119 and gold 19. These numbers come from that one run.
- Below scale 0.1 the compressed calendar cannot find enough cancelled order lines to delete, and the world stops with an error. The test uses 0.1.
- Three days is the daily loop. Gold has not been judged on all fourteen days by a test in this repository, only by the run script. Point the test at the Databricks report to judge fourteen.
- On Spark the same test would take longer. It uses DuckDB so that it runs without a cluster.