Data PlatformZoom level 1
Tests and proof: how we know every layer is right
| Part | Starting point | Called by (found by reading the code) |
|---|---|---|
| The whole platform and the simulated company, day by day | main scripts/run_platform.py:315 | module level · scripts/run_platform.py:745 |
A data platform can run without an error and still be wrong. A row can be counted twice. A delete can be missed. So the platform is proven in three different ways, and each catches what the others cannot.
The three kinds of proof
| Kind | What it is | What it proves | What it cannot prove |
|---|---|---|---|
| 1. Table-driven unit and contract tests | One test function and a table of cases. Each case is a row: a name, an input and the expected output. A contract table is the same idea for a port: every adapter of the port must pass the same rows. | That one function, or one adapter, does what each row says. | That the parts add up to a correct table at the end. |
| 2. The simulated company with an oracle | A generated tyre company runs for simulated days and writes into mock source systems. The generator also records the true answer for every day. That record is the oracle, and it is the golden data. | That each layer's output equals the truth on every simulated day. | Anything the simulated company does not do. |
| 3. Planted incidents | Eleven faults put into the simulated company's data on purpose, on known days: a late file, a file sent twice, a renamed column, deleted rows and more. | That the platform catches or handles each fault at the right layer, and raises nothing on a clean day. | Faults nobody thought to plant. |
An oracle in testing is a source of true answers that does not depend on the thing being tested. Here it is computed from the simulated company's own record of what it did, never from the platform's logs. Golden data means data whose correct result is known in advance.
The direct answer: is every layer tested against golden data?
Not every layer, and not all in the same way. The table is the short version. Proof by layer has the test files, the counts and what is missing.
| Layer | Compared with the oracle? | How |
|---|---|---|
| Metadata store | No | It holds settings and logs, not business data. It is proven by a contract table run against two adapters. |
| Ingestion and bronze | Yes, by row count | Rows appended to bronze per dataset per day equal the oracle's count, in a slow test and in the recorded runs. |
| Bronze to silver | Yes | Current rows, versions, deletes marked, each dataset's reconciliation measures and readings per hour, in a slow test that needs Spark and in the recorded runs. |
| Silver to gold | Yes in the recorded runs, partly in tests | The run script compares gold with the oracle every day. One slow test runs gold on the simulated company and checks the days its scenarios name. There is no test that asserts gold equals the oracle on all fourteen days. |
| Semantic | Yes, five known answers | Five certified metrics each carry one known answer. The oracle reproduces all five. Asking the same five of built gold is a separate command, not part of the default test run. |
| Operations | No | Alerts and screens are checked against the incident calendar on a stand-in run log: one alert per incident, none on a clean day. |
The golden-data tests for silver and gold need Spark, and the one for bronze needs the Delta package. A test run without them skips those tests. A check of the public snapshot on 19 November 2026 ran the two main test folders without Spark: 1316 passed and 219 skipped, and the skipped ones need Spark. So a green run without Spark proves the first kind of proof and not the second.
The pages under this one
| Page | What it explains |
|---|---|
| The simulated company | The simulated tyre company: its sources, its clock, its volumes and how it is run. |
| The oracle | What truth the oracle records for each day and layer, and how each layer is compared with it. |
| Planted incidents | Every planted incident: its day, the rule that detects it, the layer that handles it, and the recorded result. |
| Tables of cases | How a case table looks, the contract tables every adapter must pass, the history contract, and the same tests on two engines. |
| Proof by layer | One table with a row per layer, and the check of how many catalog obligations a test cites. |
The design decisions behind the proof
| Decision | Why |
|---|---|
| The truth never comes from the platform | A platform checked against its own logs agrees with itself even when it is wrong. |
| The simulated company is deterministic | The same seed gives the same data on any machine, so a failure can be repeated. |
| Incidents live in a file, not in code | Anyone can move an incident to another day or change its size by editing one line. |
| A check on a layer that does not exist is pending, never passing | Nothing counts as proven by default. |
| Tests are tables | A new case is a new row. The reader sees every case at a glance. |
- No test asserts that gold equals the oracle on every one of the fourteen days. The run script does that comparison, and its result is recorded in a design note.
- The fourteen-day run at full volume through gold has not been completed. The recorded attempt ran out of memory on day 12 on a laptop.
- The golden-data tests are slow and need Spark, so a run without Spark does not include them.
- The declarative silver engine has not been run against the oracle in any test in this repository.
- Operations has no oracle of its own. Its checks use a stand-in run log, not the one the engine writes.