Data PlatformZoom level 2
Table-driven tests and port contract tables
Almost every test in this platform is a table. One test function holds the logic of the check. A table beside it holds the cases, one per row: a name, an input and the expected result.
The reason is readability. A reader sees every case a function is proven on at a glance, and adding a case means adding one row. A failing case is reported by its name.
What a case table looks like
This is the table for the hygiene stage of silver. Each row has a name, the raw text, the text expected after the stage, and any settings for that case.
tests/dataplatform/test_silver_stages.py:101Nine cases of one stage, then the function that runs them.# ---------------------------------------------------------------------------------------------------- 3 hygiene
@pytest.mark.parametrize(
("case", "raw", "want", "kw"),
[
("trimmed", " a b ", "a b", {}),
("runs of spaces collapsed", "a b", "a b", {}),
("empty becomes null", " ", None, {}),
("control characters removed", "a\x07b", "ab", {}),
("byte-order mark removed", "ab", "ab", {}),
("line endings made \\n", "a\r\nb", "a\nb", {}),
("null stays null", None, None, {}),
("a column may opt out", " a ", " a ", {"column_settings": [{"name": "s", "hygiene": False}]}),
("the stage switched off leaves values", " a ", " a ", {"settings": {"processor_groups.hygiene": "disabled"}}),
],
)
def test_hygiene(spark, case, raw, want, kw):The names are sentences. When the row named empty becomes null fails, the report says exactly that, and nobody has to read the test's code to learn what broke.
A case table where the input is a timeline
A table row can hold more than two values. The retention rules are tested with rows of a delivery, whether the lander took it, and the action expected.
tests/dataplatform/test_retention.py:33The rows, and the three-line function that runs them.ACTION_CASES = [
("young and hot: keep", ("2026-12-20", "a.csv", "hot"), True, "keep"),
("older than hot: archive", ("2026-11-20", "a.csv", "hot"), True, "archive"),
("archived and young enough: keep", ("2026-11-01", "a.csv", "archive"), True, "keep"),
("past hot + archive: delete", ("2026-09-01", "a.csv", "archive"), True, "delete"),
("hot but past both limits: delete at once", ("2026-09-01", "a.csv", "hot"), True, "delete"),
("never taken by the lander: always kept", ("2026-01-01", "a.csv", "hot"), False, "keep"),
("a month window counts from its first day", ("2026-11", "m.csv", "hot"), True, "archive"),
]
@pytest.mark.parametrize(("case", "delivery", "taken", "want"), ACTION_CASES, ids=[c[0] for c in ACTION_CASES])
def test_landing_actions(case: str, delivery: tuple[str, str, str], taken: bool, want: str) -> None:Contract tables for ports
A port is an interface the platform's core calls, such as the bronze writer. An adapter is one implementation of it for one product. The core is only safe to move between products if every adapter behaves the same. A contract table states that behaviour once, and the test runs every row against every adapter.
tests/dataplatform/test_ingest_ports.py:73The bronze writer's contract, and the function that runs each row against each adapter.BRONZE_CASES = [
("a write returns its rows", write("k1", "AB"), 2),
("a table never written is empty", count(), 0),
("rows are kept", seq(write("k1", "AB"), count()), 2),
("the same batch twice changes nothing (DUP-01)", seq(write("k1", "AB"), write("k1", "AB"), count()), 2),
("a rewrite replaces only its batch", seq(write("k1", "AB"), write("k2", "C"), write("k1", "A"), count()), 2),
("batches side by side", seq(write("k1", "AB"), write("k2", "CDE"), count({"_batch_key": "k2"})), 3),
("filter on the run", seq(write("k1", "AB", "r1"), write("k2", "C", "r2"), count({"_run_id": "r2"})), 1),
("sums", seq(write("k1", "ABC"), lambda w: w.fingerprint("src.ds", None, ("qty",))["qty"]), 6),
(
"keys of one batch",
seq(write("k1", "AB"), write("k2", "C"), lambda w: w.keys("src.ds", ("id",), {"_batch_key": "k1"})),
{("A",), ("B",)},
),
("keys of a table never written", lambda w: w.keys("src.ds", ("id",), {"_batch_key": "k1"}), set()),
("an unsafe batch key is refused", write("k1' OR '1'='1", "A"), ValueError),
]
@pytest.mark.parametrize("adapter", list(WRITERS))
@pytest.mark.parametrize(("case", "act", "expect"), BRONZE_CASES, ids=[c[0] for c in BRONZE_CASES])
def test_bronze_writer_contract(tmp_path, adapter, case, act, expect):
w = WRITERS[adapter](tmp_path)
if isinstance(expect, type) and issubclass(expect, Exception):
with pytest.raises(expect):
act(w)
else:
assert act(w) == expectThe two decorators above the function are the whole mechanism. One lists the adapters and the other the cases, and the test framework runs every combination. An adapter that is not available on the machine, such as PostgreSQL with no local server, is skipped with the reason.
| Port | Test file | Adapters the table runs against |
|---|---|---|
| Metadata store | test_meta_store.py (4) | In memory, PostgreSQL |
| Source database | test_source_ports.py (9) | In memory, SQLite, PostgreSQL |
| File drop | the same file | In memory, local folder, object storage against a local stand-in |
| Event stream | the same file | In memory, local files, a cloud stream against a local stand-in |
| Bronze writer | test_ingest_ports.py (6) | In memory, Delta through a native library, Delta through Spark |
| Landing | the same file | Local files, object storage against a local stand-in |
| Silver tables and the stage runner | test_silver_ports.py (13) | Delta on Spark |
| Gold tables and the gold reader | test_gold_ports.py (2) | Delta folders |
| Table store | test_table_store.py (2) | Delta, Iceberg |
| Run log reader | test_obs_runlog.py (3) | In memory, PostgreSQL, and a customer adapter that is skipped |
| Notifier | test_obs_notify.py (3) | In memory, workspace destinations with an injected transport |
tests/dataplatform/test_source_ports.py:125The source database contract: changes applied, the live rows expected, and the feed expected.DB_CASES = {
"insert": ([ch("I", row())], [row()], [("I", 1)]),
"newer change wins": (
[ch("I", row()), ch("U", row(promised="2026-11-13", seq=2), 2)],
[row(promised="2026-11-13", seq=2)],
[("I", 1), ("U", 2)],
),
"older change arriving late does not overwrite": (
[ch("I", row()), ch("U", row(promised="2026-11-14", seq=3), 2), ch("U", row(promised="2026-11-13", seq=2), 3)],
[row(promised="2026-11-14", seq=3)],
[("I", 1), ("U", 3), ("U", 2)],
),
"hard delete leaves the feed a D": ([ch("I", row()), ch("D", row(seq=2), 2)], [], [("I", 1), ("D", 2)]),
"silent loss: in the table, not in the feed": (
[ch("I", row()), ch("I", row(line=20, seq=2), 2, in_feed=False)],
[row(), row(line=20, seq=2)],
[("I", 1)],
),
}A worked example: one row of that contract
Take the third case, an older change arriving late. Order SO1, line 10, is inserted with a promised date of 12 November. Two updates follow. The table shows the three changes in the order they are applied, and what every adapter must then hold.
| Applied | Operation | Promised date | Change sequence |
|---|---|---|---|
| first | insert | 2026-11-12 | 1 |
| second | update | 2026-11-14 | 3 |
| third | update | 2026-11-13 | 2 |
| What is checked | Expected |
|---|---|
| The live row | Promised date 2026-11-14 at change sequence 3. The older change did not overwrite the newer. |
| The change feed, in arrival order | Insert 1, update 3, update 2. The feed keeps arrival order, late change included. |
The last case is also worth reading. It says a row can be in the table and not in the change feed. That is the silent loss the simulated company plants, and the contract makes sure every source adapter can express it.
The history contract
Silver can run on two engines, the job engine and the declarative engine. They store history differently. Type 2 history keeps one row per version of a key. To make sure both engines mean the same thing, the cases are held in a file of pure data that names no engine.
src/dataplatform/silver/history_contract.py:20The eight cases every bronze-to-silver engine must pass.CASES: list[tuple[str, list[list[Change]], list[Version]]] = [
("first load: one current version per key",
[[("a", 1, "x", 1, "I", 0, 1), ("b", 5, "x", 2, "I", 0, 2)]],
[("a", 1, 1, True, False), ("b", 5, 2, True, False)]),
("a tracked change opens a version",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 2, "x", 3, "U", 5, 2)]],
[("a", 1, 1, False, False), ("a", 2, 3, True, False)]),
("an untracked change updates the version in place",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "y", 3, "U", 5, 2)]],
[("a", 1, 1, True, False)]),
("several changes in one batch become several versions, in order",
[[("a", 1, "x", 1, "I", 0, 1), ("a", 3, "x", 5, "U", 9, 3), ("a", 2, "x", 3, "U", 5, 2)]],
[("a", 1, 1, False, False), ("a", 2, 3, False, False), ("a", 3, 5, True, False)]),
("an older change arriving after a newer one goes into history, the newer stays current (SCD-02)",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 2, "x", 4, "U", 65, 3)]],
[("a", 1, 1, False, False), ("a", 2, 4, False, False), ("a", 3, 5, True, False)]),
("a late change equal to the version after it moves that version's start",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 3, "x", 4, "U", 4, 3)]],
[("a", 1, 1, False, False), ("a", 3, 4, True, False)]),
("a delete marks the key deleted, nothing removed (DEL-01)",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "x", 2, "D", 5, 2)]],
[("a", 1, 1, True, True)]),
("the same batch twice changes nothing",
[[("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)],
[("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)]],
[("a", 1, 1, False, False), ("a", 2, 2, True, False)]),
]Read one case. The fifth gives three batches for key a: an insert at sequence 1, an update to value 3 at sequence 5, and then an update to value 2 at sequence 4 that arrives last. The expected result is three versions in sequence order, with value 3 current. The late change went into history and did not become current.
| Field of a change row | Meaning |
|---|---|
| key | The business key. |
| value | A tracked column. A change to it opens a new version. |
| note | An untracked column. A change to it updates the version in place. |
| sequence | The source's order of changes. |
| op | I for insert, U for update, D for delete. |
| committed minute, feed position | When the source committed the change, and its place in the feed. |
tests/dataplatform/test_silver_stages.py:369The job engine run against the contract.@pytest.mark.parametrize(("case", "batches", "want"), HISTORY_CONTRACT, ids=[c[0] for c in HISTORY_CONTRACT])
def test_history_type2(spark, case, batches, want):
"""The SCD Type 2 contract both silver engines pass (ADR-035; silver/history_contract.py)."""
silver = _apply(spark, plan(**SCD2), _feed(batches))
assert rows(silver, "id", "v", "_seq_from", "_is_current", "_is_deleted") == want, case
assert_history_invariants(silver)The engines also differ in how they store a deleted key. The job engine keeps the last version current and flags it deleted. A managed change-capture engine closes the version. So each engine's output is first brought to one common shape, and then compared.
src/dataplatform/silver/history_contract.py:53Bringing an engine that closes deleted versions to the contract's shape.def normalise_closed(rows: Iterable[Mapping[str, Any]], key: str, value: str, start: str, end: str) -> list[Version]:
"""An engine that closes a deleted key's last version (end set, nothing after it): the last version of a key is
'current'; it is 'deleted' when it is closed."""
by_key: dict[Any, list[Mapping[str, Any]]] = {}
for r in rows:
by_key.setdefault(r[key], []).append(r)
out: list[Version] = []
for k in sorted(by_key):
vs = sorted(by_key[k], key=lambda r: int(r[start]))
for i, r in enumerate(vs):
last = i == len(vs) - 1
out.append((str(k), int(r[value]), int(r[start]), last, last and r[end] is not None))
return outThe same tests on two engines
The silver stages and the gold builds are written against one dataframe interface. They run on Spark, and they also run on DuckDB through a library that offers the same interface. Conformance means the same test tables pass on both.
tests/dataplatform/conftest.py:20The engine is chosen once per test process, before any test module is imported.def pytest_addoption(parser: pytest.Parser) -> None:
parser.addoption("--engine", choices=ENGINES, default=os.environ.get("DP_ENGINE", "spark"),
help="the frame engine for the stage and gold-build tests (docs/DUCKDB-SPIKE.md)")
def pytest_configure(config: pytest.Config) -> None:
if config.getoption("--engine") == "duckdb":
from dataplatform.adapters.duckdb_frames import activate
activate() # before collection: every `from pyspark.sql import ...` below now gets SQLFrame on DuckDBtests/dataplatform/conftest.py:37The fixture keeps the name spark. The tests only see a session and its frames.@pytest.fixture(scope="session")
def spark(engine_name: str, tmp_path_factory: pytest.TempPathFactory):
if engine_name == "duckdb":
from dataplatform.adapters.duckdb_frames import local_session as duck_session
session = duck_session()
yield session
session.stop()
return
pytest.importorskip("pyspark")
pytest.importorskip("delta")
from dataplatform.adapters.spark_silver import local_session
from dataplatform.adapters.table_store import iceberg_jar
# an Iceberg folder catalog "ice" beside Delta when the runtime jar is on this machine (ADR-027 contract tests)
warehouse = tmp_path_factory.mktemp("iceberg") if iceberg_jar() else None
session = local_session("dataplatform-tests", iceberg_warehouse=warehouse)
yield session
session.stop()The option is --engine spark, which is the default, or --engine duckdb. The engine cannot be chosen per test, because the DuckDB library takes the place of the Spark imports for the whole process. So conformance is two test processes over the same files.
Makefile:191The four test files that make up conformance, and the two runs.CONFORMANCE = tests/dataplatform/test_silver_stages.py tests/dataplatform/test_gold_engine.py \ tests/dataplatform/test_baseline_stage.py tests/dataplatform/test_privacy_stage.py engine-conformance: PYTHONPATH=src nice $(RUN_PY) --with pytest pytest -q -p no:cacheprovider -o addopts="" $(CONFORMANCE) --engine spark PYTHONPATH=src nice $(WORLD_PY) $(DUCK_DEPS) --with pytest pytest -q -p no:cacheprovider -o addopts="" $(CONFORMANCE) --engine duckdb
The recorded result is in the DuckDB design note. On 5 November 2026, 143 stage and gold-build cases passed on Spark and the same 143 passed on DuckDB. Spark took 74 seconds and DuckDB 133 seconds. Getting there took 13 rewrites into forms both engines read the same way, and the note lists each. Four cases pin known differences between the engines, such as how a record hash treats empty fields, which is why one table is always written by one engine.
How the tests themselves are kept honest
| Rule | Where it shows |
|---|---|
| A check with nothing to compare is never a pass | The reconciliation function fails a hop with no value. |
| Pending is said, not hidden | An adapter or layer that does not exist yet is a skipped row with its reason. |
| The core names no product | Several test files scan the core's source for product names and fail on any. |
| Generated files are checked for drift | A test regenerates the dashboard and view files and compares them. |
- The history contract runs against the job engine in the test suite. The declarative engine's run of the contract is in the spikes folder and needs a workspace. It is not part of the tests.
- The gold tables contract has one adapter, so it proves that adapter and not yet that two agree.
- The silver tables contract runs on Spark. The DuckDB silver adapter is exercised by the end to end run, not by this contract table.
- Conformance covers four test files. The day-by-day golden tests run on Spark only in the test suite.
- The design note says continuous integration runs the DuckDB conformance step. The public snapshot contains no workflow folder, so I could not verify that from the repository. The Makefile target exists.