cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Tests and proof › Tables of cases

Data PlatformZoom level 2

Table-driven tests and port contract tables

Almost every test in this platform is a table. One test function holds the logic of the check. A table beside it holds the cases, one per row: a name, an input and the expected result.

The reason is readability. A reader sees every case a function is proven on at a glance, and adding a case means adding one row. A failing case is reported by its name.

What a case table looks like

This is the table for the hygiene stage of silver. Each row has a name, the raw text, the text expected after the stage, and any settings for that case.

tests/dataplatform/test_silver_stages.py:101Nine cases of one stage, then the function that runs them.
# ---------------------------------------------------------------------------------------------------- 3 hygiene
@pytest.mark.parametrize(
    ("case", "raw", "want", "kw"),
    [
        ("trimmed", "  a b  ", "a b", {}),
        ("runs of spaces collapsed", "a   b", "a b", {}),
        ("empty becomes null", "   ", None, {}),
        ("control characters removed", "a\x07b", "ab", {}),
        ("byte-order mark removed", "ab", "ab", {}),
        ("line endings made \\n", "a\r\nb", "a\nb", {}),
        ("null stays null", None, None, {}),
        ("a column may opt out", "  a  ", "  a  ", {"column_settings": [{"name": "s", "hygiene": False}]}),
        ("the stage switched off leaves values", "  a  ", "  a  ", {"settings": {"processor_groups.hygiene": "disabled"}}),
    ],
)
def test_hygiene(spark, case, raw, want, kw):

The names are sentences. When the row named empty becomes null fails, the report says exactly that, and nobody has to read the test's code to learn what broke.

A case table where the input is a timeline

A table row can hold more than two values. The retention rules are tested with rows of a delivery, whether the lander took it, and the action expected.

tests/dataplatform/test_retention.py:33The rows, and the three-line function that runs them.
ACTION_CASES = [
    ("young and hot: keep", ("2026-12-20", "a.csv", "hot"), True, "keep"),
    ("older than hot: archive", ("2026-11-20", "a.csv", "hot"), True, "archive"),
    ("archived and young enough: keep", ("2026-11-01", "a.csv", "archive"), True, "keep"),
    ("past hot + archive: delete", ("2026-09-01", "a.csv", "archive"), True, "delete"),
    ("hot but past both limits: delete at once", ("2026-09-01", "a.csv", "hot"), True, "delete"),
    ("never taken by the lander: always kept", ("2026-01-01", "a.csv", "hot"), False, "keep"),
    ("a month window counts from its first day", ("2026-11", "m.csv", "hot"), True, "archive"),
]


@pytest.mark.parametrize(("case", "delivery", "taken", "want"), ACTION_CASES, ids=[c[0] for c in ACTION_CASES])
def test_landing_actions(case: str, delivery: tuple[str, str, str], taken: bool, want: str) -> None:

Contract tables for ports

One table of casesname, input,expected outputAdapter in memoryAdapter on DeltaAdapter on SparkSame rows,same expected resultsA contract table: every adapter of a port passes the same cases.
The table on the left is written once. Each adapter in the middle must give the results on the right.

A port is an interface the platform's core calls, such as the bronze writer. An adapter is one implementation of it for one product. The core is only safe to move between products if every adapter behaves the same. A contract table states that behaviour once, and the test runs every row against every adapter.

tests/dataplatform/test_ingest_ports.py:73The bronze writer's contract, and the function that runs each row against each adapter.
BRONZE_CASES = [
    ("a write returns its rows", write("k1", "AB"), 2),
    ("a table never written is empty", count(), 0),
    ("rows are kept", seq(write("k1", "AB"), count()), 2),
    ("the same batch twice changes nothing (DUP-01)", seq(write("k1", "AB"), write("k1", "AB"), count()), 2),
    ("a rewrite replaces only its batch", seq(write("k1", "AB"), write("k2", "C"), write("k1", "A"), count()), 2),
    ("batches side by side", seq(write("k1", "AB"), write("k2", "CDE"), count({"_batch_key": "k2"})), 3),
    ("filter on the run", seq(write("k1", "AB", "r1"), write("k2", "C", "r2"), count({"_run_id": "r2"})), 1),
    ("sums", seq(write("k1", "ABC"), lambda w: w.fingerprint("src.ds", None, ("qty",))["qty"]), 6),
    (
        "keys of one batch",
        seq(write("k1", "AB"), write("k2", "C"), lambda w: w.keys("src.ds", ("id",), {"_batch_key": "k1"})),
        {("A",), ("B",)},
    ),
    ("keys of a table never written", lambda w: w.keys("src.ds", ("id",), {"_batch_key": "k1"}), set()),
    ("an unsafe batch key is refused", write("k1' OR '1'='1", "A"), ValueError),
]


@pytest.mark.parametrize("adapter", list(WRITERS))
@pytest.mark.parametrize(("case", "act", "expect"), BRONZE_CASES, ids=[c[0] for c in BRONZE_CASES])
def test_bronze_writer_contract(tmp_path, adapter, case, act, expect):
    w = WRITERS[adapter](tmp_path)
    if isinstance(expect, type) and issubclass(expect, Exception):
        with pytest.raises(expect):
            act(w)
    else:
        assert act(w) == expect

The two decorators above the function are the whole mechanism. One lists the adapters and the other the cases, and the test framework runs every combination. An adapter that is not available on the machine, such as PostgreSQL with no local server, is skipped with the reason.

PortTest fileAdapters the table runs against
Metadata storetest_meta_store.py (4)In memory, PostgreSQL
Source databasetest_source_ports.py (9)In memory, SQLite, PostgreSQL
File dropthe same fileIn memory, local folder, object storage against a local stand-in
Event streamthe same fileIn memory, local files, a cloud stream against a local stand-in
Bronze writertest_ingest_ports.py (6)In memory, Delta through a native library, Delta through Spark
Landingthe same fileLocal files, object storage against a local stand-in
Silver tables and the stage runnertest_silver_ports.py (13)Delta on Spark
Gold tables and the gold readertest_gold_ports.py (2)Delta folders
Table storetest_table_store.py (2)Delta, Iceberg
Run log readertest_obs_runlog.py (3)In memory, PostgreSQL, and a customer adapter that is skipped
Notifiertest_obs_notify.py (3)In memory, workspace destinations with an injected transport
tests/dataplatform/test_source_ports.py:125The source database contract: changes applied, the live rows expected, and the feed expected.
DB_CASES = {
    "insert": ([ch("I", row())], [row()], [("I", 1)]),
    "newer change wins": (
        [ch("I", row()), ch("U", row(promised="2026-11-13", seq=2), 2)],
        [row(promised="2026-11-13", seq=2)],
        [("I", 1), ("U", 2)],
    ),
    "older change arriving late does not overwrite": (
        [ch("I", row()), ch("U", row(promised="2026-11-14", seq=3), 2), ch("U", row(promised="2026-11-13", seq=2), 3)],
        [row(promised="2026-11-14", seq=3)],
        [("I", 1), ("U", 3), ("U", 2)],
    ),
    "hard delete leaves the feed a D": ([ch("I", row()), ch("D", row(seq=2), 2)], [], [("I", 1), ("D", 2)]),
    "silent loss: in the table, not in the feed": (
        [ch("I", row()), ch("I", row(line=20, seq=2), 2, in_feed=False)],
        [row(), row(line=20, seq=2)],
        [("I", 1)],
    ),
}

A worked example: one row of that contract

Take the third case, an older change arriving late. Order SO1, line 10, is inserted with a promised date of 12 November. Two updates follow. The table shows the three changes in the order they are applied, and what every adapter must then hold.

AppliedOperationPromised dateChange sequence
firstinsert2026-11-121
secondupdate2026-11-143
thirdupdate2026-11-132
What is checkedExpected
The live rowPromised date 2026-11-14 at change sequence 3. The older change did not overwrite the newer.
The change feed, in arrival orderInsert 1, update 3, update 2. The feed keeps arrival order, late change included.

The last case is also worth reading. It says a row can be in the table and not in the change feed. That is the silent loss the simulated company plants, and the contract makes sure every source adapter can express it.

The history contract

Silver can run on two engines, the job engine and the declarative engine. They store history differently. Type 2 history keeps one row per version of a key. To make sure both engines mean the same thing, the cases are held in a file of pure data that names no engine.

src/dataplatform/silver/history_contract.py:20The eight cases every bronze-to-silver engine must pass.
CASES: list[tuple[str, list[list[Change]], list[Version]]] = [
    ("first load: one current version per key",
     [[("a", 1, "x", 1, "I", 0, 1), ("b", 5, "x", 2, "I", 0, 2)]],
     [("a", 1, 1, True, False), ("b", 5, 2, True, False)]),
    ("a tracked change opens a version",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 2, "x", 3, "U", 5, 2)]],
     [("a", 1, 1, False, False), ("a", 2, 3, True, False)]),
    ("an untracked change updates the version in place",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "y", 3, "U", 5, 2)]],
     [("a", 1, 1, True, False)]),
    ("several changes in one batch become several versions, in order",
     [[("a", 1, "x", 1, "I", 0, 1), ("a", 3, "x", 5, "U", 9, 3), ("a", 2, "x", 3, "U", 5, 2)]],
     [("a", 1, 1, False, False), ("a", 2, 3, False, False), ("a", 3, 5, True, False)]),
    ("an older change arriving after a newer one goes into history, the newer stays current (SCD-02)",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 2, "x", 4, "U", 65, 3)]],
     [("a", 1, 1, False, False), ("a", 2, 4, False, False), ("a", 3, 5, True, False)]),
    ("a late change equal to the version after it moves that version's start",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 3, "x", 4, "U", 4, 3)]],
     [("a", 1, 1, False, False), ("a", 3, 4, True, False)]),
    ("a delete marks the key deleted, nothing removed (DEL-01)",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "x", 2, "D", 5, 2)]],
     [("a", 1, 1, True, True)]),
    ("the same batch twice changes nothing",
     [[("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)],
      [("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)]],
     [("a", 1, 1, False, False), ("a", 2, 2, True, False)]),
]

Read one case. The fifth gives three batches for key a: an insert at sequence 1, an update to value 3 at sequence 5, and then an update to value 2 at sequence 4 that arrives last. The expected result is three versions in sequence order, with value 3 current. The late change went into history and did not become current.

Field of a change rowMeaning
keyThe business key.
valueA tracked column. A change to it opens a new version.
noteAn untracked column. A change to it updates the version in place.
sequenceThe source's order of changes.
opI for insert, U for update, D for delete.
committed minute, feed positionWhen the source committed the change, and its place in the feed.
tests/dataplatform/test_silver_stages.py:369The job engine run against the contract.
@pytest.mark.parametrize(("case", "batches", "want"), HISTORY_CONTRACT, ids=[c[0] for c in HISTORY_CONTRACT])
def test_history_type2(spark, case, batches, want):
    """The SCD Type 2 contract both silver engines pass (ADR-035; silver/history_contract.py)."""
    silver = _apply(spark, plan(**SCD2), _feed(batches))
    assert rows(silver, "id", "v", "_seq_from", "_is_current", "_is_deleted") == want, case
    assert_history_invariants(silver)

The engines also differ in how they store a deleted key. The job engine keeps the last version current and flags it deleted. A managed change-capture engine closes the version. So each engine's output is first brought to one common shape, and then compared.

src/dataplatform/silver/history_contract.py:53Bringing an engine that closes deleted versions to the contract's shape.
def normalise_closed(rows: Iterable[Mapping[str, Any]], key: str, value: str, start: str, end: str) -> list[Version]:
    """An engine that closes a deleted key's last version (end set, nothing after it): the last version of a key is
    'current'; it is 'deleted' when it is closed."""
    by_key: dict[Any, list[Mapping[str, Any]]] = {}
    for r in rows:
        by_key.setdefault(r[key], []).append(r)
    out: list[Version] = []
    for k in sorted(by_key):
        vs = sorted(by_key[k], key=lambda r: int(r[start]))
        for i, r in enumerate(vs):
            last = i == len(vs) - 1
            out.append((str(k), int(r[value]), int(r[start]), last, last and r[end] is not None))
    return out

The same tests on two engines

The silver stages and the gold builds are written against one dataframe interface. They run on Spark, and they also run on DuckDB through a library that offers the same interface. Conformance means the same test tables pass on both.

tests/dataplatform/conftest.py:20The engine is chosen once per test process, before any test module is imported.
def pytest_addoption(parser: pytest.Parser) -> None:
    parser.addoption("--engine", choices=ENGINES, default=os.environ.get("DP_ENGINE", "spark"),
                     help="the frame engine for the stage and gold-build tests (docs/DUCKDB-SPIKE.md)")


def pytest_configure(config: pytest.Config) -> None:
    if config.getoption("--engine") == "duckdb":
        from dataplatform.adapters.duckdb_frames import activate

        activate()  # before collection: every `from pyspark.sql import ...` below now gets SQLFrame on DuckDB
tests/dataplatform/conftest.py:37The fixture keeps the name spark. The tests only see a session and its frames.
@pytest.fixture(scope="session")
def spark(engine_name: str, tmp_path_factory: pytest.TempPathFactory):
    if engine_name == "duckdb":
        from dataplatform.adapters.duckdb_frames import local_session as duck_session

        session = duck_session()
        yield session
        session.stop()
        return
    pytest.importorskip("pyspark")
    pytest.importorskip("delta")
    from dataplatform.adapters.spark_silver import local_session
    from dataplatform.adapters.table_store import iceberg_jar

    # an Iceberg folder catalog "ice" beside Delta when the runtime jar is on this machine (ADR-027 contract tests)
    warehouse = tmp_path_factory.mktemp("iceberg") if iceberg_jar() else None
    session = local_session("dataplatform-tests", iceberg_warehouse=warehouse)
    yield session
    session.stop()

The option is --engine spark, which is the default, or --engine duckdb. The engine cannot be chosen per test, because the DuckDB library takes the place of the Spark imports for the whole process. So conformance is two test processes over the same files.

Makefile:191The four test files that make up conformance, and the two runs.
CONFORMANCE = tests/dataplatform/test_silver_stages.py tests/dataplatform/test_gold_engine.py \
	tests/dataplatform/test_baseline_stage.py tests/dataplatform/test_privacy_stage.py
engine-conformance:
	PYTHONPATH=src nice $(RUN_PY) --with pytest pytest -q -p no:cacheprovider -o addopts="" $(CONFORMANCE) --engine spark
	PYTHONPATH=src nice $(WORLD_PY) $(DUCK_DEPS) --with pytest pytest -q -p no:cacheprovider -o addopts="" $(CONFORMANCE) --engine duckdb

The recorded result is in the DuckDB design note. On 5 November 2026, 143 stage and gold-build cases passed on Spark and the same 143 passed on DuckDB. Spark took 74 seconds and DuckDB 133 seconds. Getting there took 13 rewrites into forms both engines read the same way, and the note lists each. Four cases pin known differences between the engines, such as how a record hash treats empty fields, which is why one table is always written by one engine.

How the tests themselves are kept honest

RuleWhere it shows
A check with nothing to compare is never a passThe reconciliation function fails a hop with no value.
Pending is said, not hiddenAn adapter or layer that does not exist yet is a skipped row with its reason.
The core names no productSeveral test files scan the core's source for product names and fail on any.
Generated files are checked for driftA test regenerates the dashboard and view files and compares them.
Not built yet, or known gaps
  • The history contract runs against the job engine in the test suite. The declarative engine's run of the contract is in the spikes folder and needs a workspace. It is not part of the tests.
  • The gold tables contract has one adapter, so it proves that adapter and not yet that two agree.
  • The silver tables contract runs on Spark. The DuckDB silver adapter is exercised by the end to end run, not by this contract table.
  • Conformance covers four test files. The day-by-day golden tests run on Spark only in the test suite.
  • The design note says continuous integration runs the DuckDB conformance step. The public snapshot contains no workflow folder, so I could not verify that from the repository. The Makefile target exists.