Data PlatformZoom level 2
Two engines for silver: the job engine and the declarative engine
Bronze to silver can be run by two engines. The job engine is our own runner: it calls the thirteen stages and writes with one merge. The declarative engine runs the stages inside a declarative pipeline and lets the pipeline's change capture keep the history. A dataset picks one with a single setting.
Two engines are only safe if two things hold. A dataset must never be run by an engine that cannot do what its manifest asks. And both engines must mean the same thing by history. This page shows how each of the two is enforced, and how far each has been checked.
Picking the engine
src/dataplatform/silver/engines.py:35The dataset's setting first, then the domain's, then the default.def engine_of(settings: Mapping[str, str], platform: Mapping[str, str]) -> str:
return str(settings.get("silver_engine") or platform.get("silver_engine") or DEFAULT_ENGINE)| Key | What it means | Allowed values | Default |
|---|---|---|---|
silver_engine | Which engine builds silver for this dataset. It can be set on the dataset or for the whole domain. | job, declarative | job |
table_format | The open table format the dataset's tables are written in. It limits which engine may be chosen. | iceberg, delta, hudi | Set by the deployment |
What each engine can do, written as data
src/dataplatform/silver/engines.py:27The capabilities of each engine: the history types it writes and the table formats it writes natively.CAPS: dict[str, EngineCaps] = {
"job": EngineCaps(frozenset({"scd1", "scd2", "append", "snapshot", "source_versioned"}), None),
# Spike 13-14 Nov: managed change capture keeps SCD Type 1 and 2; the bound declarative product writes its own
# format natively and the open format only as added metadata, so a native open-format contract stays on `job`.
"declarative": EngineCaps(frozenset({"scd1", "scd2", "append"}), frozenset({"delta"})),
}The job engine writes all five history types, in any format, because it writes through the table port. The declarative engine writes type 1, type 2 and append, and only Delta natively. The comment records why. In the spike, the pipeline's tables were Delta tables that other engines can read as Iceberg through added metadata. They were not native Iceberg tables.
Because the capabilities are data, a deployment can pass its own table to engine_problems and widen or narrow an engine without changing this file.
The refusals, and why each exists
src/dataplatform/silver/engines.py:46Every reason a dataset cannot run on its chosen engine. An empty list means it can.def engine_problems(name: str, manifest: Mapping[str, object], settings: Mapping[str, str],
platform: Mapping[str, str], caps: Mapping[str, EngineCaps] | None = None) -> list[str]:
"""Why this dataset cannot run on its chosen engine; empty when it can."""
caps = CAPS if caps is None else caps
engine = engine_of(settings, platform)
if engine not in caps:
return [f"{name}: silver_engine '{engine}' is not one of {sorted(caps)}"]
c, out = caps[engine], []
hist = history_type(manifest)
if hist and hist not in c.history:
out.append(f"{name}: history '{hist}' is not supported by silver_engine '{engine}' "
f"(supports {sorted(c.history)}); use silver_engine 'job'")
load = manifest.get("load")
deletes = load.get("deletes") if isinstance(load, Mapping) else None
if engine == "declarative" and isinstance(deletes, Mapping) and deletes.get("detect") == "key_compare":
out.append(f"{name}: deletes found by comparing full copies are not supported by silver_engine "
f"'{engine}'; use silver_engine 'job'")
fmt = str(settings.get("table_format") or platform.get("table_format") or "")
if c.formats is not None and fmt and fmt not in c.formats:
out.append(f"{name}: table_format '{fmt}' is not written natively by silver_engine '{engine}' "
f"(writes {sorted(c.formats)}); a contract that needs native '{fmt}' stays on silver_engine 'job'")
return out| Combination | Message says | Why it is refused |
|---|---|---|
| An engine name that is not known | silver_engine is not one of the known engines | A typing mistake must not fall back to a default silently. |
| Declarative with snapshot history | history is not supported, use the job engine | Replacing a window means comparing a file with what silver holds. A streaming flow never has the whole table. |
| Declarative with source versioned history | history is not supported, use the job engine | Checking a key's versions for overlaps and gaps needs all of them together. |
| Declarative with deletes detected by key compare | deletes found by comparing full copies are not supported | Finding a missing key means comparing the copy with silver's live keys. |
| Declarative with a table format it does not write natively | the format is not written natively | A contract that promises a native open format must not be met with a different format plus metadata. |
The same list is checked in two places. The loader checks it when the manifests are loaded into the metadata store, so a bad combination is refused before anything runs. The recipe checks it again when the pipeline is declared. The loader page covers the loader.
src/dataplatform/meta/loader.py:143The loader adds the engine problems to the manifest's other errors. Any error stops the load. errors += [f"{path.name}: {p}" for p in engine_problems(
f"{doc['source_name']}.{doc['dataset_name']}", manifest, doc["settings"], domain_doc["settings"])]The history contract both engines must pass
The contract is eight cases written as plain data. Each case gives batches of changes in arrival order, and what silver must mean afterwards as one line per version: key, value, the sequence the version starts at, whether it is current, whether it is deleted.
src/dataplatform/silver/history_contract.py:20The eight cases.CASES: list[tuple[str, list[list[Change]], list[Version]]] = [
("first load: one current version per key",
[[("a", 1, "x", 1, "I", 0, 1), ("b", 5, "x", 2, "I", 0, 2)]],
[("a", 1, 1, True, False), ("b", 5, 2, True, False)]),
("a tracked change opens a version",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 2, "x", 3, "U", 5, 2)]],
[("a", 1, 1, False, False), ("a", 2, 3, True, False)]),
("an untracked change updates the version in place",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "y", 3, "U", 5, 2)]],
[("a", 1, 1, True, False)]),
("several changes in one batch become several versions, in order",
[[("a", 1, "x", 1, "I", 0, 1), ("a", 3, "x", 5, "U", 9, 3), ("a", 2, "x", 3, "U", 5, 2)]],
[("a", 1, 1, False, False), ("a", 2, 3, False, False), ("a", 3, 5, True, False)]),
("an older change arriving after a newer one goes into history, the newer stays current (SCD-02)",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 2, "x", 4, "U", 65, 3)]],
[("a", 1, 1, False, False), ("a", 2, 4, False, False), ("a", 3, 5, True, False)]),
("a late change equal to the version after it moves that version's start",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 3, "x", 4, "U", 4, 3)]],
[("a", 1, 1, False, False), ("a", 3, 4, True, False)]),
("a delete marks the key deleted, nothing removed (DEL-01)",
[[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "x", 2, "D", 5, 2)]],
[("a", 1, 1, True, True)]),
("the same batch twice changes nothing",
[[("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)],
[("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)]],
[("a", 1, 1, False, False), ("a", 2, 2, True, False)]),
]| Case | What it pins down |
|---|---|
| 1 | A first load gives one current version per key. |
| 2 | A tracked change opens a version. |
| 3 | A change that is not tracked updates the version in place. |
| 4 | Several changes in one batch become versions in sequence order. |
| 5 | An older change that arrives after a newer one goes into history. The newer one stays current. |
| 6 | A late change equal to the version after it moves that version's start. |
| 7 | A delete marks the key deleted. Nothing is removed. |
| 8 | The same batch twice changes nothing. |
The type 2 page works through all eight with before and after tables.
The engines store a deleted key differently, so their tables are first brought to the contract's shape. The job engine needs no help: its columns are the contract's values. For an engine that closes the deleted version there is a helper. The file's description calls this step normalise. The function it provides is normalise_closed.
src/dataplatform/silver/history_contract.py:53For an engine that closes a deleted key: the last version is current, and it is deleted when it has an end.def normalise_closed(rows: Iterable[Mapping[str, Any]], key: str, value: str, start: str, end: str) -> list[Version]:
"""An engine that closes a deleted key's last version (end set, nothing after it): the last version of a key is
'current'; it is 'deleted' when it is closed."""
by_key: dict[Any, list[Mapping[str, Any]]] = {}
for r in rows:
by_key.setdefault(r[key], []).append(r)
out: list[Version] = []
for k in sorted(by_key):
vs = sorted(by_key[k], key=lambda r: int(r[start]))
for i, r in enumerate(vs):
last = i == len(vs) - 1
out.append((str(k), int(r[value]), int(r[start]), last, last and r[end] is not None))
return outSide by side
Each row is one job that bronze to silver has to do. The two columns say who does it on each engine.
| Job engine | Declarative engine | |
|---|---|---|
| Who orders changes | Our history stage, by the order column | The change capture, by sequence_by |
| Who writes | Our writer, one merge on _row_key or an append with a transaction number | The pipeline |
| Deletes from a change feed | Marked: the version stays current and is flagged. Removed when the rule is remove. | The key's last version is closed. The two apply rules are not told apart. |
| Deletes by comparing full copies | Yes, behind the mass-delete guard | Refused |
| Late changes | Placed in history by the order column. A touched key's versions are computed again. | Placed in history by the sequence |
| Full copies | Yes | Only when deletes are not detected by key compare |
| Snapshots | Yes | Refused |
| Source versioned history | Yes | Refused |
| Duplicates | Stage 10, including a look into silver | For append datasets, the pipeline drops repeats by key. For type 1 and type 2, the change capture's keys and sequence. |
| Quarantine | Set-aside rows go to the quarantine table | The same, through its own flow |
| Hold on a critical rule | The whole batch is held and the bookmark stays | Not possible. The rows are quarantined and the rest is written. |
| Run log | Every stage's counts go to the step log | Not written. The pipeline's own event log is the record. |
| What was read | The file log and the bookmark | The pipeline's stream checkpoints |
| Table format | Any, through the table port | Delta natively |
| Where it runs | Spark, and DuckDB for tests and small runs | Only inside the pipeline product |
When to choose which
| If the dataset | Choose | Because |
|---|---|---|
| Is a full copy that can lose rows | Job | Only the job engine finds deletes by comparison. |
| Is a snapshot or keeps its own versions at the source | Job | The declarative engine refuses both. |
| Has a critical rule that must stop the batch | Job | Only the job engine can hold a batch. |
| Must be in a native open format other than Delta | Job | The declarative engine writes Delta natively. |
| Must appear in the platform's run log, stage by stage | Job | The pipeline does not write it. |
Is read downstream by its _is_current and _is_deleted columns | Job | The declarative table does not have them. |
| Is a change feed or an event stream, and the team wants the product to manage the writes | Declarative | Type 1, type 2 and append are what it supports, from the same manifest. |
| Is anything else, or you are unsure | Job | It is the default and takes every dataset. |
How it is tested
tests/dataplatform/test_silver_engines.py is four tables. They cover which engine a dataset gets, the history type read from a manifest, every refusal with the words of its message, and loading the real tyre manifests with the engine setting changed.
tests/dataplatform/test_silver_engines.py:45The refusal table. The last column is the words each message must contain. An empty list means accepted.@pytest.mark.parametrize(
("case", "manifest", "settings", "platform", "says"),
[
("job takes everything", SNAPSHOT, {"table_format": "iceberg"}, {}, []),
("declarative takes scd2", SCD2, {"silver_engine": "declarative", "table_format": "delta"}, {}, []),
("declarative takes append", EVENT, {"silver_engine": "declarative"}, {}, []),
("declarative refuses snapshot history", SNAPSHOT, {"silver_engine": "declarative"}, {},
["history 'snapshot'"]),
("declarative refuses source-versioned history", SOURCE_VERSIONED, {"silver_engine": "declarative"}, {},
["history 'source_versioned'"]),
("declarative refuses a native format it does not write", SCD2, {"silver_engine": "declarative"},
{"table_format": "iceberg"}, ["table_format 'iceberg'"]),
("both problems reported", SNAPSHOT, {"silver_engine": "declarative", "table_format": "iceberg"}, {},
["history 'snapshot'", "table_format 'iceberg'"]),
("declarative refuses full-copy deletes", {**SCD2, "load": {"deletes": {"detect": "key_compare"}}},
{"silver_engine": "declarative"}, {}, ["comparing full copies"]),
("job takes full-copy deletes", {**SCD2, "load": {"deletes": {"detect": "key_compare"}}}, {}, {}, []),
("unknown engine", SCD2, {"silver_engine": "notebook"}, {}, ["silver_engine 'notebook' is not one of"]),
],
)
def test_engine_problems(case, manifest, settings, platform, says):
got = engine_problems("erp.x", manifest, settings, platform)
assert len(got) == len(says)
assert all(s in g for s, g in zip(says, got, strict=True))tests/dataplatform/test_silver_engines.py:71A deployment's own capability table lifts a refusal.def test_a_binding_can_widen_an_engine():
wide = {"declarative": EngineCaps(frozenset({"scd1", "scd2", "append"}), frozenset({"delta", "iceberg"}))}
assert engine_problems("erp.x", SCD2, {"silver_engine": "declarative", "table_format": "iceberg"}, {}, wide) == []The history contract is not run in that file. The job engine runs it in tests/dataplatform/test_silver_stages.py, once per case. The declarative engine runs it through the spike's runner, which needs the pipeline product.
spikes/sdp_silver/lakeflow/run_contract.py:54The declarative side of the contract: read each case's table, bring it to the contract's shape, compare.for case, _, want in CASES:
rows = sql(f"SELECT k, v, __START_AT, __END_AT FROM dev_silver.spike_contract.c_{slug(case)}")
got = normalise_closed(rows, "k", "v", "__START_AT", "__END_AT")
good = got == wantWhat is verified and what is not
Each row is one claim about the two engines. The second column says how it was checked for this page, or that it could not be.
| Claim | Status |
|---|---|
| Six of the nine tyre datasets are accepted by the declarative engine and three are refused | Verified. engine_problems was run on each of the nine manifests with the engine set to declarative. Dealers, deliveries, sales order lines, lab inspection results, production records and press readings returned no problem. Materials and presses were refused for deletes by key compare. Dealer sell-out was refused for snapshot history. |
| The job engine passes the eight contract cases | Verified by running the eight cases through the stage functions on DuckDB and reading the result. The test file was not run as a whole for this page. The design note records it passing on Spark. |
| The loader refuses a tyre dataset that asks for an engine it cannot use | Verified in the code, where the loader adds the problems to its errors. The test with the real tyre manifests has two cases and both are accepted. Its refused branch is never taken. |
| The declarative engine places a late change in history and closes a deleted key | Recorded, not repeated. The spike's notes say all five of its checks passed on the managed pipeline service on 13 and 14 November 2026. |
| The declarative engine passes the eight contract cases | Not verified. The pipeline and the runner are in the repository. No recorded result is. |
| The declarative engine's tables equal the job engine's on the six datasets | Not verified. A comparison script is in the repository beside the pipeline. It compares live rows, version counts and business columns for those six datasets. It needs a deployed workspace, and no output of it is recorded in the repository. The public snapshot's design notes do not mention the declarative engine at all. |
- No test runs the two engines on the same data in one place. The contract is shared, but each engine runs it in its own harness.
- The contract covers type 2 only. Type 1 and append on the declarative engine have recipe tests and no shared cases.
- The comparison of the two engines' tables looks at live rows. It does not compare how a deleted key is stored, which is the known difference.
- The capability table lists what the declarative engine supports. It does not list what it lacks at run time: the hold on a critical rule and the run log are not refusals, so a dataset that needs them can still be moved to the declarative engine without a warning.