cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › Job engine and declarative engine

Data PlatformZoom level 2

Two engines for silver: the job engine and the declarative engine

Bronze to silver can be run by two engines. The job engine is our own runner: it calls the thirteen stages and writes with one merge. The declarative engine runs the stages inside a declarative pipeline and lets the pipeline's change capture keep the history. A dataset picks one with a single setting.

Two engines are only safe if two things hold. A dataset must never be run by an engine that cannot do what its manifest asks. And both engines must mean the same thing by history. This page shows how each of the two is enforced, and how far each has been checked.

Both lanes start from the same plan and are judged by the same eight cases. The dashed boxes belong to the product, not to our code.

Picking the engine

src/dataplatform/silver/engines.py:35The dataset's setting first, then the domain's, then the default.
def engine_of(settings: Mapping[str, str], platform: Mapping[str, str]) -> str:
    return str(settings.get("silver_engine") or platform.get("silver_engine") or DEFAULT_ENGINE)
KeyWhat it meansAllowed valuesDefault
silver_engineWhich engine builds silver for this dataset. It can be set on the dataset or for the whole domain.job, declarativejob
table_formatThe open table format the dataset's tables are written in. It limits which engine may be chosen.iceberg, delta, hudiSet by the deployment

What each engine can do, written as data

src/dataplatform/silver/engines.py:27The capabilities of each engine: the history types it writes and the table formats it writes natively.
CAPS: dict[str, EngineCaps] = {
    "job": EngineCaps(frozenset({"scd1", "scd2", "append", "snapshot", "source_versioned"}), None),
    # Spike 13-14 Nov: managed change capture keeps SCD Type 1 and 2; the bound declarative product writes its own
    # format natively and the open format only as added metadata, so a native open-format contract stays on `job`.
    "declarative": EngineCaps(frozenset({"scd1", "scd2", "append"}), frozenset({"delta"})),
}

The job engine writes all five history types, in any format, because it writes through the table port. The declarative engine writes type 1, type 2 and append, and only Delta natively. The comment records why. In the spike, the pipeline's tables were Delta tables that other engines can read as Iceberg through added metadata. They were not native Iceberg tables.

Because the capabilities are data, a deployment can pass its own table to engine_problems and widen or narrow an engine without changing this file.

The refusals, and why each exists

src/dataplatform/silver/engines.py:46Every reason a dataset cannot run on its chosen engine. An empty list means it can.
def engine_problems(name: str, manifest: Mapping[str, object], settings: Mapping[str, str],
                    platform: Mapping[str, str], caps: Mapping[str, EngineCaps] | None = None) -> list[str]:
    """Why this dataset cannot run on its chosen engine; empty when it can."""
    caps = CAPS if caps is None else caps
    engine = engine_of(settings, platform)
    if engine not in caps:
        return [f"{name}: silver_engine '{engine}' is not one of {sorted(caps)}"]
    c, out = caps[engine], []
    hist = history_type(manifest)
    if hist and hist not in c.history:
        out.append(f"{name}: history '{hist}' is not supported by silver_engine '{engine}' "
                   f"(supports {sorted(c.history)}); use silver_engine 'job'")
    load = manifest.get("load")
    deletes = load.get("deletes") if isinstance(load, Mapping) else None
    if engine == "declarative" and isinstance(deletes, Mapping) and deletes.get("detect") == "key_compare":
        out.append(f"{name}: deletes found by comparing full copies are not supported by silver_engine "
                   f"'{engine}'; use silver_engine 'job'")
    fmt = str(settings.get("table_format") or platform.get("table_format") or "")
    if c.formats is not None and fmt and fmt not in c.formats:
        out.append(f"{name}: table_format '{fmt}' is not written natively by silver_engine '{engine}' "
                   f"(writes {sorted(c.formats)}); a contract that needs native '{fmt}' stays on silver_engine 'job'")
    return out
CombinationMessage saysWhy it is refused
An engine name that is not knownsilver_engine is not one of the known enginesA typing mistake must not fall back to a default silently.
Declarative with snapshot historyhistory is not supported, use the job engineReplacing a window means comparing a file with what silver holds. A streaming flow never has the whole table.
Declarative with source versioned historyhistory is not supported, use the job engineChecking a key's versions for overlaps and gaps needs all of them together.
Declarative with deletes detected by key comparedeletes found by comparing full copies are not supportedFinding a missing key means comparing the copy with silver's live keys.
Declarative with a table format it does not write nativelythe format is not written nativelyA contract that promises a native open format must not be met with a different format plus metadata.

The same list is checked in two places. The loader checks it when the manifests are loaded into the metadata store, so a bad combination is refused before anything runs. The recipe checks it again when the pipeline is declared. The loader page covers the loader.

src/dataplatform/meta/loader.py:143The loader adds the engine problems to the manifest's other errors. Any error stops the load.
        errors += [f"{path.name}: {p}" for p in engine_problems(
            f"{doc['source_name']}.{doc['dataset_name']}", manifest, doc["settings"], domain_doc["settings"])]

The history contract both engines must pass

The contract is eight cases written as plain data. Each case gives batches of changes in arrival order, and what silver must mean afterwards as one line per version: key, value, the sequence the version starts at, whether it is current, whether it is deleted.

CASES: list[tuple[str, list[list[Change]], list[Version]]] = [
    ("first load: one current version per key",
     [[("a", 1, "x", 1, "I", 0, 1), ("b", 5, "x", 2, "I", 0, 2)]],
     [("a", 1, 1, True, False), ("b", 5, 2, True, False)]),
    ("a tracked change opens a version",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 2, "x", 3, "U", 5, 2)]],
     [("a", 1, 1, False, False), ("a", 2, 3, True, False)]),
    ("an untracked change updates the version in place",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "y", 3, "U", 5, 2)]],
     [("a", 1, 1, True, False)]),
    ("several changes in one batch become several versions, in order",
     [[("a", 1, "x", 1, "I", 0, 1), ("a", 3, "x", 5, "U", 9, 3), ("a", 2, "x", 3, "U", 5, 2)]],
     [("a", 1, 1, False, False), ("a", 2, 3, False, False), ("a", 3, 5, True, False)]),
    ("an older change arriving after a newer one goes into history, the newer stays current (SCD-02)",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 2, "x", 4, "U", 65, 3)]],
     [("a", 1, 1, False, False), ("a", 2, 4, False, False), ("a", 3, 5, True, False)]),
    ("a late change equal to the version after it moves that version's start",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 3, "x", 5, "U", 5, 2)], [("a", 3, "x", 4, "U", 4, 3)]],
     [("a", 1, 1, False, False), ("a", 3, 4, True, False)]),
    ("a delete marks the key deleted, nothing removed (DEL-01)",
     [[("a", 1, "x", 1, "I", 0, 1)], [("a", 1, "x", 2, "D", 5, 2)]],
     [("a", 1, 1, True, True)]),
    ("the same batch twice changes nothing",
     [[("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)],
      [("a", 1, "x", 1, "I", 0, 1), ("a", 2, "x", 2, "U", 1, 2)]],
     [("a", 1, 1, False, False), ("a", 2, 2, True, False)]),
]
CaseWhat it pins down
1A first load gives one current version per key.
2A tracked change opens a version.
3A change that is not tracked updates the version in place.
4Several changes in one batch become versions in sequence order.
5An older change that arrives after a newer one goes into history. The newer one stays current.
6A late change equal to the version after it moves that version's start.
7A delete marks the key deleted. Nothing is removed.
8The same batch twice changes nothing.

The type 2 page works through all eight with before and after tables.

The engines store a deleted key differently, so their tables are first brought to the contract's shape. The job engine needs no help: its columns are the contract's values. For an engine that closes the deleted version there is a helper. The file's description calls this step normalise. The function it provides is normalise_closed.

src/dataplatform/silver/history_contract.py:53For an engine that closes a deleted key: the last version is current, and it is deleted when it has an end.
def normalise_closed(rows: Iterable[Mapping[str, Any]], key: str, value: str, start: str, end: str) -> list[Version]:
    """An engine that closes a deleted key's last version (end set, nothing after it): the last version of a key is
    'current'; it is 'deleted' when it is closed."""
    by_key: dict[Any, list[Mapping[str, Any]]] = {}
    for r in rows:
        by_key.setdefault(r[key], []).append(r)
    out: list[Version] = []
    for k in sorted(by_key):
        vs = sorted(by_key[k], key=lambda r: int(r[start]))
        for i, r in enumerate(vs):
            last = i == len(vs) - 1
            out.append((str(k), int(r[value]), int(r[start]), last, last and r[end] is not None))
    return out

Side by side

Each row is one job that bronze to silver has to do. The two columns say who does it on each engine.

Job engineDeclarative engine
Who orders changesOur history stage, by the order columnThe change capture, by sequence_by
Who writesOur writer, one merge on _row_key or an append with a transaction numberThe pipeline
Deletes from a change feedMarked: the version stays current and is flagged. Removed when the rule is remove.The key's last version is closed. The two apply rules are not told apart.
Deletes by comparing full copiesYes, behind the mass-delete guardRefused
Late changesPlaced in history by the order column. A touched key's versions are computed again.Placed in history by the sequence
Full copiesYesOnly when deletes are not detected by key compare
SnapshotsYesRefused
Source versioned historyYesRefused
DuplicatesStage 10, including a look into silverFor append datasets, the pipeline drops repeats by key. For type 1 and type 2, the change capture's keys and sequence.
QuarantineSet-aside rows go to the quarantine tableThe same, through its own flow
Hold on a critical ruleThe whole batch is held and the bookmark staysNot possible. The rows are quarantined and the rest is written.
Run logEvery stage's counts go to the step logNot written. The pipeline's own event log is the record.
What was readThe file log and the bookmarkThe pipeline's stream checkpoints
Table formatAny, through the table portDelta natively
Where it runsSpark, and DuckDB for tests and small runsOnly inside the pipeline product

When to choose which

If the datasetChooseBecause
Is a full copy that can lose rowsJobOnly the job engine finds deletes by comparison.
Is a snapshot or keeps its own versions at the sourceJobThe declarative engine refuses both.
Has a critical rule that must stop the batchJobOnly the job engine can hold a batch.
Must be in a native open format other than DeltaJobThe declarative engine writes Delta natively.
Must appear in the platform's run log, stage by stageJobThe pipeline does not write it.
Is read downstream by its _is_current and _is_deleted columnsJobThe declarative table does not have them.
Is a change feed or an event stream, and the team wants the product to manage the writesDeclarativeType 1, type 2 and append are what it supports, from the same manifest.
Is anything else, or you are unsureJobIt is the default and takes every dataset.

How it is tested

tests/dataplatform/test_silver_engines.py is four tables. They cover which engine a dataset gets, the history type read from a manifest, every refusal with the words of its message, and loading the real tyre manifests with the engine setting changed.

tests/dataplatform/test_silver_engines.py:45The refusal table. The last column is the words each message must contain. An empty list means accepted.
@pytest.mark.parametrize(
    ("case", "manifest", "settings", "platform", "says"),
    [
        ("job takes everything", SNAPSHOT, {"table_format": "iceberg"}, {}, []),
        ("declarative takes scd2", SCD2, {"silver_engine": "declarative", "table_format": "delta"}, {}, []),
        ("declarative takes append", EVENT, {"silver_engine": "declarative"}, {}, []),
        ("declarative refuses snapshot history", SNAPSHOT, {"silver_engine": "declarative"}, {},
         ["history 'snapshot'"]),
        ("declarative refuses source-versioned history", SOURCE_VERSIONED, {"silver_engine": "declarative"}, {},
         ["history 'source_versioned'"]),
        ("declarative refuses a native format it does not write", SCD2, {"silver_engine": "declarative"},
         {"table_format": "iceberg"}, ["table_format 'iceberg'"]),
        ("both problems reported", SNAPSHOT, {"silver_engine": "declarative", "table_format": "iceberg"}, {},
         ["history 'snapshot'", "table_format 'iceberg'"]),
        ("declarative refuses full-copy deletes", {**SCD2, "load": {"deletes": {"detect": "key_compare"}}},
         {"silver_engine": "declarative"}, {}, ["comparing full copies"]),
        ("job takes full-copy deletes", {**SCD2, "load": {"deletes": {"detect": "key_compare"}}}, {}, {}, []),
        ("unknown engine", SCD2, {"silver_engine": "notebook"}, {}, ["silver_engine 'notebook' is not one of"]),
    ],
)
def test_engine_problems(case, manifest, settings, platform, says):
    got = engine_problems("erp.x", manifest, settings, platform)
    assert len(got) == len(says)
    assert all(s in g for s, g in zip(says, got, strict=True))
tests/dataplatform/test_silver_engines.py:71A deployment's own capability table lifts a refusal.
def test_a_binding_can_widen_an_engine():
    wide = {"declarative": EngineCaps(frozenset({"scd1", "scd2", "append"}), frozenset({"delta", "iceberg"}))}
    assert engine_problems("erp.x", SCD2, {"silver_engine": "declarative", "table_format": "iceberg"}, {}, wide) == []

The history contract is not run in that file. The job engine runs it in tests/dataplatform/test_silver_stages.py, once per case. The declarative engine runs it through the spike's runner, which needs the pipeline product.

spikes/sdp_silver/lakeflow/run_contract.py:54The declarative side of the contract: read each case's table, bring it to the contract's shape, compare.
for case, _, want in CASES:
    rows = sql(f"SELECT k, v, __START_AT, __END_AT FROM dev_silver.spike_contract.c_{slug(case)}")
    got = normalise_closed(rows, "k", "v", "__START_AT", "__END_AT")
    good = got == want

What is verified and what is not

Each row is one claim about the two engines. The second column says how it was checked for this page, or that it could not be.

ClaimStatus
Six of the nine tyre datasets are accepted by the declarative engine and three are refusedVerified. engine_problems was run on each of the nine manifests with the engine set to declarative. Dealers, deliveries, sales order lines, lab inspection results, production records and press readings returned no problem. Materials and presses were refused for deletes by key compare. Dealer sell-out was refused for snapshot history.
The job engine passes the eight contract casesVerified by running the eight cases through the stage functions on DuckDB and reading the result. The test file was not run as a whole for this page. The design note records it passing on Spark.
The loader refuses a tyre dataset that asks for an engine it cannot useVerified in the code, where the loader adds the problems to its errors. The test with the real tyre manifests has two cases and both are accepted. Its refused branch is never taken.
The declarative engine places a late change in history and closes a deleted keyRecorded, not repeated. The spike's notes say all five of its checks passed on the managed pipeline service on 13 and 14 November 2026.
The declarative engine passes the eight contract casesNot verified. The pipeline and the runner are in the repository. No recorded result is.
The declarative engine's tables equal the job engine's on the six datasetsNot verified. A comparison script is in the repository beside the pipeline. It compares live rows, version counts and business columns for those six datasets. It needs a deployed workspace, and no output of it is recorded in the repository. The public snapshot's design notes do not mention the declarative engine at all.
Not built yet, or known gaps
  • No test runs the two engines on the same data in one place. The contract is shared, but each engine runs it in its own harness.
  • The contract covers type 2 only. Type 1 and append on the declarative engine have recipe tests and no shared cases.
  • The comparison of the two engines' tables looks at live rows. It does not compare how a deleted key is stored, which is the known difference.
  • The capability table lists what the declarative engine supports. It does not list what it lacks at run time: the hold on a critical rule and the run log are not refusals, so a dataset that needs them can still be moved to the declarative engine without a warning.