cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver

Data PlatformZoom level 1

Bronze to silver: from data as received to data you can trust

Bronze holds every batch exactly as the source sent it, mistakes and repeats included. Silver is the first layer people and gold can trust: one row per business key or per version, typed, cleaned, with deletes marked and history kept.

The work between the two is the same for every dataset. Nobody writes a job per table. One piece of code reads the dataset's settings from the metadata store and runs the same thirteen stages in the same order. What changes from one dataset to the next is only the settings.

Read it left to right. The highlighted box is where the rows are changed. Click any box to zoom in.

One run for one dataset, in order

  1. Take the lease. The runner takes a lease on the dataset. A lease is a short lock with a time limit, kept in the metadata store. A second run that starts at the same time finds the dataset busy and writes nothing.
  2. Build the plan. The dataset's merged settings, contract columns and keys become one plan object. A setting the engine does not know stops the dataset here, before any data is read.
  3. Pick the batches. The reader asks the file log which bronze batches were loaded since silver last ran, and reads only those.
  4. Run the stages. The new rows and the silver table as it stands go through the thirteen stages. The stages never read or write. They return the rows to write, the rows to set aside, and counts.
  5. Hold on a critical failure. If any row broke a rule marked critical, nothing is written and the bookmark stays where it was. The batch stays blocked until someone voids it or the source resends it.
  6. Write once. The writer applies all rows in one merge and appends the set-aside rows to the quarantine table.
  7. Record and move the bookmark. Each stage's counts go to the step log. The bookmark moves only when the dataset run succeeded, so a failed run is taken again whole next time.

The four design decisions that shape this layer

DecisionWhat it meansWhy
The stages are fixed and orderedEvery dataset passes the same thirteen stages in the same order. A dataset can switch six of them off. It cannot add a stage or reorder them.A fixed list can be tested once and trusted everywhere. A setting outside the list is refused, so nothing is silently ignored.
The stages are pure functionsA stage receives rows and returns rows. Reading bronze and writing silver happen only at the two edges.Pure functions run the same on Spark and on DuckDB, so the same tests run on a laptop in seconds and on the cluster.
History is chosen by the kind of datasetMaster data keeps every version (type 2). Transactions keep the latest state (type 1). Events are appended. Snapshots replace their window.The kind is a fact about the data. The dataset states its kind once, and the right history follows.
Deletes are softA row deleted at the source stays in silver, marked deleted, with the time. Nothing is removed unless the dataset asks for that.Gold and auditors need to know that a row existed and when it went away.

What a silver table looks like

Every silver table has the business columns first, then the columns silver adds. The added columns all start with an underscore. A row is live when _is_current is true and _is_deleted is false.

ColumnWhat it holds
_key_hashA hash of the business key. Every later step joins and groups on it.
_row_keyThe identity of the row in silver. The writer merges on it. For type 2 it is the key plus the version's start, so each version is its own row.
_seq_fromFor type 2: the source order value at which the version starts.
_valid_from, _valid_toThe time span during which the version was true. An open end means it is still true.
_is_currentTrue on the latest version of a key.
_is_deleted, _deleted_atSet when the source deleted the row, with the time of the delete.
_dq_flagsThe names of the minor quality rules the row broke. The row is kept.
_record_hashA hash of the business values of the row. Nothing in the platform reads it yet.
_batch_key, _source_file, _bronze_run_id, _silver_run_id, _silver_loaded_atWhere the row came from and which runs carried it. This is the row's provenance.
src/dataplatform/silver/plan.py:74The columns silver adds, as the code lists them.
SILVER_COLUMNS = (
    "_key_hash",
    "_record_hash",
    "_seq_from",
    "_valid_from",
    "_valid_to",
    "_is_current",
    "_is_deleted",
    "_deleted_at",
    "_dq_flags",
    "_row_key",
    "_bronze_run_id",
    "_silver_run_id",
    "_silver_loaded_at",
)

Two engines, one set of stages

The stages can run in two ways. The job engine is the default: our own runner calls the stages and writes with one merge. The declarative engine runs the same stage functions inside a declarative pipeline and lets the pipeline's change capture keep the history. The setting silver_engine picks one per dataset. Both must pass the same table of history cases. See Declarative engine and Job engine and declarative engine.

How we know it is right

In the recorded 14-day run of the tyre company at full scale (4 November 2026, local Spark), 294 numbers were compared between silver and the oracle over the 14 days, and all were equal. The oracle is the simulated company's own record of what is true. Tests and proof explains it. The run's table is in docs/W1C-SILVER.md.

Not built yet, or known gaps
  • The plan does not yet receive the domain's quality defaults from the metadata store, so the severity actions fall back to defaults written in the plan code.
  • On the declarative engine, three of the nine tyre datasets are refused (two full copies that detect deletes by key compare, and the snapshot). They run on the job engine.
  • A critical rule holds the whole batch on the job engine. The declarative engine quarantines the rows but cannot hold the batch.

Look inside