Data PlatformZoom level 2
The writer: one merge per dataset, and why a rerun changes nothing
By the time rows reach the writer, every decision is made. The history stage has given each row a _row_key and an _action. The writer's only job is to apply them to the silver table in one transaction.
This split is deliberate. All the logic about versions, late changes and deletes sits in pure functions that are easy to test. The writer has no logic to get wrong.
What the two columns mean
| History type | What _row_key is | So the merge does this |
|---|---|---|
| Type 1 | The hash of the business key | One row per key. A change overwrites it. |
| Type 2 | The hash of the business key and the version's start | One row per version. A new version is an insert. Closing the old version is an update of that version's row. |
| Append | The hash of the business key | Rows are appended. No merge is needed. |
| Snapshot | The hash of the business key and the batch | A newer file for the same window adds its rows and updates the older rows to not current. |
The action is upsert for a row to write. It is delete in two cases. The first is a stale row: when a late change rebuilds a key's history, a version row that the new history no longer contains is removed. The second is a dataset whose delete setting is remove, where a deleted key's rows are taken out instead of marked.
src/dataplatform/adapters/spark_silver.py:171The write edge. Quarantine first, then append or merge. def write(self, plan: SilverPlan, result: StageResult) -> int:
path = self.silver_path(plan.source, plan.dataset)
txn = str(result.txn)
held = sum(v for k, v in result.counts.items() if k.endswith(".quarantined"))
if result.quarantine is not None and held:
self._append(result.quarantine, self.silver_path(plan.source, plan.dataset, "__quarantine"),
f"quarantine:{plan.table}", txn)
written = result.counts.get("rows_out", 0) + result.counts.get("rows_removed", 0)
if not written:
return 0
if plan.history == "append" or not self._exists(path):
self._append(result.rows.where(F.col("_action") == "upsert").drop("_action"), path,
f"silver:{plan.table}", txn)
else:
self._merge(result.rows, path)
return written- Quarantine. If any stage set rows aside, they are appended to the dataset's quarantine table, with the stage and the reason.
- Nothing to write. If the stages produced no rows to write or remove, the writer returns zero.
- First load or an append dataset. The rows are appended. A merge needs a table to merge into, and events only ever append.
- Everything else. One merge into the silver table.
src/dataplatform/adapters/spark_silver.py:193The merge. New columns are added first, then one statement applies every row. def _merge(self, rows: DataFrame, path: Path) -> None:
from delta.tables import DeltaTable
target = DeltaTable.forPath(self.spark, str(path))
have = {f.name for f in target.toDF().schema.fields}
new = [f for f in rows.schema.fields if f.name not in have and f.name != "_action"]
if new: # new columns reach silver without a release (ADR-008)
added = ", ".join(f"{_quoted(f.name)} {f.dataType.simpleString()}" for f in new)
# DDL takes no bind parameters, so the identifiers are quoted instead: _quoted() escapes every column name
# and the table path, and the types come from Spark's own schema, never from data.
# shiftleft: allow SL-SEC-01 -- identifiers only, each escaped by _quoted(); no values in the statement
self.spark.sql(f"ALTER TABLE delta.{_quoted(str(path))} ADD COLUMNS ({added})")
target = DeltaTable.forPath(self.spark, str(path))
cols = [c for c in rows.columns if c != "_action"]
values: dict[str, str | Column] = {f"`{c}`": f"s.`{c}`" for c in cols}
(
target.alias("t")
.merge(rows.alias("s"), "t._row_key = s._row_key")
.whenMatchedDelete(condition="s._action = 'delete'")
.whenMatchedUpdate(condition="s._action = 'upsert'", set=values)
.whenNotMatchedInsert(condition="s._action = 'upsert'", values=values)
.execute()Three things to notice. First, the match is on _row_key only. Second, a new column in the source reaches silver without a release: the writer adds any column the table does not have before merging. Third, the three clauses map one to one to the picture above.
Why running the same batch twice changes nothing
A write that can be repeated safely is called idempotent. Silver gets this from two places.
For merges, the row key and the version boundaries are computed from the data, never from the run. The same batch produces the same row keys and the same values, so the merge finds every row already there and updates it to what it already is.
For appends, the writer tags the write with an application name and a transaction number, which is the bookmark value. Delta remembers the last number per application and ignores a write that carries a number it has already seen.
src/dataplatform/adapters/spark_silver.py:188The idempotent append: the same application and number is written once. def _append(self, df: DataFrame, path: Path, app: str, txn: str) -> None:
(df.write.format("delta").mode("append").option("txnAppId", app).option("txnVersion", txn)
.option("mergeSchema", "true").save(str(path)))How it is tested
tests/dataplatform/test_silver_ports.py is the contract table for the silver tables port. Each adapter must pass the same cases: a write, the measures after it, the same append twice, removal of a stale row, a new column arriving, quarantine, and the bronze read.
- Type 2 history is written as one merge of whole version rows today. A full copy that changes many keys rewrites more files than a merge of only the differences would.
- The transaction number guard covers appends. Merges rely on computed row keys instead.