Data PlatformZoom level 4
Append history: events are only ever added
An event is something that happened at a moment: a press reading, a lab inspection, a production record. It does not change afterwards. Append history adds new events to silver and touches nothing that is already there.
This is the cheapest history, and the biggest tables use it. In the tyre company the press readings alone grow by 622,080 rows a day in the recorded run. Rewriting or comparing versions for such a table would be wasted work.
upsert.A worked example
Three press readings arrive. The third one was measured at 00:01 but reached the platform an hour late, in the next batch.
| Arrives in | Reading | Event time |
|---|---|---|
| batch 1 | r1 | 00:00 |
| batch 1 | r2 | 01:01 |
| batch 2 | r3 | 00:01 |
All three are appended. The late reading keeps its own event time, so it is counted in the hour it was measured, not the hour it arrived.
| Reading | _valid_from | _is_current | _is_deleted |
|---|---|---|---|
| r1 | 00:00 | yes | no |
| r2 | 01:01 | yes | no |
| r3 | 00:01 | yes | no |
The settings that drive it
data-platform-catalog/slices/02-tyre/manifest/datasets/press_stream.press_reading.yaml:18Press readings: a repeat is found by the reading id.dedup: {mode: business_key_first, business_keys: [reading_id]}data-platform-catalog/slices/02-tyre/manifest/datasets/press_stream.press_reading.yaml:23The kind is event, so the history type is append.history: {kind: event}
load:
pattern: append
batch_key: [source_batch_id]
new_columns: hold_in_bronze
deletes: {detect: none, apply: none}
late_lookback_days: 2| Key | What it means | Value for press readings |
|---|---|---|
history.kind | An event dataset is appended | event |
config.watermark_column | When this column is a timestamp it is the event time | event_ts |
load.late_lookback_days | How far back stage 10 looks in silver for a repeat. Absent means all of silver. | 2 |
The code
src/dataplatform/stages/history.py:197The history columns of a row that has no history of its own.def _live() -> dict[str, Column]:
"""A row that is current, not deleted, with no version history of its own."""
return {
"_valid_from": F.lit(None).cast("timestamp"),
"_valid_to": F.lit(None).cast("timestamp"),
"_is_current": F.lit(True),
"_is_deleted": F.lit(False),
"_deleted_at": F.lit(None).cast("timestamp"),
"_seq_from": F.lit(None).cast("string"),
}src/dataplatform/stages/history.py:209The whole of append history.def append(inc: DataFrame, cur: DataFrame | None, plan: SilverPlan) -> DataFrame:
if plan.delete_apply == "none" and "_op" in inc.columns: # a delete row is not a new fact (as in scd1 and scd2)
inc = inc.where(F.coalesce(F.col("_op"), F.lit("U")) != "D")
has_event = bool(plan.event_time) and plan.event_time in inc.columns
when = F.col(plan.event_time).cast("timestamp") if has_event else _ts(inc, plan)
return _finish(inc.withColumns({**_live(), "_valid_from": when, "_row_key": F.col("_key_hash")}), None, plan)_live returns the column values of a live row: current, not deleted, no validity and no start in the source order. append lays those over the rows and then sets two columns of its own. _valid_from becomes the event time when the plan has one and the column is in the batch. Otherwise it is the effective time, which may be empty. The row key is the key hash.
The function never looks at silver. It does not need to. Stage 10 has already dropped any reading whose key is in silver within the lookback, and the writer appends with a transaction number so that a repeated write is skipped. The writer page explains the transaction number.
How it is tested
tests/dataplatform/test_silver_stages.py:453Three cases, run through all the stages: new rows, a repeat across batches, a repeat inside a batch. ("new rows appended", [[("r1", 1, 1, 1)], [("r2", 1, 1, 2)]], ["r1", "r2"]),
("a replay across batches is dropped (DUP-04)", [[("r1", 1, 1, 1)], [("r1", 1, 1, 2), ("r2", 1, 1, 3)]],
["r1", "r2"]),
("a replay within a batch is dropped", [[("r1", 1, 1, 1), ("r1", 1, 1, 2)]], ["r1"]),
],
)
def test_history_append(spark, case, batches, want):
p = plan(dedup={"mode": "business_key_first", "business_keys": ["id"]})
silver = _apply(spark, p, batches, "id string, v long, seq long, _seq long")
assert [r[0] for r in rows(silver, "id")] == want, casetests/dataplatform/test_silver_stages.py:465The worked example above: three readings, two event hours.def test_late_readings_keep_their_event_time(spark):
"""LATE-03: a reading that arrives late is appended with its own event time, so it counts in its event hour."""
p = plan(columns=[("id", "string"), ("event_ts", "timestamp")], settings={"watermark_column": "event_ts"},
dedup={"mode": "business_key_first", "business_keys": ["id"]})
silver = _apply(spark, p, [[("r1", ts(0)), ("r2", ts(61))], [("r3", ts(1))]], "id string, event_ts timestamp")
hours = silver.groupBy(F.date_trunc("hour", "_valid_from").alias("h")).count().collect()
assert sorted(r["count"] for r in hours) == [1, 2]In the recorded 14-day run of the tyre company at full scale (4 November 2026, local Spark), 188 readings arrived late on day 3 and silver's count per event hour equalled the oracle in all 24 hours. On day 13 the stream was replayed, 19,440 repeated rows were dropped, and silver held 8,087,040 readings, equal to the oracle. Both are quoted from docs/W1C-SILVER.md.
- Events cannot be deleted or corrected through this path. The function does not read the operation column, so a delete row in an event feed is not treated as a delete.
- An event dataset without an event time and without an effective time gets an empty
_valid_from.