cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Bronze to silver › The thirteen stages › 12 History › Append: events

Data PlatformZoom level 4

Append history: events are only ever added

An event is something that happened at a moment: a press reading, a lab inspection, a production record. It does not change afterwards. Append history adds new events to silver and touches nothing that is already there.

This is the cheapest history, and the biggest tables use it. In the tyre company the press readings alone grow by 622,080 rows a day in the recorded run. Rewriting or comparing versions for such a table would be wasted work.

The function in the middle adds columns and nothing else. Removing repeats is stage 10's job.
Goes inRows that stage 10 has already cleared of repeats, both inside the batch and against silver.
What happensEach row gets the history columns of a live row and its event time.
Comes outThe same rows, with the key hash as row key and the action upsert.

A worked example

Three press readings arrive. The third one was measured at 00:01 but reached the platform an hour late, in the next batch.

Arrives inReadingEvent time
batch 1r100:00
batch 1r201:01
batch 2r300:01

All three are appended. The late reading keeps its own event time, so it is counted in the hour it was measured, not the hour it arrived.

Reading_valid_from_is_current_is_deleted
r100:00yesno
r201:01yesno
r300:01yesno

The settings that drive it

dedup: {mode: business_key_first, business_keys: [reading_id]}
history: {kind: event}
load:
  pattern: append
  batch_key: [source_batch_id]
  new_columns: hold_in_bronze
  deletes: {detect: none, apply: none}
  late_lookback_days: 2
KeyWhat it meansValue for press readings
history.kindAn event dataset is appendedevent
config.watermark_columnWhen this column is a timestamp it is the event timeevent_ts
load.late_lookback_daysHow far back stage 10 looks in silver for a repeat. Absent means all of silver.2

The code

src/dataplatform/stages/history.py:197The history columns of a row that has no history of its own.
def _live() -> dict[str, Column]:
    """A row that is current, not deleted, with no version history of its own."""
    return {
        "_valid_from": F.lit(None).cast("timestamp"),
        "_valid_to": F.lit(None).cast("timestamp"),
        "_is_current": F.lit(True),
        "_is_deleted": F.lit(False),
        "_deleted_at": F.lit(None).cast("timestamp"),
        "_seq_from": F.lit(None).cast("string"),
    }
src/dataplatform/stages/history.py:209The whole of append history.
def append(inc: DataFrame, cur: DataFrame | None, plan: SilverPlan) -> DataFrame:
    if plan.delete_apply == "none" and "_op" in inc.columns:  # a delete row is not a new fact (as in scd1 and scd2)
        inc = inc.where(F.coalesce(F.col("_op"), F.lit("U")) != "D")
    has_event = bool(plan.event_time) and plan.event_time in inc.columns
    when = F.col(plan.event_time).cast("timestamp") if has_event else _ts(inc, plan)
    return _finish(inc.withColumns({**_live(), "_valid_from": when, "_row_key": F.col("_key_hash")}), None, plan)

_live returns the column values of a live row: current, not deleted, no validity and no start in the source order. append lays those over the rows and then sets two columns of its own. _valid_from becomes the event time when the plan has one and the column is in the batch. Otherwise it is the effective time, which may be empty. The row key is the key hash.

The function never looks at silver. It does not need to. Stage 10 has already dropped any reading whose key is in silver within the lookback, and the writer appends with a transaction number so that a repeated write is skipped. The writer page explains the transaction number.

How it is tested

tests/dataplatform/test_silver_stages.py:453Three cases, run through all the stages: new rows, a repeat across batches, a repeat inside a batch.
        ("new rows appended", [[("r1", 1, 1, 1)], [("r2", 1, 1, 2)]], ["r1", "r2"]),
        ("a replay across batches is dropped (DUP-04)", [[("r1", 1, 1, 1)], [("r1", 1, 1, 2), ("r2", 1, 1, 3)]],
         ["r1", "r2"]),
        ("a replay within a batch is dropped", [[("r1", 1, 1, 1), ("r1", 1, 1, 2)]], ["r1"]),
    ],
)
def test_history_append(spark, case, batches, want):
    p = plan(dedup={"mode": "business_key_first", "business_keys": ["id"]})
    silver = _apply(spark, p, batches, "id string, v long, seq long, _seq long")
    assert [r[0] for r in rows(silver, "id")] == want, case
tests/dataplatform/test_silver_stages.py:465The worked example above: three readings, two event hours.
def test_late_readings_keep_their_event_time(spark):
    """LATE-03: a reading that arrives late is appended with its own event time, so it counts in its event hour."""
    p = plan(columns=[("id", "string"), ("event_ts", "timestamp")], settings={"watermark_column": "event_ts"},
             dedup={"mode": "business_key_first", "business_keys": ["id"]})
    silver = _apply(spark, p, [[("r1", ts(0)), ("r2", ts(61))], [("r3", ts(1))]], "id string, event_ts timestamp")
    hours = silver.groupBy(F.date_trunc("hour", "_valid_from").alias("h")).count().collect()
    assert sorted(r["count"] for r in hours) == [1, 2]

In the recorded 14-day run of the tyre company at full scale (4 November 2026, local Spark), 188 readings arrived late on day 3 and silver's count per event hour equalled the oracle in all 24 hours. On day 13 the stream was replayed, 19,440 repeated rows were dropped, and silver held 8,087,040 readings, equal to the oracle. Both are quoted from docs/W1C-SILVER.md.

Not built yet, or known gaps
  • Events cannot be deleted or corrected through this path. The function does not read the operation column, so a delete row in an event feed is not treated as a delete.
  • An event dataset without an event time and without an effective time gets an empty _valid_from.