Data PlatformZoom level 2
Lineage: where a table's rows came from
Lineage is the record of which data fed which table. It answers two questions: where did this number come from, and what breaks downstream if this source changes.
From bronze onward, the engine's own catalog records lineage as jobs read and write tables, and the platform adopts that screen. The catalog cannot see the hops before bronze: source system, mover, landing, bronze. Those exist only in the platform's file log. This part turns the file log into lineage events for that first hop.
What is recorded, and at what grain
| Question | Answer |
|---|---|
| What is one lineage record? | One event for one batch, which is one record of the file log. |
| What does it connect? | One source dataset to one bronze table. |
| Is there column lineage? | No. The grain is dataset to table. Columns are not traced by this code. |
| Which batches produce an event? | A loaded batch produces a COMPLETE event with an output. A batch rejected for breaking its contract, rejected for volume, or held as a mass delete produces a FAIL event with no output, because nothing reached bronze. |
| Which produce none? | A duplicate that the lander skipped. Nothing moved. |
| What shape is the event? | The run event of OpenLineage, an open standard for lineage events. |
| What ties it to the rest? | The run id of the run that handled the batch. |
A worked example
Three deliveries of order lines are in the file log. The first column is the file, the second its status, and the third the event that comes out.
| File | Status in the file log | Event |
|---|---|---|
sol_2026-11-09.parquet | loaded | COMPLETE, input erp_db / sales_order_line, output dev_bronze.erp.sales_order_line |
sol_2026-11-09_resent.parquet | skipped_duplicate | None |
sol_2026-11-10.parquet | rejected_contract | FAIL, the same input, no output |
The code
src/dataplatform/obs/lineage.py:24The whole of it: one loop over file log records.def events_from_files(files: Iterable[FileRecord], system_of: dict[str, str], env: str, at: str,
bronze_namespace: str = "lakehouse") -> list[dict[str, Any]]:
"""Run events for the batches in `files`. `system_of` maps a source name to its system (erp -> erp_db), which
names the input dataset's namespace; `at` is the event time (ISO, from the clock port). Duplicates skipped by the
lander produce no event: nothing moved."""
out = []
for f in files:
if f.status not in LOADED + HELD:
continue
loaded = f.status in LOADED
event: dict[str, Any] = {
"eventType": "COMPLETE" if loaded else "FAIL",
"eventTime": at,
"producer": PRODUCER,
"schemaURL": SCHEMA_URL,
"run": {"runId": f.run_id, "facets": {"batch": {"batch_key": f.batch_key, "window": f.window,
"file": f.file_name, "rows": f.rows,
"declared_rows": f.declared_rows,
"status": f.status, "reason": f.reason}}},
"job": {"namespace": f"dataplatform.{env}", "name": f"land.{f.source}.{f.dataset}"},
"inputs": [{"namespace": system_of.get(f.source, f.source), "name": f.dataset,
"facets": {"delivery": {"file": f.file_name, "checksum": f.checksum}}}],
"outputs": [{"namespace": bronze_namespace, "name": f"{env}_bronze.{f.source}.{f.dataset}"}] if loaded
else [],
}
out.append(event)
return outWalk through it. A record whose status is neither loaded nor one of the three held statuses is skipped. The event's run carries the batch's own facts as a facet, which is OpenLineage's word for an extra block of detail: the batch key, the window, the file, the rows counted and declared, the status and the reason. The job is named after the landing step for the dataset. The input's namespace is the source system, found through the map, so erp becomes erp_db. The output is the bronze table's name for the environment, and it is present only for a loaded batch.
Lineage in the operations tables
There is a second, separate source. The platform's own telemetry, copied once per business day, includes table-to-table edges with the run that wrote them. The operations builder turns those into the table ops_lineage with four columns: day, run id, source table and target table. No screen of ours reads it now, because browsing lineage is the platform's own screen.
src/dataplatform/obs/ops.py:281The table-to-table edges from the platform's telemetry.def lineage_table(system: SystemSnapshot, tz: str) -> list[Row]:
return [{"day": business_day(x.at, tz), "run_id": x.run_id, "source_table": x.source_table,
"target_table": x.target_table} for x in system.lineage]The settings that drive it
No dataset setting is read. The caller passes the environment, the name of the bronze namespace, which defaults to lakehouse, and the map from source to system.
How it is tested
tests/dataplatform/test_obs_lineage.py has three cases in a table and one more test. Each case gives a file status and expects an event type and a number of outputs.
tests/dataplatform/test_obs_lineage.py:16The three cases. A fourth test checks that a skipped duplicate produces no event.CASES = [
("a loaded batch: complete, source in, bronze out", "loaded", "COMPLETE", 1),
("a held batch: fail, nothing out", "held_mass_delete", "FAIL", 0),
("a rejected batch: fail, nothing out", "rejected_contract", "FAIL", 0),
]- Nothing calls this function outside its test. No emitter adapter sends the events to a lineage service, and the local run does not write them anywhere.
- The event's run id is the file record's run id as stored, such as
ingest-2026-11-09. OpenLineage expects a UUID there. The conversion helper exists on the run recorder, which the ingestion runner does not use. - There is no column-level lineage from this code.
- The mover and the landing area do not appear as separate steps. The event goes from the source dataset straight to the bronze table.
- The table
ops_lineageis still built although no screen reads it. The code comments call removing it a follow-up.