Home › Key concepts › Lineage
Key concept · Walk (intermediate) · the same on every platform
Lineage
A map of where each table and number came from, and what depends on it.
What it is
Lineage records which jobs read which tables and wrote which others, down to columns. It answers 'where did this number come from?' and 'what breaks if I change this column?'. OpenLineage is an open standard for collecting it.
See it happen: Problem Lab: count and sum at every hop to find lost rows (simulation), recorded from a real run.
Why it matters
It turns an hour of detective work into a click, and it is the basis for impact analysis and audits.
An example
Before dropping a column you see it feeds two gold tables and one ML model.
The common mistake
Drawing lineage by hand in a slide that is out of date the week after.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Lineagetrace a number back to the source rows | Amazon DataZone lineage | Purview lineage | Dataplex lineage | Unity Catalog lineage | OpenLineage + Marquez |
| Catalog and accesswho owns which table, who may read it, where it came from | Glue Data Catalog + Lake Formation | Microsoft Purview | Dataplex | Unity Catalog | OpenMetadata |
Related ideas
Where it shows up
Part of the Walk level on these paths: Data Engineering (trusting data).