Home › Key concepts › Lineage

Key concept · Walk (intermediate) · the same on every platform

Lineage

A map of where each table and number came from, and what depends on it.

What it is

Lineage records which jobs read which tables and wrote which others, down to columns. It answers 'where did this number come from?' and 'what breaks if I change this column?'. OpenLineage is an open standard for collecting it.

Why it matters

It turns an hour of detective work into a click, and it is the basis for impact analysis and audits.

An example

Before dropping a column you see it feeds two gold tables and one ML model.

The common mistake

Drawing lineage by hand in a slide that is out of date the week after.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Lineagetrace a number back to the source rowsAmazon DataZone lineagePurview lineageDataplex lineageUnity Catalog lineageOpenLineage + Marquez
Catalog and accesswho owns which table, who may read it, where it came fromGlue Data Catalog + Lake FormationMicrosoft PurviewDataplexUnity CatalogOpenMetadata

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (trusting data).