Home › Key concepts › Schema evolution

Key concept · Walk (intermediate) · the same on every platform

Schema evolution

How a table's columns can change over time without breaking readers.

What it is

Adding a nullable column is usually safe. Renaming, removing or changing a column's type usually breaks someone. Table formats can allow safe changes automatically and block unsafe ones.

Why it matters

Sources change all the time. Deciding what is allowed, up front, keeps pipelines running.

An example

A new 'discount_code' column appears in the source and is added to bronze automatically; a type change from number to text is blocked and alerts the owner.

The common mistake

Letting every schema change through silently, and finding the broken report a month later.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Open table formatfiles that behave like a database table, with historyApache Iceberg (S3 Tables)Delta Lake in OneLakeIceberg on BigLakeDelta LakeDelta Lake or Iceberg

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (trusting data).