Home › Key concepts › Full vs incremental loads

Key concept · Crawl (beginner) · the same on every platform

Full vs incremental loads

Reload everything every time, or only what is new since last time.

What it is

A full load copies the whole table each run. An incremental load copies only rows changed since the last run, usually tracked by a watermark: the highest 'updated at' time you have already loaded.

Why it matters

Full loads are simple but get slower and dearer as data grows. Incremental loads stay fast but need care with late and changed rows.

An example

The pipeline stores 2026-10-27 09:00 as the watermark and next run asks only for rows updated after it.

The common mistake

Using 'created at' as the watermark, so rows that are later updated are never picked up again.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Ingestioncopy data in from apps, databases and APIsAWS DMS / AppFlowData Factory copy / Fabric mirroringDatastreamLakeflow Connect / Auto Loaderdlt or Airbyte
Orchestrationrun the steps in order, on time, and retry the failuresMWAA (Airflow) / Step FunctionsData Factory / Fabric pipelinesCloud Composer (Airflow)Lakeflow JobsAirflow or Dagster

Related ideas

Where it shows up

Part of the Crawl level on these paths: Data Engineering (moving data).