Home › Key concepts › Full vs incremental loads
Key concept · Crawl (beginner) · the same on every platform
Full vs incremental loads
Reload everything every time, or only what is new since last time.
What it is
A full load copies the whole table each run. An incremental load copies only rows changed since the last run, usually tracked by a watermark: the highest 'updated at' time you have already loaded.
Why it matters
Full loads are simple but get slower and dearer as data grows. Incremental loads stay fast but need care with late and changed rows.
An example
The pipeline stores 2026-10-27 09:00 as the watermark and next run asks only for rows updated after it.
The common mistake
Using 'created at' as the watermark, so rows that are later updated are never picked up again.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Ingestioncopy data in from apps, databases and APIs | AWS DMS / AppFlow | Data Factory copy / Fabric mirroring | Datastream | Lakeflow Connect / Auto Loader | dlt or Airbyte |
| Orchestrationrun the steps in order, on time, and retry the failures | MWAA (Airflow) / Step Functions | Data Factory / Fabric pipelines | Cloud Composer (Airflow) | Lakeflow Jobs | Airflow or Dagster |
Related ideas
Where it shows up
Part of the Crawl level on these paths: Data Engineering (moving data).