Home › Key concepts › Backfills and reprocessing
Key concept · Walk (intermediate) · the same on every platform
Backfills and reprocessing
Re-run a pipeline for past dates, safely.
What it is
A backfill runs a pipeline for a range of past dates, to load history or to fix data after a bug. It only works well when each run is idempotent and parameterised by date.
See it happen: Problem Lab: a backfill after a bug, re-runs and late data (simulation), recorded from a real run.
Why it matters
Bugs are found late. Being able to rebuild last month in one command turns a crisis into a chore.
An example
A currency bug is fixed and the last 30 days of silver and gold are rebuilt overnight.
The common mistake
Pipelines that read 'today' from the clock, so they cannot be run for any other day.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Orchestrationrun the steps in order, on time, and retry the failures | MWAA (Airflow) / Step Functions | Data Factory / Fabric pipelines | Cloud Composer (Airflow) | Lakeflow Jobs | Airflow or Dagster |
| Batch computeprocess a lot of data at once | AWS Glue / EMR (Spark) | Fabric Spark | Dataproc (Spark) | Databricks jobs (Spark) | PySpark or Polars |
Related ideas
Where it shows up
Part of the Walk level on these paths: Data Engineering (running data).