Home › Key concepts › Backfills and reprocessing

Key concept · Walk (intermediate) · the same on every platform

Backfills and reprocessing

Re-run a pipeline for past dates, safely.

What it is

A backfill runs a pipeline for a range of past dates, to load history or to fix data after a bug. It only works well when each run is idempotent and parameterised by date.

Why it matters

Bugs are found late. Being able to rebuild last month in one command turns a crisis into a chore.

An example

A currency bug is fixed and the last 30 days of silver and gold are rebuilt overnight.

The common mistake

Pipelines that read 'today' from the clock, so they cannot be run for any other day.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Orchestrationrun the steps in order, on time, and retry the failuresMWAA (Airflow) / Step FunctionsData Factory / Fabric pipelinesCloud Composer (Airflow)Lakeflow JobsAirflow or Dagster
Batch computeprocess a lot of data at onceAWS Glue / EMR (Spark)Fabric SparkDataproc (Spark)Databricks jobs (Spark)PySpark or Polars

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (running data).