Home › Key concepts › Orchestration and DAGs
Key concept · Crawl (beginner) · the same on every platform
Orchestration and DAGs
Run each step in the right order, on time, and retry what fails.
What it is
A DAG (directed acyclic graph) is a list of tasks with arrows showing which must finish before which, and no loops. An orchestrator runs the DAG on a schedule or trigger, retries failures, and alerts you.
Why it matters
Pipelines are many steps with dependencies. Running them by hand or by separate timers breaks the moment one step is late.
An example
Load orders, then load customers, then build silver, then gold; if orders fail, nothing after them runs on stale data.
The common mistake
Chaining jobs by clock time ('gold at 3 a.m. because bronze usually finishes by 2').
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Orchestrationrun the steps in order, on time, and retry the failures | MWAA (Airflow) / Step Functions | Data Factory / Fabric pipelines | Cloud Composer (Airflow) | Lakeflow Jobs | Airflow or Dagster |
Related ideas
Where it shows up
Part of the Crawl level on these paths: Data Engineering (running data).