Home › Key concepts › Orchestration and DAGs

Key concept · Crawl (beginner) · the same on every platform

Orchestration and DAGs

Run each step in the right order, on time, and retry what fails.

What it is

A DAG (directed acyclic graph) is a list of tasks with arrows showing which must finish before which, and no loops. An orchestrator runs the DAG on a schedule or trigger, retries failures, and alerts you.

Why it matters

Pipelines are many steps with dependencies. Running them by hand or by separate timers breaks the moment one step is late.

An example

Load orders, then load customers, then build silver, then gold; if orders fail, nothing after them runs on stale data.

The common mistake

Chaining jobs by clock time ('gold at 3 a.m. because bronze usually finishes by 2').

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Orchestrationrun the steps in order, on time, and retry the failuresMWAA (Airflow) / Step FunctionsData Factory / Fabric pipelinesCloud Composer (Airflow)Lakeflow JobsAirflow or Dagster

Related ideas

Where it shows up

Part of the Crawl level on these paths: Data Engineering (running data).