Home › AWS › AWS data services › MWAA

AWS data services · Orchestration · about 10 min read

Amazon Managed Workflows for Apache Airflow (MWAA)

Apache Airflow run by AWS: Python workflows that schedule and chain tasks across many systems, as always-on environments or the newer Serverless option.

RoleOrchestration
StatusGenerally available. Airflow 3.3.1 (since 1 Sep 2026), 3.2.1, 3.0.6 and 2.11 are supported; 2.4.3, 2.5.1 and 2.6.3 reached end of support on 30 Dec 2025. MWAA Serverless (Airflow 3, YAML workflows) launched Nov 2025.
Verdict on this siteKeep if you have it
AzureAzure Data Factory workflow orchestration manager (managed Airflow), Fabric Apache Airflow jobs
Google CloudCloud Composer
DatabricksLakeflow Jobs
Open sourceApache Airflow (also Dagster, Prefect)
Verdict: Keep if you have it. Only if the company already runs Airflow. It is a strong tool, but a second orchestrator to operate; on this platform Lakeflow Jobs runs everything after the landing folder, so MWAA at most triggers Databricks jobs and keeps the company's other, non-Databricks workflows.

In plain words

What it is

An orchestrator decides what runs, in which order, and when. Apache Airflow is the best-known open-source orchestrator. You describe a workflow as a DAG (a directed acyclic graph: a set of tasks with arrows saying which must finish before which) in a Python file. A scheduler reads the DAG files, starts tasks on time, waits for their upstream tasks, retries failures and records every run in a metadata database. A web interface shows which runs passed and which failed.

Running Airflow yourself means looking after a scheduler, workers, a web server, a database and a queue, and upgrading all of it. Amazon MWAA does that for you. You put DAG files, a requirements file and plugins in an S3 bucket, choose an environment size, and AWS runs the Airflow parts inside your VPC. You still own the DAG code, the Python dependencies and the version upgrades. Since November 2025 there is also MWAA Serverless: no environment at all, workflows written in YAML, each task getting its own short-lived compute, billed only while tasks run.

A concrete example. A retail group already runs 300 Airflow DAGs: nightly loads from its point-of-sale system, refreshes of a marketing tool, exports to suppliers. When it adopts Databricks, one DAG can wait for the SAP export to land, then call the Databricks Jobs API to run the order pipeline, and finally refresh a supplier report outside Databricks. Inside Databricks, the order pipeline's own steps (bronze, silver, gold, quality checks) stay as one Lakeflow job.

MWAA is a hired station master for a big railway junction that serves many lines; useful if you run many lines, but if all your trains belong to one company with its own dispatcher, you are paying for a second station master.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toDAG files and requirements in S3EventBridgeSecrets ManagerMWAAOrchestrationDatabricks Lakeflow Jobs (through the Jobs API)Glue ETL jobsCloudWatch

The lit lane is where MWAA works. Boxes on the left feed it; boxes on the right use what it produces.

Orchestration runs across every lane. On this platform the main orchestrator is Lakeflow Jobs: everything after the landing folder runs in Databricks, so the orchestrator runs there too, with file-arrival and table-update triggers, retries, alerts and run history in system tables. On the AWS side, EventBridge rules and Scheduler start movers, and Step Functions is used only if a mover needs several AWS steps.

MWAA appears only when a company already runs Airflow for many systems beyond Databricks. Then it sits above Lakeflow Jobs as an outer layer: an Airflow task triggers a Databricks job by its id and waits for the result, and Databricks owns everything inside that job. Re-creating each Databricks task as its own Airflow task is the pattern to avoid, because it splits retries, lineage and run history across two tools.

MWAA reads DAGs and dependencies from S3, writes task logs to CloudWatch Logs, and reaches Databricks through the Airflow Databricks provider using a service principal token kept in Secrets Manager. Its run history lives in the Airflow metadata database, not in the Databricks system tables, so joining it to cost and quality data needs the Databricks job run id passed between the two.

How it works

The ideas you need

IdeaWhat it means
DAGA workflow written in Python: tasks plus the order between them. Airflow re-reads every DAG file often, so slow top-level code slows the whole environment.
Task and operatorA task is one step. An operator is the kind of step, for example DatabricksRunNowOperator to start a Databricks job or S3KeySensor to wait for a file.
SchedulerThe process that decides which tasks are ready to run. MWAA runs two by default and up to five.
WorkerA machine that runs tasks. MWAA scales workers between a minimum and maximum you set, up to 25 by default.
Environment classThe size of an MWAA environment, from micro to 2XL. It sets scheduler, worker and web server capacity and the hourly price.
Metadata databaseThe database where Airflow stores runs, task states and connections. MWAA manages it; you cannot query it directly with SQL.
Sensor and deferrable operatorA sensor waits for something, such as a file. A deferrable operator hands the waiting to a light process (the triggerer) so it does not hold a worker slot.
MWAA ServerlessA deployment option with no environment: YAML workflows in the DAG factory format, each task on isolated compute under the workflow's own IAM role, scheduled by EventBridge Scheduler, with no Airflow web interface.
Provider packageA Python package that adds operators for one system, such as apache-airflow-providers-databricks. Versions must match the constraints file for the Airflow version.

Choosing

When to use it, and when not

Use it when

  • Use MWAA when the company already runs Airflow DAGs and has people who maintain them; adding Databricks jobs as tasks is cheaper than moving every workflow.
  • Use it when one workflow must span many systems beyond Databricks, such as a SaaS refresh, a file export to suppliers and a database job, in one place.
  • Use it when you want Airflow without running its scheduler, database and upgrades yourself.
  • Use MWAA Serverless for a few sporadic AWS-native workflows where an always-on environment would sit idle most of the day.
  • Use it when the same Airflow DAGs must run on several clouds, with Cloud Composer or self-managed Airflow elsewhere.

Avoid it when

  • Avoid it as the orchestrator for a new platform where everything after landing runs in Databricks; Lakeflow Jobs does that with no second tool.
  • Avoid modelling each Databricks notebook or pipeline step as its own Airflow task; trigger whole Databricks jobs instead.
  • Avoid it for event-driven reactions that need sub-minute response; EventBridge or Lakeflow file-arrival triggers react faster and cost less.
  • Avoid it when nobody on the team knows Airflow; dependency pinning, upgrades and DAG performance need that skill.
  • Avoid an always-on environment for a dozen DAGs that run once a night; the hourly environment price dominates.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Dependency conflictsA new provider or library in requirements.txt breaks installation; the environment update fails or tasks fail with import errors.Pin versions with the Airflow constraints file, test with the MWAA local runner before deploying, and change one dependency at a time.
Slow DAG parsingHeavy top-level code or hundreds of DAG files make the scheduler slow; tasks start late and the queue grows.Keep top-level code light, move work into tasks, and watch parse time per file.
Two orchestrators, two historiesFailures are retried in Airflow and in Databricks, run history is split, and nobody knows which tool owns a rerun.Airflow triggers whole Databricks jobs and waits; retries for Databricks tasks live in the Lakeflow job; pass the job run id back into Airflow logs.
Upgrades and end of supportVersions reach end of support (2.4.3, 2.5.1 and 2.6.3 ended on 30 Dec 2025); environments on them get no support, and Airflow 2 to 3 is not an in-place upgrade.Plan a yearly upgrade; move from Airflow 2 to 3 by creating a new environment and migrating DAGs, using the MWAA migration guide.
Sensors holding worker slotsMany sensors waiting for files fill all worker slots, so real tasks queue.Use deferrable operators or reschedule mode, or let EventBridge start the DAG when the file lands.
Worker scaling lagA burst of tasks at midnight waits while extra workers start; small environments hit the maximum workers.Set minimum workers for known peaks, spread start times, and size the environment class for scheduler load.
Secrets in Airflow variablesDatabricks tokens or database passwords stored as plain Airflow variables are visible to anyone with UI access.Use the Secrets Manager backend for connections and variables, and service principals with short-lived tokens.

Alternatives

What to compare it with

AlternativePick the alternative whenPick MWAA when
Databricks Lakeflow JobsAll the work is in Databricks; you want triggers, retries, lineage and cost in one place with no second tool. This is this site's main orchestrator.The company already runs Airflow across many non-Databricks systems.
AWS Step FunctionsA few AWS-only steps need a visual state machine with built-in service calls and no Python environment.Workflows are many, written in Python, and span non-AWS systems.
Amazon EventBridge Scheduler and rulesYou only need to start something on a time or an event, such as a file landing.You need dependencies between many tasks, backfills and a run history.
Azure Data Factory managed Airflow or Fabric Airflow jobsThe platform runs on Azure.The platform runs on AWS.
Google Cloud ComposerThe platform runs on Google Cloud.The platform runs on AWS.
Self-managed Airflow on EKSYou need custom executors, plugins or versions MWAA does not offer, and have a team to run it.You want AWS to run the scheduler, database and patches.

Cost

How you pay

Provisioned MWAA charges an hourly price for the environment by class (micro, small, medium, large, extra large, 2XL), which covers the schedulers, a base worker and the web server, plus hourly prices for extra workers, extra schedulers and extra web servers, and a monthly price per gigabyte of metadata database storage. The environment bills every hour it exists, busy or idle. MWAA Serverless instead charges per task for its run time, with a one-minute minimum and per-second billing after that. Check the MWAA pricing page for your region.

What drives the bill

  • Environment class and how many hours it exists; dev and test environments bill around the clock too.
  • Extra workers and schedulers during peaks.
  • Number of environments; teams often run one per team and per stage.
  • For Serverless, the number of tasks and their duration, with the one-minute minimum per task.
  • The compute the DAGs start elsewhere, such as Databricks jobs, which is billed by those services.

Cost traps

  • A dev environment left running all month for a few test runs.
  • Sensors that poll for hours, keeping workers scaled up.
  • Many tiny tasks on Serverless, each paying the one-minute minimum.
  • Duplicated orchestration: paying for MWAA to watch Databricks jobs that Lakeflow Jobs could schedule on its own.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • The environment runs in your VPC in private subnets; choose private web server access so the Airflow UI is reachable only through your network, and use VPC endpoints for S3, CloudWatch Logs, SQS, KMS and Secrets Manager.
  • The execution role is shared by every DAG in a provisioned environment, so keep it narrow; MWAA Serverless gives each workflow its own role, which is a cleaner fit for least privilege.
  • Store Databricks service principal credentials and database passwords in Secrets Manager through the secrets backend, never in DAG code or plain variables.
  • Encrypt the DAG bucket, logs and metadata with a customer managed KMS key where policy requires it; the DAG bucket must block public access.
  • Airflow UI access is controlled through IAM and mapped to Airflow roles (Admin, Op, User, Viewer); CloudTrail records MWAA API calls.
  • Task logs can echo data; keep personal data out of log lines, especially from SQL or API responses.

Operating it

What to watch, and the limits

WatchWhy
SchedulerHeartbeatShows the scheduler is alive; a gap means nothing is being started.
QueuedTasks and RunningTasksTasks waiting versus running; a growing queue means too few workers or slots.
TotalParseTime (DAG processing)How long the scheduler takes to read all DAGs; rising time delays every run.
TaskInstanceFailures and DAG run durationFailures and late runs per DAG, the signals for alerts.
ApproximateAgeOfOldestTaskHow long the oldest queued task has waited; a direct measure of starvation.
Databricks job run results in system.lakeflow tablesThe truth for the Databricks side, joined by job run id to the Airflow task.
LimitValue and notes
Environments per account per region10 by default; adjustable.
Workers per environment25 by default; adjustable.
Schedulers2 by default, maximum 5, for Airflow 2 and later.
Web servers per environment5 by default; adjustable.
Supported Airflow versions3.3.1, 3.2.1, 3.0.6, 2.11.2, 2.11.0, 2.10.3, 2.10.1, 2.9.2, 2.8.1 and 2.7.2; minor-version upgrades in place only.

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
New platform, all processing in Databricks, no Airflow todayLakeflow Jobs plus EventBridge for AWS-side triggersDrop MWAA: a second orchestrator with nothing to orchestrate that Lakeflow Jobs cannot.
Retail group with 300 Airflow DAGs adopting DatabricksKeep MWAA; each DAG triggers whole Databricks jobs and waitsKeep if you have it: moving every DAG costs more than it saves; Databricks tasks stay inside Lakeflow jobs.
Five nightly AWS-only workflows in a small teamMWAA Serverless or Step FunctionsPursue with care: pay only while tasks run; check operator support and the lack of an Airflow UI first.
Environment still on Airflow 2.6.3Create a new environment on a supported version and migratePursue: 2.6.3 has been out of support since 30 Dec 2025.

FAQ

Questions people ask

Do I need Airflow if I use Databricks?

Usually not. Lakeflow Jobs schedules and chains Databricks tasks with triggers, retries and alerts. Keep Airflow if the company already uses it for many other systems, and let it trigger whole Databricks jobs.

What is MWAA Serverless?

A deployment option launched in November 2025 with no Airflow environment to run. Workflows are YAML files, each task runs on its own compute under the workflow's IAM role, and you pay only while tasks run. There is no Airflow web interface.

Which Airflow versions does MWAA support?

As of November 2026, Airflow 3.3.1, 3.2.1 and 3.0.6, and 2.x versions from 2.7.2 to 2.11.2. Versions 2.4.3, 2.5.1 and 2.6.3 reached end of support on 30 December 2025.

How does MWAA start a Databricks job?

With the Databricks provider's DatabricksRunNowOperator, using a connection that holds a service principal credential from Secrets Manager. The operator starts the job by id and waits for it to finish.

Is MWAA expensive?

A provisioned environment bills every hour it exists, even when idle, plus extra workers. For a few nightly workflows that is often more than the work itself; Serverless or Step Functions can be cheaper.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

Data platform on AWS and Databricks · AWS Step Functions · Amazon EventBridge · AWS Glue Workflows · Medallion architecture · All AWS data services

← Glue Workflows · Step Functions →