Home › AWS › AWS data services › Step Functions

AWS data services · Orchestration · about 10 min read

AWS Step Functions

A serverless state machine that chains AWS service calls with waits, branches, retries and parallel steps, drawn and run as a visual workflow.

RoleOrchestration
StatusGenerally available. Standard workflows run up to one year; Express workflows up to five minutes.
Verdict on this sitePursue with care
AzureAzure Logic Apps or Durable Functions
Google CloudWorkflows
DatabricksLakeflow Jobs (for Databricks tasks)
Open sourceTemporal, or Apache Airflow for batch DAGs
Verdict: Pursue with care. Use it only if a mover needs several AWS steps, for example start an AppFlow flow, wait, run a Lambda, then call one Databricks job. Good glue for AWS services; do not put Databricks pipeline steps inside it, because Lakeflow Jobs is the orchestrator for everything after landing.

In plain words

What it is

AWS Step Functions runs workflows described as state machines. A state machine is a list of named steps, called states, and the rules for moving between them: do this, then that; if the result says error, go there; wait ten minutes; run these three in parallel. You write the definition in a JSON language called Amazon States Language, or draw it in Workflow Studio, and Step Functions runs it, records every step, and retries failures as you specified. There are no servers and nothing to patch.

Each Task state calls something: a Lambda function, an ECS task, a Glue job, an AppFlow flow, a DMS task, an SNS message, or any of hundreds of AWS API actions through the AWS SDK integrations. An HTTP Task can call an outside API, such as the Databricks Jobs API, with credentials kept in an EventBridge connection. A Map state runs the same steps for every item in a list, and the Distributed Map can fan out to thousands of child runs over files in S3.

A concrete example. A tyre maker gets a weekly dealer-sales extract from a SaaS system through AppFlow. The workflow starts the AppFlow flow, waits until it finishes, runs a small Lambda that checks the file count and writes a manifest, and only then calls the Databricks Jobs API to start the dealer-sales job. If AppFlow fails three times, it posts to an SNS topic that pages the on-call engineer. Everything after the files land is Databricks' job.

Step Functions is a laminated checklist that a tireless clerk follows step by step, phoning each department, waiting for answers and ticking every box in ink so you can see exactly where it stopped.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toEventBridgeTransfer FamilyAppFlowStep FunctionsOrchestrationLambdaDatabricks Lakeflow Jobs (through the Jobs API)SNS

The lit lane is where Step Functions works. Boxes on the left feed it; boxes on the right use what it produces.

Step Functions sits in the orchestration band, on the AWS side of the platform. Lakeflow Jobs is the main orchestrator and runs everything from the landing folder onwards. EventBridge rules and Scheduler cover the simple AWS-side triggers: a file arrived, a DMS task finished, a nightly time. Step Functions is for the middle case, where moving data in needs several AWS steps in order, with waits and error handling.

Typical uses here are a mover sequence (start AppFlow, wait, validate with Lambda, call one Databricks job), a supplier SFTP flow behind Transfer Family (which can start a workflow when a file arrives), or a monthly archive job that copies and checks files. The boundary is firm: Step Functions may start one Databricks job and wait for it, but it does not model bronze, silver and gold steps. Those live in one Lakeflow job, with its retries, lineage and run history in the Databricks system tables.

Each run is recorded with its input, output and every state transition. Standard workflows keep this history for 90 days after a run closes; Express workflows send it to CloudWatch Logs. To join a run to the platform's run log, pass the Databricks job run id back into the workflow output and log it.

How it works

The ideas you need

IdeaWhat it means
State machineThe workflow definition: states and the transitions between them, written in Amazon States Language.
ExecutionOne run of a state machine with one input. Each execution has its own history of every step.
Standard workflowFor long, exactly-once runs: up to one year, full history kept for 90 days, priced per state transition.
Express workflowFor high-volume, short runs: up to five minutes, at-least-once, priced per request and duration, history only in CloudWatch Logs.
Task state and integration patternsA Task calls a service. Request-response returns at once; run-a-job (.sync) waits for the job to finish; wait-for-callback pauses until something sends a task token back.
Retry and CatchPer-state rules: retry these errors with backoff, then catch what is left and go to a failure path.
Map and Distributed MapRun the same steps for each item. Distributed Map reads items from S3 and runs up to 10,000 child executions in parallel.
HTTP TaskCalls an outside HTTPS API using an EventBridge connection for credentials; each call must finish within 60 seconds.
JSONata and variablesA newer query language and workflow variables that make passing data between states simpler than the older JSONPath fields.
RedriveRestart a failed Standard execution from the failed step instead of from the start, within 14 days.

Choosing

When to use it, and when not

Use it when

  • Use it when getting data in needs several AWS steps in a fixed order, such as start a flow, wait, validate, then notify.
  • Use it when you need visible, auditable error handling with retries and a clear failure path for AWS-side work.
  • Use it when a step must wait for a person or an outside system, using the callback pattern with a task token.
  • Use Distributed Map to fan out a one-off job over many S3 files, such as re-checking every file of a supplier archive.
  • Use it to start one Databricks job at the end of an AWS-side sequence and wait for the result.

Avoid it when

  • Avoid it for steps inside Databricks; bronze, silver, gold and quality tasks belong in one Lakeflow job.
  • Avoid it when a single trigger is enough; an EventBridge rule or schedule is simpler and cheaper.
  • Avoid passing data through it; inputs and outputs are capped at 256 KiB, so pass S3 paths, not records.
  • Avoid tight polling loops in Standard workflows; every loop pays for state transitions and fills the 25,000-event history.
  • Avoid it as a general company scheduler for hundreds of batch DAGs across many tools; that is Airflow's or Lakeflow Jobs' ground.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Payload over 256 KiBA state returns a large result and the execution fails with a data limit error.Write large results to S3 and pass the S3 path; use ResultSelector or JSONata to keep only needed fields.
History limit of 25,000 eventsA long polling loop or a big inline Map fills the history and the execution fails.Use .sync integrations instead of polling, Distributed Map for large fan-outs, or start a new execution to continue.
Unbounded retriesRetry with no maximum keeps calling a failing service, costs transitions and hides the outage.Set MaxAttempts and backoff per error type, then Catch to a failure path that alerts.
Waiting on DatabricksA loop that polls the Jobs API every few seconds costs transitions and can time out an HTTP Task.Poll every few minutes with a Wait state, or have the Databricks job send the result back with a callback token.
Throttled downstream servicesA Map with high concurrency floods Lambda, DynamoDB or an outside API, causing throttling errors.Set MaxConcurrency on Map, use retries with backoff and jitter, and check the target's quotas.
State transition throttlingMany Standard executions at once hit the regional state transition rate (800 per second outside the largest regions) and slow down.Use Express for high-volume short work, spread start times, or request a quota increase.
Two orchestrators fightingStep Functions retries a Databricks call while the Lakeflow job also retries, creating duplicate runs.Let Lakeflow own retries for Databricks tasks; make the Step Functions call idempotent with an idempotency token.

Alternatives

What to compare it with

AlternativePick the alternative whenPick Step Functions when
Databricks Lakeflow JobsThe steps run in Databricks; you want one place for triggers, retries, lineage and cost.The steps are AWS services outside Databricks, such as AppFlow, DMS, Lambda and SNS.
Amazon EventBridge Scheduler and rulesYou only need to start one thing on a time or event.You need several steps in order, with waits, branches and error paths.
Amazon MWAAThe company already runs Airflow and has many Python DAGs across many systems.A few AWS-side workflows need no environment, no Python dependencies and pay-per-step pricing.
AWS Lambda durable functionsDevelopers prefer to write the workflow as ordinary code with checkpoints rather than as a state machine.You want a visual workflow, built-in integrations with hundreds of AWS actions and a step-by-step history.
Azure Logic Apps or Durable Functions, Google Cloud WorkflowsThe platform runs on Azure or Google Cloud.The platform runs on AWS.

Cost

How you pay

Standard workflows are charged per state transition, each time a step completes, including retries, after a monthly free tier. Express workflows are charged per request and by duration, rounded to 100 milliseconds, scaled by the memory used. The services a workflow calls, such as Lambda, AppFlow or Databricks, bill separately. For a few movers a day the Step Functions cost is usually small; loops and large Maps are what make it grow. Check the pricing page for your region.

What drives the bill

  • Number of state transitions per Standard execution, times executions per month.
  • Retries and polling loops, which add transitions.
  • Distributed Map child executions, each with its own transitions.
  • Express request count and duration for high-volume short workflows.
  • CloudWatch Logs ingestion if full execution data logging is switched on.

Cost traps

  • Polling a long job every few seconds in a Standard workflow, paying a transition for each check.
  • A Distributed Map over millions of tiny files, each item a paid child step.
  • Logging full input and output to CloudWatch Logs for busy Express workflows.
  • Assuming Express is always cheaper; for long waits it is not available, and for heavy memory use the duration cost grows.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Each state machine has an execution role; grant only the actions its states call, such as appflow:StartFlow on one flow and lambda:InvokeFunction on one function.
  • Keep Databricks service principal credentials in an EventBridge connection, which stores them in Secrets Manager, never in the definition or input.
  • Execution history stores inputs and outputs; encrypt with a customer managed KMS key where policy requires it and keep personal data out of payloads by passing S3 paths.
  • Use an interface VPC endpoint so private workloads start executions without the public internet; HTTP Tasks to Databricks should target the workspace over its private path where your network design allows.
  • CloudTrail records StartExecution and definition changes; deploy state machines from code (CDK or Terraform) so changes are reviewed.

Operating it

What to watch, and the limits

WatchWhy
ExecutionsFailed and ExecutionsTimedOutFailed or timed-out runs per state machine; the core alarms.
ExecutionThrottledState transitions throttled by the regional quota.
ExecutionTimeRun duration; a rising trend shows a slow mover or a slow Databricks job.
ExecutionsAborted and redrivesRuns stopped by hand or restarted; a sign of manual firefighting.
Lambda and AppFlow errors in the same runThe failed state's own service metrics, to find which step broke.
LimitValue and notes
Input or output size256 KiB per task, state or execution; hard quota.
Execution history25,000 events per Standard execution; the execution fails beyond it.
Maximum duration1 year for Standard; 5 minutes for Express.
Distributed MapUp to 10,000 parallel child executions per Map Run; 1,000 open Map Runs per account.
State transitions (Standard)5,000 per second in N. Virginia, Oregon and Ireland; 800 per second elsewhere; adjustable.
HTTP Task duration60 seconds per request; hard quota.

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
Weekly dealer-sales extract via AppFlow, then one Databricks jobStandard workflow: start flow, wait, Lambda check, HTTP Task to the Jobs APIPursue with care: a real multi-step AWS mover; keep it to one Databricks job call.
Start a Databricks job when a supplier file lands in S3EventBridge rule or Lakeflow file-arrival triggerDrop Step Functions here: a single trigger needs no state machine.
Chain bronze, silver and gold stepsOne Lakeflow jobDrop Step Functions for this: Databricks steps belong in Lakeflow Jobs.
Re-validate 2 million archived files onceDistributed Map over the S3 inventory with low concurrencyPursue with care: fast to build; estimate child executions and cost first.

FAQ

Questions people ask

Can Step Functions run a Databricks job?

Yes. An HTTP Task can call the Databricks Jobs API with credentials from an EventBridge connection, and a Wait loop or a callback tells it when the job is done. On this platform, call one whole Lakeflow job rather than its individual steps.

Standard or Express workflow?

Standard for anything longer than five minutes, anything that needs exactly-once runs or a visible history, which covers data movers. Express for high-volume, short event handling.

Step Functions or Airflow?

Step Functions for a few AWS-side workflows with no environment to run. Airflow (MWAA) when the company already runs many Python DAGs across many systems.

What is the payload limit in Step Functions?

256 KiB for any task, state or execution input or output. Pass S3 paths and small identifiers, not data.

Is Step Functions expensive?

For a few movers a day, the cost is usually small next to the services it calls. Cost grows with polling loops, retries and large Map fan-outs, because Standard workflows pay per state transition.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

Data platform on AWS and Databricks · Ingestion on the data platform · Amazon EventBridge · Amazon MWAA · AWS Lambda · All AWS data services

← MWAA · Direct Connect, VPN, Transit Gateway →