Home › AWS › AWS data services › Step Functions
AWS data services · Orchestration · about 10 min read
AWS Step Functions
A serverless state machine that chains AWS service calls with waits, branches, retries and parallel steps, drawn and run as a visual workflow.
In plain words
What it is
AWS Step Functions runs workflows described as state machines. A state machine is a list of named steps, called states, and the rules for moving between them: do this, then that; if the result says error, go there; wait ten minutes; run these three in parallel. You write the definition in a JSON language called Amazon States Language, or draw it in Workflow Studio, and Step Functions runs it, records every step, and retries failures as you specified. There are no servers and nothing to patch.
Each Task state calls something: a Lambda function, an ECS task, a Glue job, an AppFlow flow, a DMS task, an SNS message, or any of hundreds of AWS API actions through the AWS SDK integrations. An HTTP Task can call an outside API, such as the Databricks Jobs API, with credentials kept in an EventBridge connection. A Map state runs the same steps for every item in a list, and the Distributed Map can fan out to thousands of child runs over files in S3.
A concrete example. A tyre maker gets a weekly dealer-sales extract from a SaaS system through AppFlow. The workflow starts the AppFlow flow, waits until it finishes, runs a small Lambda that checks the file count and writes a manifest, and only then calls the Databricks Jobs API to start the dealer-sales job. If AppFlow fails three times, it posts to an SNS topic that pages the on-call engineer. Everything after the files land is Databricks' job.
Step Functions is a laminated checklist that a tireless clerk follows step by step, phoning each department, waiting for answers and ticking every box in ink so you can see exactly where it stopped.
Big picture
Where it fits
The lit lane is where Step Functions works. Boxes on the left feed it; boxes on the right use what it produces.
Step Functions sits in the orchestration band, on the AWS side of the platform. Lakeflow Jobs is the main orchestrator and runs everything from the landing folder onwards. EventBridge rules and Scheduler cover the simple AWS-side triggers: a file arrived, a DMS task finished, a nightly time. Step Functions is for the middle case, where moving data in needs several AWS steps in order, with waits and error handling.
Typical uses here are a mover sequence (start AppFlow, wait, validate with Lambda, call one Databricks job), a supplier SFTP flow behind Transfer Family (which can start a workflow when a file arrives), or a monthly archive job that copies and checks files. The boundary is firm: Step Functions may start one Databricks job and wait for it, but it does not model bronze, silver and gold steps. Those live in one Lakeflow job, with its retries, lineage and run history in the Databricks system tables.
Each run is recorded with its input, output and every state transition. Standard workflows keep this history for 90 days after a run closes; Express workflows send it to CloudWatch Logs. To join a run to the platform's run log, pass the Databricks job run id back into the workflow output and log it.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| State machine | The workflow definition: states and the transitions between them, written in Amazon States Language. |
| Execution | One run of a state machine with one input. Each execution has its own history of every step. |
| Standard workflow | For long, exactly-once runs: up to one year, full history kept for 90 days, priced per state transition. |
| Express workflow | For high-volume, short runs: up to five minutes, at-least-once, priced per request and duration, history only in CloudWatch Logs. |
| Task state and integration patterns | A Task calls a service. Request-response returns at once; run-a-job (.sync) waits for the job to finish; wait-for-callback pauses until something sends a task token back. |
| Retry and Catch | Per-state rules: retry these errors with backoff, then catch what is left and go to a failure path. |
| Map and Distributed Map | Run the same steps for each item. Distributed Map reads items from S3 and runs up to 10,000 child executions in parallel. |
| HTTP Task | Calls an outside HTTPS API using an EventBridge connection for credentials; each call must finish within 60 seconds. |
| JSONata and variables | A newer query language and workflow variables that make passing data between states simpler than the older JSONPath fields. |
| Redrive | Restart a failed Standard execution from the failed step instead of from the start, within 14 days. |
Choosing
When to use it, and when not
Use it when
- Use it when getting data in needs several AWS steps in a fixed order, such as start a flow, wait, validate, then notify.
- Use it when you need visible, auditable error handling with retries and a clear failure path for AWS-side work.
- Use it when a step must wait for a person or an outside system, using the callback pattern with a task token.
- Use Distributed Map to fan out a one-off job over many S3 files, such as re-checking every file of a supplier archive.
- Use it to start one Databricks job at the end of an AWS-side sequence and wait for the result.
Avoid it when
- Avoid it for steps inside Databricks; bronze, silver, gold and quality tasks belong in one Lakeflow job.
- Avoid it when a single trigger is enough; an EventBridge rule or schedule is simpler and cheaper.
- Avoid passing data through it; inputs and outputs are capped at 256 KiB, so pass S3 paths, not records.
- Avoid tight polling loops in Standard workflows; every loop pays for state transitions and fills the 25,000-event history.
- Avoid it as a general company scheduler for hundreds of batch DAGs across many tools; that is Airflow's or Lakeflow Jobs' ground.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Payload over 256 KiB | A state returns a large result and the execution fails with a data limit error. | Write large results to S3 and pass the S3 path; use ResultSelector or JSONata to keep only needed fields. |
| History limit of 25,000 events | A long polling loop or a big inline Map fills the history and the execution fails. | Use .sync integrations instead of polling, Distributed Map for large fan-outs, or start a new execution to continue. |
| Unbounded retries | Retry with no maximum keeps calling a failing service, costs transitions and hides the outage. | Set MaxAttempts and backoff per error type, then Catch to a failure path that alerts. |
| Waiting on Databricks | A loop that polls the Jobs API every few seconds costs transitions and can time out an HTTP Task. | Poll every few minutes with a Wait state, or have the Databricks job send the result back with a callback token. |
| Throttled downstream services | A Map with high concurrency floods Lambda, DynamoDB or an outside API, causing throttling errors. | Set MaxConcurrency on Map, use retries with backoff and jitter, and check the target's quotas. |
| State transition throttling | Many Standard executions at once hit the regional state transition rate (800 per second outside the largest regions) and slow down. | Use Express for high-volume short work, spread start times, or request a quota increase. |
| Two orchestrators fighting | Step Functions retries a Databricks call while the Lakeflow job also retries, creating duplicate runs. | Let Lakeflow own retries for Databricks tasks; make the Step Functions call idempotent with an idempotency token. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick Step Functions when |
|---|---|---|
| Databricks Lakeflow Jobs | The steps run in Databricks; you want one place for triggers, retries, lineage and cost. | The steps are AWS services outside Databricks, such as AppFlow, DMS, Lambda and SNS. |
| Amazon EventBridge Scheduler and rules | You only need to start one thing on a time or event. | You need several steps in order, with waits, branches and error paths. |
| Amazon MWAA | The company already runs Airflow and has many Python DAGs across many systems. | A few AWS-side workflows need no environment, no Python dependencies and pay-per-step pricing. |
| AWS Lambda durable functions | Developers prefer to write the workflow as ordinary code with checkpoints rather than as a state machine. | You want a visual workflow, built-in integrations with hundreds of AWS actions and a step-by-step history. |
| Azure Logic Apps or Durable Functions, Google Cloud Workflows | The platform runs on Azure or Google Cloud. | The platform runs on AWS. |
Cost
How you pay
Standard workflows are charged per state transition, each time a step completes, including retries, after a monthly free tier. Express workflows are charged per request and by duration, rounded to 100 milliseconds, scaled by the memory used. The services a workflow calls, such as Lambda, AppFlow or Databricks, bill separately. For a few movers a day the Step Functions cost is usually small; loops and large Maps are what make it grow. Check the pricing page for your region.
What drives the bill
- Number of state transitions per Standard execution, times executions per month.
- Retries and polling loops, which add transitions.
- Distributed Map child executions, each with its own transitions.
- Express request count and duration for high-volume short workflows.
- CloudWatch Logs ingestion if full execution data logging is switched on.
Cost traps
- Polling a long job every few seconds in a Standard workflow, paying a transition for each check.
- A Distributed Map over millions of tiny files, each item a paid child step.
- Logging full input and output to CloudWatch Logs for busy Express workflows.
- Assuming Express is always cheaper; for long waits it is not available, and for heavy memory use the duration cost grows.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Each state machine has an execution role; grant only the actions its states call, such as appflow:StartFlow on one flow and lambda:InvokeFunction on one function.
- Keep Databricks service principal credentials in an EventBridge connection, which stores them in Secrets Manager, never in the definition or input.
- Execution history stores inputs and outputs; encrypt with a customer managed KMS key where policy requires it and keep personal data out of payloads by passing S3 paths.
- Use an interface VPC endpoint so private workloads start executions without the public internet; HTTP Tasks to Databricks should target the workspace over its private path where your network design allows.
- CloudTrail records StartExecution and definition changes; deploy state machines from code (CDK or Terraform) so changes are reviewed.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| ExecutionsFailed and ExecutionsTimedOut | Failed or timed-out runs per state machine; the core alarms. |
| ExecutionThrottled | State transitions throttled by the regional quota. |
| ExecutionTime | Run duration; a rising trend shows a slow mover or a slow Databricks job. |
| ExecutionsAborted and redrives | Runs stopped by hand or restarted; a sign of manual firefighting. |
| Lambda and AppFlow errors in the same run | The failed state's own service metrics, to find which step broke. |
| Limit | Value and notes |
|---|---|
| Input or output size | 256 KiB per task, state or execution; hard quota. |
| Execution history | 25,000 events per Standard execution; the execution fails beyond it. |
| Maximum duration | 1 year for Standard; 5 minutes for Express. |
| Distributed Map | Up to 10,000 parallel child executions per Map Run; 1,000 open Map Runs per account. |
| State transitions (Standard) | 5,000 per second in N. Virginia, Oregon and Ireland; 800 per second elsewhere; adjustable. |
| HTTP Task duration | 60 seconds per request; hard quota. |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Weekly dealer-sales extract via AppFlow, then one Databricks job | Standard workflow: start flow, wait, Lambda check, HTTP Task to the Jobs API | Pursue with care: a real multi-step AWS mover; keep it to one Databricks job call. |
| Start a Databricks job when a supplier file lands in S3 | EventBridge rule or Lakeflow file-arrival trigger | Drop Step Functions here: a single trigger needs no state machine. |
| Chain bronze, silver and gold steps | One Lakeflow job | Drop Step Functions for this: Databricks steps belong in Lakeflow Jobs. |
| Re-validate 2 million archived files once | Distributed Map over the S3 inventory with low concurrency | Pursue with care: fast to build; estimate child executions and cost first. |
FAQ
Questions people ask
Can Step Functions run a Databricks job?
Yes. An HTTP Task can call the Databricks Jobs API with credentials from an EventBridge connection, and a Wait loop or a callback tells it when the job is done. On this platform, call one whole Lakeflow job rather than its individual steps.
Standard or Express workflow?
Standard for anything longer than five minutes, anything that needs exactly-once runs or a visible history, which covers data movers. Express for high-volume, short event handling.
Step Functions or Airflow?
Step Functions for a few AWS-side workflows with no environment to run. Airflow (MWAA) when the company already runs many Python DAGs across many systems.
What is the payload limit in Step Functions?
256 KiB for any task, state or execution input or output. Pass S3 paths and small identifiers, not data.
Is Step Functions expensive?
For a few movers a day, the cost is usually small next to the services it calls. Cost grows with polling loops, retries and large Map fan-outs, because Standard workflows pay per state transition.
Sources
Checked against
- AWS docs: Step Functions service quotas
- AWS docs: Choosing workflow type in Step Functions
- AWS docs: Call HTTPS APIs in Step Functions workflows
- AWS: Step Functions pricing
Checked 2026-11-11.
Related
Where it connects
Data platform on AWS and Databricks · Ingestion on the data platform · Amazon EventBridge · Amazon MWAA · AWS Lambda · All AWS data services