Home › AWS › AWS data services › Lambda
AWS data services · Processing and compute · about 12 min read
AWS Lambda
Runs a small piece of your code when something happens, for up to 15 minutes, and bills only while it runs.
In plain words
What it is
AWS Lambda runs a function, a short piece of your code, when an event happens. The event can be a file arriving in an S3 bucket, a message on a queue, a record on a Kinesis stream, a schedule, or an HTTP call. You upload the code, say how much memory it needs, and AWS finds a machine, starts your code, runs it and throws the machine away again. You never see a server, and you pay for the milliseconds the code runs, not for idle time.
Think of a tyre or food manufacturer whose suppliers drop price files over SFTP. Each file lands in an S3 landing bucket. A Lambda function wakes up on the S3 event, checks that the file name matches the agreed pattern, writes one row into a file log, and calls the Databricks Jobs API to start the lander job. The whole function runs for less than a second. Nothing sits waiting between files, and nothing costs money at night when no files arrive.
The limits shape what Lambda is good for. One run lasts at most 15 minutes. Memory goes up to about 10 GB, and CPU grows with memory. The local disk, called /tmp, holds up to 10 GB and is wiped when the environment is recycled. So Lambda is excellent for small, quick, event-driven steps and poor for joining a year of orders to a customer table. That second job belongs to Spark on Databricks.
Two newer options stretch the model. Lambda Managed Instances, launched in late 2025, runs your functions on EC2 machines in your account that Lambda manages, with EC2 pricing plus a management fee and a 90-minute limit for asynchronous and event-source runs. Durable functions, launched in December 2025, let one function run a multi-step process with checkpoints, waits and retries. Both are real, but neither turns Lambda into a data engine.
A doorbell that wakes one helper, who does a short, specific job and goes home the moment it is done.
Big picture
Where it fits
The lit lane is where Lambda works. Boxes on the left feed it; boxes on the right use what it produces.
On this platform Lambda lives at the edges of the Move in lane and in the orchestration band. It is the small piece of code that connects an AWS event to the Databricks side. Typical callers are S3 events when supplier files land, EventBridge rules when a DMS task finishes or fails, and Step Functions when an AppFlow flow needs a check before the next step.
What Lambda hands on is usually not data but a signal. It writes a row into a control table or a file log, or it starts a Databricks job through the Jobs API with the file path as a parameter. Databricks then reads the files, cleans them, and writes bronze, silver and gold Delta tables. That split keeps the heavy, governed work inside one engine with lineage and quality rules, and keeps Lambda tiny and cheap.
Before you write a Lambda, check whether Databricks already does the job. A Lakeflow Jobs file-arrival trigger can watch an S3 path and start a job on its own, and Auto Loader can discover new files without any event plumbing. Use Lambda when the trigger or the action is on the AWS side, such as tagging a file, moving it out of quarantine after a Macie scan, or answering an AWS event that Databricks cannot see.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Function | Your code plus its settings: runtime (Python, Java, Node.js and others), memory, timeout, environment variables and an execution role. |
| Trigger and event source mapping | What starts the function. Some services push events (S3, EventBridge); for queues and streams (SQS, Kinesis, DynamoDB Streams) Lambda polls for you through an event source mapping and passes batches. |
| Execution environment and cold start | The small isolated machine that runs your code. The first call after idle must start one, which adds a delay called a cold start; later calls reuse it while it stays warm. |
| Concurrency | How many copies run at the same moment. An account starts with 1,000 per Region by default. Reserved concurrency caps or protects one function; provisioned concurrency keeps copies warm for a fee. |
| Execution role | The IAM role the function runs as. It decides which buckets, secrets and APIs the code may touch. |
| Synchronous and asynchronous calls | A synchronous caller waits for the answer. An asynchronous call is queued by Lambda, retried twice on error, and failed events can go to a dead-letter queue or an on-failure destination. |
| Layers and container images | Ways to package dependencies. A zip package with layers is capped at 250 MB unzipped; a container image can be up to 10 GB. |
| Lambda Managed Instances | A compute type where functions run on EC2 instances in your account, managed by Lambda. One environment can serve many calls at once, you can use Savings Plans, and the asynchronous limit is 90 minutes. |
| Durable functions | Functions that save checkpoints between steps so a process can wait, retry and resume over a long time without paying while it waits. |
Choosing
When to use it, and when not
Use it when
- You need to react to an AWS event, such as a new file in the landing bucket or a failed DMS task, and start something on the Databricks side.
- The work is small and quick: validate a file name, write a log row, tag an object, send an alert, or call an API.
- Traffic is spiky or rare, so paying only per run is much cheaper than keeping a container or cluster up.
- You need a tiny enrichment step in a streaming path, for example a Firehose transformation that drops a field before files are written.
- A Step Functions workflow needs a custom check between two AWS service calls.
Avoid it when
- The job reads, joins or rewrites large data; Spark on Databricks is built for that and keeps lineage and quality rules.
- A run can take longer than 15 minutes, or its run time grows with data volume and will one day cross that line.
- You are building a chain of many functions that call each other, which becomes a hidden orchestrator with no single run history.
- A Lakeflow Jobs file-arrival trigger or Auto Loader already covers the need without new AWS code.
- The code must hold a steady, long connection, such as a Kafka consumer that runs all day; use a container instead.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Timeout creep | A function that took two minutes on 10,000 orders takes 16 minutes after the business grows, and it is killed mid-run with partial output. | Keep functions to signalling and small checks; hand data work to a Databricks job; alarm on Duration approaching the timeout. |
| Duplicate events | S3 and asynchronous calls deliver at least once, so the same supplier file can start two lander runs. | Make every function idempotent: record the file path and its ETag in a log table and skip anything already seen. |
| Silent async failures | An asynchronous call fails three times and the event disappears; nobody notices a missing price file for a week. | Configure an on-failure destination or dead-letter queue, alarm on its depth, and alarm on the Errors metric. |
| Throttling at bursts | A backfill drops 50,000 files at once; Lambda hits the account concurrency limit and throttles other functions too. | Put SQS between S3 and the function, set reserved concurrency per function, and watch Throttles and ConcurrentExecutions. |
| Stream consumer stuck | One bad record in a Kinesis or SQS batch fails the whole batch again and again, and the iterator age grows. | Use partial batch responses, bisect batch on error, a maximum retry count and an on-failure destination. |
| Cold starts in a VPC | Functions that reach private systems start slower and need IP addresses in the subnets. | Usually fine for batch glue; use provisioned concurrency only where a person is waiting; size subnets for the ENIs. |
| Secrets in environment variables | A Databricks token is pasted into an environment variable and shows up in the console and in exports. | Use a Databricks service principal with OAuth, keep the secret in Secrets Manager, and fetch and cache it at start. |
| Function sprawl | Dozens of small functions with no owner, no tests and old runtimes that reach end of support. | Deploy all functions from code with CDK or Terraform, tag an owner, and track runtime deprecation dates. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick Lambda when |
|---|---|---|
| Lakeflow Jobs file-arrival trigger | The trigger is a new file in S3 and the action is a Databricks job; no AWS code is needed. | The trigger or the action is on the AWS side, or you need a check before Databricks sees the file. |
| Amazon ECS and AWS Fargate | The code runs longer than 15 minutes, holds a long connection, or needs a bigger container. | The step is short, event-driven and rare enough that per-run pricing wins. |
| AWS Step Functions | Several steps must run in order with waits, retries and a visible run history. | There is one step, or you need custom code inside a Step Functions state. |
| AWS Glue Python shell | A team already runs Glue and wants a scheduled small Python job next to its Glue jobs. | You need event triggers, fast start and pay per millisecond. |
| Azure Functions or Google Cloud Run functions | The platform runs on Azure or Google Cloud; the role is the same. | You are on AWS. |
| A Databricks serverless job | The step touches table data or needs Unity Catalog lineage and permissions. | The step only reacts to an AWS event or calls an AWS API. |
Cost
How you pay
Default Lambda bills per request and per GB-second of run time, measured in milliseconds, with a monthly free tier. Arm (Graviton) functions cost less per GB-second than x86. Provisioned concurrency adds a charge for keeping environments warm. Lambda Managed Instances bills the EC2 instances plus a management fee (documented as 15 percent of the EC2 price), and EC2 Savings Plans and Reserved Instances apply to the EC2 part. Durable functions add charges for durable operations and stored state. Check the Lambda pricing page for your region.
What drives the bill
- Memory size multiplied by run time, for every call.
- Number of calls, which for S3 triggers equals the number of files.
- Provisioned concurrency left on around the clock.
- CloudWatch Logs ingestion, which for chatty functions can cost more than Lambda itself.
- NAT gateway traffic when a function in a private subnet calls public endpoints without VPC endpoints.
Cost traps
- A function that waits on a slow API bills for the waiting time, not just the work.
- Recursive loops, such as a function that writes into the same bucket that triggers it, can run up a large bill in minutes; Lambda detects some loops, but not every pattern.
- Debug-level logging left on in production fills CloudWatch Logs at a per-GB price.
- Moving data work into Lambda to save Databricks DBUs often costs more in engineering time, retries and failed runs than it saves.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Give each function its own execution role with only the actions and resources it needs, for example read on one landing prefix and permission to call one secret.
- Call Databricks as a service principal using OAuth machine-to-machine credentials kept in Secrets Manager, never a personal access token in an environment variable.
- Run functions that reach private systems inside the VPC, with VPC endpoints for S3, STS, Secrets Manager and Kinesis, and reach Databricks through PrivateLink so nothing goes over the internet.
- Environment variables are encrypted at rest; use a customer managed KMS key where policy requires it, and keep secrets out of them altogether.
- CloudTrail records changes to functions and their configuration; function logs go to CloudWatch Logs, so never log personal data such as customer names, emails or driver details from file contents.
- Use code signing and deploy only from the pipeline, so nobody edits production code in the console.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| Errors and Invocations | The share of failed runs; a jump usually means a changed file format or an expired credential. |
| Duration | Run time against the timeout; a steady climb warns that the function will soon be killed. |
| Throttles and ConcurrentExecutions | Whether bursts hit the concurrency limit and delay or drop work. |
| AsyncEventsDropped and dead-letter queue depth | Events that failed every retry; each one is a file or message nobody processed. |
| IteratorAge | For Kinesis and DynamoDB Streams triggers, how far behind the function is reading. |
| Databricks job run results | The job the function started must also succeed; join the run id it returned to the job run system tables. |
| Limit | Value and notes |
|---|---|
| Timeout | 900 seconds (15 minutes); 90 minutes on Lambda Managed Instances for asynchronous and event-source calls |
| Memory | 128 MB to 10,240 MB; about one vCPU at 1,769 MB |
| Payload | 6 MB request and response for synchronous calls, 200 MB for streamed responses, 1 MB for asynchronous calls |
| /tmp storage | 512 MB to 10,240 MB, wiped when the environment is recycled |
| Concurrency | 1,000 per Region by default, can be raised; new accounts may start lower |
| Package size | 250 MB unzipped for zip plus layers; 10 GB for a container image |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Supplier price files land in S3 a few times a day | S3 event to a small Lambda that logs the file and starts the Databricks lander job | Pursue with care: tiny, cheap and clear; make it idempotent and alarm on failures. |
| Start a Databricks job when files land, with no extra checks | Lakeflow Jobs file-arrival trigger instead of Lambda | Pursue: no AWS code to own, and the run history stays in one place. |
| Nightly join of orders, customers and SAP material data | Lambda with pandas reading from S3 | Drop: this is Spark work for Databricks, and it will outgrow 15 minutes. |
| Chain of ten functions moving a file through checks | Replace with one Step Functions workflow or one Databricks job | Drop: a hidden orchestrator with no single run history. |
| Steady, high-volume event handler that already runs on Lambda | Lambda Managed Instances with a Savings Plan | Keep if you have it: can be cheaper at steady load, but test the multi-concurrency model first. |
FAQ
Questions people ask
Can AWS Lambda replace Spark for ETL?
Only for very small files and simple steps. A run is capped at 15 minutes and about 10 GB of memory, and there is no shared catalog, lineage or quality rules. For a data platform, let Lambda signal and let Spark on Databricks do the processing.
How do I trigger a Databricks job from AWS Lambda?
Call the Databricks Jobs API run-now endpoint from the function, signed in as a service principal with an OAuth secret kept in Secrets Manager, and pass the file path as a job parameter. Reach the workspace over PrivateLink if the workspace is private.
What is the maximum run time of AWS Lambda?
Fifteen minutes for standard Lambda. Functions on Lambda Managed Instances can run up to 90 minutes when called asynchronously or by an event source mapping. Durable functions can span longer processes by checkpointing between steps.
Is Lambda cheaper than a container for small jobs?
Usually yes when calls are rare or bursty, because you pay nothing while idle. For a function that is busy all day, a container on Fargate or Lambda Managed Instances with a Savings Plan can be cheaper.
Why did my S3-triggered Lambda run twice for one file?
S3 event notifications are delivered at least once, so duplicates happen. Record each file path and version or ETag in a log and skip anything already processed.
Sources
Checked against
- AWS docs: Lambda quotas
- AWS docs: Lambda Managed Instances
- AWS What's New: Lambda durable functions
- AWS Lambda pricing
Checked 2026-11-11.
Related
Where it connects
Data platform on AWS and Databricks · Ingestion patterns · AWS Step Functions · Amazon EventBridge · Amazon ECS and AWS Fargate · All AWS data services
← Glue ETL jobs · API Gateway →