Home › AWS › AWS data services › Lambda

AWS data services · Processing and compute · about 12 min read

AWS Lambda

Runs a small piece of your code when something happens, for up to 15 minutes, and bills only while it runs.

RoleProcessing and compute
StatusGenerally available. Lambda Managed Instances (functions on EC2 in your account, launched Nov 2025) and durable functions (multi-step, launched Dec 2025) are also available.
Verdict on this sitePursue with care
AzureAzure Functions
Google CloudCloud Run functions
Databricksnone: use a Lakeflow Jobs file-arrival trigger or a small serverless job
Open sourceOpenFaaS or Knative functions
Verdict: Pursue with care. Good small glue on the AWS side: react to a file landing in S3 or a DMS task event, call the Databricks Jobs API, or do a tiny clean-up. Never make it the data processing engine; Databricks does that, and the 15-minute limit will catch you.

In plain words

What it is

AWS Lambda runs a function, a short piece of your code, when an event happens. The event can be a file arriving in an S3 bucket, a message on a queue, a record on a Kinesis stream, a schedule, or an HTTP call. You upload the code, say how much memory it needs, and AWS finds a machine, starts your code, runs it and throws the machine away again. You never see a server, and you pay for the milliseconds the code runs, not for idle time.

Think of a tyre or food manufacturer whose suppliers drop price files over SFTP. Each file lands in an S3 landing bucket. A Lambda function wakes up on the S3 event, checks that the file name matches the agreed pattern, writes one row into a file log, and calls the Databricks Jobs API to start the lander job. The whole function runs for less than a second. Nothing sits waiting between files, and nothing costs money at night when no files arrive.

The limits shape what Lambda is good for. One run lasts at most 15 minutes. Memory goes up to about 10 GB, and CPU grows with memory. The local disk, called /tmp, holds up to 10 GB and is wiped when the environment is recycled. So Lambda is excellent for small, quick, event-driven steps and poor for joining a year of orders to a customer table. That second job belongs to Spark on Databricks.

Two newer options stretch the model. Lambda Managed Instances, launched in late 2025, runs your functions on EC2 machines in your account that Lambda manages, with EC2 pricing plus a management fee and a 90-minute limit for asynchronous and event-source runs. Durable functions, launched in December 2025, let one function run a multi-step process with checkpoints, waits and retries. Both are real, but neither turns Lambda into a data engine.

A doorbell that wakes one helper, who does a short, specific job and goes home the moment it is done.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toS3EventBridgeSQSLambdaProcessing and computeDatabricks Jobs APIDynamoDBStep Functions

The lit lane is where Lambda works. Boxes on the left feed it; boxes on the right use what it produces.

On this platform Lambda lives at the edges of the Move in lane and in the orchestration band. It is the small piece of code that connects an AWS event to the Databricks side. Typical callers are S3 events when supplier files land, EventBridge rules when a DMS task finishes or fails, and Step Functions when an AppFlow flow needs a check before the next step.

What Lambda hands on is usually not data but a signal. It writes a row into a control table or a file log, or it starts a Databricks job through the Jobs API with the file path as a parameter. Databricks then reads the files, cleans them, and writes bronze, silver and gold Delta tables. That split keeps the heavy, governed work inside one engine with lineage and quality rules, and keeps Lambda tiny and cheap.

Before you write a Lambda, check whether Databricks already does the job. A Lakeflow Jobs file-arrival trigger can watch an S3 path and start a job on its own, and Auto Loader can discover new files without any event plumbing. Use Lambda when the trigger or the action is on the AWS side, such as tagging a file, moving it out of quarantine after a Macie scan, or answering an AWS event that Databricks cannot see.

How it works

The ideas you need

IdeaWhat it means
FunctionYour code plus its settings: runtime (Python, Java, Node.js and others), memory, timeout, environment variables and an execution role.
Trigger and event source mappingWhat starts the function. Some services push events (S3, EventBridge); for queues and streams (SQS, Kinesis, DynamoDB Streams) Lambda polls for you through an event source mapping and passes batches.
Execution environment and cold startThe small isolated machine that runs your code. The first call after idle must start one, which adds a delay called a cold start; later calls reuse it while it stays warm.
ConcurrencyHow many copies run at the same moment. An account starts with 1,000 per Region by default. Reserved concurrency caps or protects one function; provisioned concurrency keeps copies warm for a fee.
Execution roleThe IAM role the function runs as. It decides which buckets, secrets and APIs the code may touch.
Synchronous and asynchronous callsA synchronous caller waits for the answer. An asynchronous call is queued by Lambda, retried twice on error, and failed events can go to a dead-letter queue or an on-failure destination.
Layers and container imagesWays to package dependencies. A zip package with layers is capped at 250 MB unzipped; a container image can be up to 10 GB.
Lambda Managed InstancesA compute type where functions run on EC2 instances in your account, managed by Lambda. One environment can serve many calls at once, you can use Savings Plans, and the asynchronous limit is 90 minutes.
Durable functionsFunctions that save checkpoints between steps so a process can wait, retry and resume over a long time without paying while it waits.

Choosing

When to use it, and when not

Use it when

  • You need to react to an AWS event, such as a new file in the landing bucket or a failed DMS task, and start something on the Databricks side.
  • The work is small and quick: validate a file name, write a log row, tag an object, send an alert, or call an API.
  • Traffic is spiky or rare, so paying only per run is much cheaper than keeping a container or cluster up.
  • You need a tiny enrichment step in a streaming path, for example a Firehose transformation that drops a field before files are written.
  • A Step Functions workflow needs a custom check between two AWS service calls.

Avoid it when

  • The job reads, joins or rewrites large data; Spark on Databricks is built for that and keeps lineage and quality rules.
  • A run can take longer than 15 minutes, or its run time grows with data volume and will one day cross that line.
  • You are building a chain of many functions that call each other, which becomes a hidden orchestrator with no single run history.
  • A Lakeflow Jobs file-arrival trigger or Auto Loader already covers the need without new AWS code.
  • The code must hold a steady, long connection, such as a Kafka consumer that runs all day; use a container instead.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Timeout creepA function that took two minutes on 10,000 orders takes 16 minutes after the business grows, and it is killed mid-run with partial output.Keep functions to signalling and small checks; hand data work to a Databricks job; alarm on Duration approaching the timeout.
Duplicate eventsS3 and asynchronous calls deliver at least once, so the same supplier file can start two lander runs.Make every function idempotent: record the file path and its ETag in a log table and skip anything already seen.
Silent async failuresAn asynchronous call fails three times and the event disappears; nobody notices a missing price file for a week.Configure an on-failure destination or dead-letter queue, alarm on its depth, and alarm on the Errors metric.
Throttling at burstsA backfill drops 50,000 files at once; Lambda hits the account concurrency limit and throttles other functions too.Put SQS between S3 and the function, set reserved concurrency per function, and watch Throttles and ConcurrentExecutions.
Stream consumer stuckOne bad record in a Kinesis or SQS batch fails the whole batch again and again, and the iterator age grows.Use partial batch responses, bisect batch on error, a maximum retry count and an on-failure destination.
Cold starts in a VPCFunctions that reach private systems start slower and need IP addresses in the subnets.Usually fine for batch glue; use provisioned concurrency only where a person is waiting; size subnets for the ENIs.
Secrets in environment variablesA Databricks token is pasted into an environment variable and shows up in the console and in exports.Use a Databricks service principal with OAuth, keep the secret in Secrets Manager, and fetch and cache it at start.
Function sprawlDozens of small functions with no owner, no tests and old runtimes that reach end of support.Deploy all functions from code with CDK or Terraform, tag an owner, and track runtime deprecation dates.

Alternatives

What to compare it with

AlternativePick the alternative whenPick Lambda when
Lakeflow Jobs file-arrival triggerThe trigger is a new file in S3 and the action is a Databricks job; no AWS code is needed.The trigger or the action is on the AWS side, or you need a check before Databricks sees the file.
Amazon ECS and AWS FargateThe code runs longer than 15 minutes, holds a long connection, or needs a bigger container.The step is short, event-driven and rare enough that per-run pricing wins.
AWS Step FunctionsSeveral steps must run in order with waits, retries and a visible run history.There is one step, or you need custom code inside a Step Functions state.
AWS Glue Python shellA team already runs Glue and wants a scheduled small Python job next to its Glue jobs.You need event triggers, fast start and pay per millisecond.
Azure Functions or Google Cloud Run functionsThe platform runs on Azure or Google Cloud; the role is the same.You are on AWS.
A Databricks serverless jobThe step touches table data or needs Unity Catalog lineage and permissions.The step only reacts to an AWS event or calls an AWS API.

Cost

How you pay

Default Lambda bills per request and per GB-second of run time, measured in milliseconds, with a monthly free tier. Arm (Graviton) functions cost less per GB-second than x86. Provisioned concurrency adds a charge for keeping environments warm. Lambda Managed Instances bills the EC2 instances plus a management fee (documented as 15 percent of the EC2 price), and EC2 Savings Plans and Reserved Instances apply to the EC2 part. Durable functions add charges for durable operations and stored state. Check the Lambda pricing page for your region.

What drives the bill

  • Memory size multiplied by run time, for every call.
  • Number of calls, which for S3 triggers equals the number of files.
  • Provisioned concurrency left on around the clock.
  • CloudWatch Logs ingestion, which for chatty functions can cost more than Lambda itself.
  • NAT gateway traffic when a function in a private subnet calls public endpoints without VPC endpoints.

Cost traps

  • A function that waits on a slow API bills for the waiting time, not just the work.
  • Recursive loops, such as a function that writes into the same bucket that triggers it, can run up a large bill in minutes; Lambda detects some loops, but not every pattern.
  • Debug-level logging left on in production fills CloudWatch Logs at a per-GB price.
  • Moving data work into Lambda to save Databricks DBUs often costs more in engineering time, retries and failed runs than it saves.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Give each function its own execution role with only the actions and resources it needs, for example read on one landing prefix and permission to call one secret.
  • Call Databricks as a service principal using OAuth machine-to-machine credentials kept in Secrets Manager, never a personal access token in an environment variable.
  • Run functions that reach private systems inside the VPC, with VPC endpoints for S3, STS, Secrets Manager and Kinesis, and reach Databricks through PrivateLink so nothing goes over the internet.
  • Environment variables are encrypted at rest; use a customer managed KMS key where policy requires it, and keep secrets out of them altogether.
  • CloudTrail records changes to functions and their configuration; function logs go to CloudWatch Logs, so never log personal data such as customer names, emails or driver details from file contents.
  • Use code signing and deploy only from the pipeline, so nobody edits production code in the console.

Operating it

What to watch, and the limits

WatchWhy
Errors and InvocationsThe share of failed runs; a jump usually means a changed file format or an expired credential.
DurationRun time against the timeout; a steady climb warns that the function will soon be killed.
Throttles and ConcurrentExecutionsWhether bursts hit the concurrency limit and delay or drop work.
AsyncEventsDropped and dead-letter queue depthEvents that failed every retry; each one is a file or message nobody processed.
IteratorAgeFor Kinesis and DynamoDB Streams triggers, how far behind the function is reading.
Databricks job run resultsThe job the function started must also succeed; join the run id it returned to the job run system tables.
LimitValue and notes
Timeout900 seconds (15 minutes); 90 minutes on Lambda Managed Instances for asynchronous and event-source calls
Memory128 MB to 10,240 MB; about one vCPU at 1,769 MB
Payload6 MB request and response for synchronous calls, 200 MB for streamed responses, 1 MB for asynchronous calls
/tmp storage512 MB to 10,240 MB, wiped when the environment is recycled
Concurrency1,000 per Region by default, can be raised; new accounts may start lower
Package size250 MB unzipped for zip plus layers; 10 GB for a container image

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
Supplier price files land in S3 a few times a dayS3 event to a small Lambda that logs the file and starts the Databricks lander jobPursue with care: tiny, cheap and clear; make it idempotent and alarm on failures.
Start a Databricks job when files land, with no extra checksLakeflow Jobs file-arrival trigger instead of LambdaPursue: no AWS code to own, and the run history stays in one place.
Nightly join of orders, customers and SAP material dataLambda with pandas reading from S3Drop: this is Spark work for Databricks, and it will outgrow 15 minutes.
Chain of ten functions moving a file through checksReplace with one Step Functions workflow or one Databricks jobDrop: a hidden orchestrator with no single run history.
Steady, high-volume event handler that already runs on LambdaLambda Managed Instances with a Savings PlanKeep if you have it: can be cheaper at steady load, but test the multi-concurrency model first.

FAQ

Questions people ask

Can AWS Lambda replace Spark for ETL?

Only for very small files and simple steps. A run is capped at 15 minutes and about 10 GB of memory, and there is no shared catalog, lineage or quality rules. For a data platform, let Lambda signal and let Spark on Databricks do the processing.

How do I trigger a Databricks job from AWS Lambda?

Call the Databricks Jobs API run-now endpoint from the function, signed in as a service principal with an OAuth secret kept in Secrets Manager, and pass the file path as a job parameter. Reach the workspace over PrivateLink if the workspace is private.

What is the maximum run time of AWS Lambda?

Fifteen minutes for standard Lambda. Functions on Lambda Managed Instances can run up to 90 minutes when called asynchronously or by an event source mapping. Durable functions can span longer processes by checkpointing between steps.

Is Lambda cheaper than a container for small jobs?

Usually yes when calls are rare or bursty, because you pay nothing while idle. For a function that is busy all day, a container on Fargate or Lambda Managed Instances with a Savings Plan can be cheaper.

Why did my S3-triggered Lambda run twice for one file?

S3 event notifications are delivered at least once, so duplicates happen. Record each file path and version or ETag in a log and skip anything already processed.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

Data platform on AWS and Databricks · Ingestion patterns · AWS Step Functions · Amazon EventBridge · Amazon ECS and AWS Fargate · All AWS data services

← Glue ETL jobs · API Gateway →