Home › AWS › AWS data services › ECS and Fargate
AWS data services · Processing and compute · about 11 min read
Amazon Elastic Container Service (ECS) and AWS Fargate
Runs your containers on AWS: ECS decides where and how many, and Fargate supplies the machines so you never manage servers.
In plain words
What it is
A container is a packaged program with everything it needs: the code, the libraries and the settings, in one image that runs the same on a laptop and in the cloud. Amazon Elastic Container Service, or ECS, is AWS's own system for running containers. You describe a task, which is one or more containers with their CPU, memory, network and IAM role, and ECS starts it, keeps the right number running as a service, restarts it when it fails, and spreads copies across availability zones.
AWS Fargate is the place those tasks can run without you owning any servers. You say a task needs two vCPU and 8 GB of memory, and Fargate supplies an isolated slice of a machine for exactly that, billed per second. The other choices are EC2 instances that you manage, or ECS Managed Instances, launched in late 2025, where ECS picks, patches and replaces the EC2 machines for you and you pay EC2 prices.
A retail example: the chain's loyalty app vendor offers only a paged REST API with a strict rate limit, and AppFlow has no connector for it. A small Python container polls the API every few minutes, writes newline JSON files of new loyalty events into the S3 landing bucket, and records what it fetched in a log table. It runs all day, holds an OAuth session, and would be killed by Lambda's 15-minute limit, so a single Fargate task as an ECS service is a natural fit.
A shipping-container yard: ECS is the yard manager who decides which boxes go where and replaces any that break, and Fargate is the crane you rent by the minute instead of buying.
Big picture
Where it fits
The lit lane is where ECS and Fargate works. Boxes on the left feed it; boxes on the right use what it produces.
In the reference design ECS does not appear by name. Movers are managed AWS services: DMS for databases, AppFlow for SaaS and SAP OData, Transfer Family for supplier SFTP, IoT Core and SiteWise for plants, and Kinesis for streams. Databricks runs the lander, silver, gold and quality work, and Lakeflow Jobs orchestrates it. That leaves a narrow gap: a source with no managed connector that needs a long-running process.
When that gap appears, a container on Fargate sits in the Move in lane. It reads from the awkward source, writes files into the S3 landing bucket or records onto a Kinesis stream, and stops there. Everything after landing, including tokenising personal data, belongs to Databricks so that every source passes the same lander and run log. The container should be dumb and replaceable: fetch, land, log.
Check two cheaper options first. A Databricks job using our puller's REST connector can often call the API itself on serverless compute, which keeps the code inside the platform's run history and lineage. And if the call is short, a Lambda function on a schedule is simpler. Reach for ECS only when the process must run for a long time, hold a connection, or use a library that needs a full container, and preferably when the company already runs ECS with a pipeline, logging and on-call in place.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Task definition | The recipe for a task: container images, CPU and memory, environment, secrets, logging, ports and the IAM roles to use. |
| Task and service | A task is one running copy. A service keeps a set number of tasks running, replaces failed ones and can sit behind a load balancer. |
| Cluster | A logical group of tasks and capacity. On Fargate it holds no servers of its own; it is a name and a set of settings. |
| Capacity provider | Where tasks run: Fargate, Fargate Spot, an Auto Scaling group of your own EC2 instances, or ECS Managed Instances. A strategy can mix them with weights. |
| Fargate Spot | Spare Fargate capacity at a large discount that AWS can take back with a two-minute warning. Fine for work that can restart. |
| Task role and execution role | The task role is what your code may do in AWS, such as write to one S3 prefix. The execution role lets ECS pull the image and read secrets at start. |
| awsvpc networking | Each Fargate task gets its own network interface and private IP in your subnets, with its own security group. |
| ECS Managed Instances | EC2 capacity that ECS selects, patches and replaces for you, at least every 14 days, while you still choose instance attributes such as GPUs or large memory. |
| ECS Express Mode | A one-step way to run a web service from an image; it sets up the service, load balancer, scaling and a URL. Aimed at web apps and APIs, not data jobs. |
Choosing
When to use it, and when not
Use it when
- A source has no managed AWS or Databricks connector and the puller must run longer than Lambda's 15 minutes or hold a long connection.
- The company already runs ECS with a deployment pipeline, logging, alerting and people on call who know it.
- You need a container image with system libraries, such as an SAP RFC library or a vendor SDK, that will not fit a Lambda package.
- A small always-on worker, such as a Kafka Connect style process that moves events from an on-premises broker onto Kinesis, needs a supervisor that restarts it.
- You want to run a scheduled container task triggered by EventBridge Scheduler for a nightly fetch that takes an hour.
Avoid it when
- The work is transforming, joining or cleaning data; Databricks does that with lineage, quality rules and Unity Catalog permissions.
- A managed mover already exists for the source: DMS, AppFlow, Transfer Family, Firehose or Lakeflow Connect.
- Nobody on the data team has run containers in production and the company has no container platform to lean on.
- The job is short and event-driven; Lambda is simpler and cheaper.
- You are tempted to run Spark yourself in containers to save money; you would be building a second, unmanaged processing platform.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Silent stall | The puller stays running but stops fetching after a token expires; ECS sees a healthy task and nobody sees missing files. | Add a health check that fails when no successful fetch has happened recently, and alarm on landed file counts per source. |
| Duplicate or missing pages | A restart mid-run re-fetches some pages or skips others, so landing has duplicates or gaps. | Store a watermark per source after each successful page, make writes idempotent with deterministic file names, and let the lander dedupe. |
| Image pulls through NAT | Private tasks pull large images through a NAT gateway, adding cost and slow starts. | Use ECR with interface VPC endpoints and an S3 gateway endpoint; keep images small. |
| Spot interruption mid-write | A Fargate Spot task is reclaimed and a half-written file lands in S3. | Write to a temporary key and rename on completion, handle the stop signal, or keep long single-copy pullers on regular Fargate. |
| Secrets baked into images | API keys end up in the image or in plain environment variables visible in the task definition. | Reference Secrets Manager entries in the task definition so ECS injects them at start; rotate them. |
| Unpatched images | The base image ages and collects known vulnerabilities while the task keeps running. | Rebuild images on a schedule, enable ECR image scanning, and redeploy; Managed Instances also replace hosts at least every 14 days. |
| Out of memory restarts | A large API response is held in memory and the task is killed, then restarts in a loop. | Stream responses to S3 in chunks, set memory with headroom, and alarm on task stop reasons. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick ECS and Fargate when |
|---|---|---|
| AWS Lambda | The step is short, event-driven and fits in 15 minutes and 10 GB. | The process runs long, holds a connection or needs a full container image. |
| Amazon EKS | The company runs Kubernetes as its standard and the puller should live with other Kubernetes workloads. | You want containers without running Kubernetes. |
| A Databricks job on serverless compute | The puller can run as Python inside the platform and the network can reach the source; you keep one run history. | The process must run all day or needs software Databricks compute cannot host. |
| Amazon AppFlow or Lakeflow Connect | A managed connector exists for the source. | No connector exists and you must write your own puller. |
| Azure Container Apps or Google Cloud Run | The platform runs on Azure or Google Cloud; the role is the same. | You are on AWS. |
Cost
How you pay
ECS itself has no charge. On Fargate you pay per second, with a one-minute minimum, for the vCPU, memory and any ephemeral storage above the included 20 GB that a task requests, with lower prices for Arm (Graviton) tasks and a large discount on Fargate Spot. On EC2 or ECS Managed Instances you pay for the instances; Managed Instances adds a per-instance management fee that depends on the instance type, and Savings Plans apply to the EC2 part. Load balancers, NAT gateways, CloudWatch Logs and ECR storage are billed separately. Check the Fargate pricing page for your region.
What drives the bill
- vCPU and memory requested per task, multiplied by hours running; an always-on puller runs about 730 hours a month.
- Number of tasks kept running by each service.
- Load balancers, which bill per hour even when traffic is low.
- NAT gateway data processing for image pulls and API calls without VPC endpoints.
- CloudWatch Logs volume from chatty containers.
Cost traps
- Over-sized tasks: a puller given 4 vCPU and 16 GB because that was the template, while it uses a tenth of it.
- A load balancer in front of a worker that never receives inbound traffic.
- Forgotten scheduled tasks and dev services that keep running after a project ends.
- Running your own Spark in containers to avoid DBUs, then paying for the engineering and the failures instead.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Give every task its own task role with least privilege, for example write to one landing prefix only, and keep the execution role limited to pulling the image and reading named secrets.
- Run tasks in private subnets with no public IP, and reach S3, ECR, Secrets Manager, STS, CloudWatch Logs and Kinesis through VPC endpoints.
- Inject secrets from Secrets Manager or Parameter Store through the task definition, never in the image or plain environment variables.
- Fargate ephemeral storage is encrypted by default; use a customer managed KMS key where policy asks for it, and encrypt the landing bucket with the catalog's key.
- Scan images in ECR, pin images by digest, and record deployments; CloudTrail logs ECS API calls such as task definition changes.
- If the puller handles personal data, such as loyalty customer emails, land it raw only in the restricted landing bucket and let the lander tokenise it; never write it to logs.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| RunningTaskCount against desired count | Whether the service has the tasks it should; a gap means crashes or failed placements. |
| CPUUtilization and MemoryUtilization | Whether the task is sized right, or close to being killed for memory. |
| Task stopped reasons (ECS events in EventBridge) | Why tasks stop: out of memory, failed health check, Spot interruption or image pull failure. |
| Files landed per source per hour | The business signal: a running task that lands nothing is a broken mover. |
| Ephemeral storage used (Container Insights) | Whether temporary files are filling the task's disk. |
| Source watermark age | How far behind the source the puller is, read from its own log table. |
| Limit | Value and notes |
|---|---|
| Fargate task size | Up to 16 vCPU with 120 GB, and on Linux platform 1.4.0 up to 32 vCPU with 244 GB |
| Fargate ephemeral storage | 20 GiB by default, up to 200 GiB per task |
| Fargate Spot | Can be reclaimed with a two-minute warning |
| ECS Managed Instances | Instances are drained and replaced at least every 14 days; no custom AMIs or SSH |
| Fargate networking | awsvpc mode only; one network interface and private IP per task |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Loyalty vendor REST API with no connector, polled all day | One Fargate task as an ECS service writing to S3 landing | Keep if you have it: a good fit when the company already runs ECS; otherwise try a Databricks job first. |
| Moving events from an on-premises broker onto Kinesis | A small connector worker on Fargate in a private subnet | Pursue with care: keep it to moving events; watch lag and restarts. |
| Nightly sales aggregation for 400 stores | Pandas or Spark in a Fargate task | Drop: this is processing; run it on Databricks. |
| A one-off backfill fetch of three years of API history | Scheduled Fargate Spot tasks with checkpoints | Pursue with care: cheap if each page is checkpointed so interruptions only cost a retry. |
FAQ
Questions people ask
What is the difference between ECS and Fargate?
ECS is the service that runs and supervises containers. Fargate is one of the places ECS can run them, where AWS supplies the machines per task. You can also run ECS tasks on your own EC2 instances or on ECS Managed Instances.
ECS or EKS for a data platform?
If you only need a few containers, ECS on Fargate is simpler: no Kubernetes to learn, upgrade or pay for. Choose EKS when the company already standardises on Kubernetes. On a Databricks platform you may need neither.
Can I run Spark on Fargate?
Technically yes, but you would be running and tuning your own Spark platform without a catalog, lineage or autoscaling built for Spark. On this platform Spark runs on Databricks.
Is Fargate Spot safe for data jobs?
Yes for work that checkpoints and can restart, because AWS can take the capacity back with two minutes' notice. Keep a single always-on puller on regular Fargate.
What are ECS Managed Instances?
A compute option launched in late 2025 where ECS chooses, patches and replaces EC2 instances for you. You get EC2 choices such as GPUs and Savings Plans without running the servers yourself.
Sources
Checked against
- AWS docs: Fargate task definition differences and sizes
- AWS docs: Fargate task ephemeral storage
- AWS docs: Amazon ECS Managed Instances
- AWS What's New: Amazon ECS Express Mode
- AWS Fargate pricing
Checked 2026-11-11.
Related
Where it connects
Ingestion patterns · Data platform on AWS and Databricks · AWS Lambda · Amazon EKS · Amazon AppFlow · All AWS data services