Home › AWS › AWS data services › EKS
AWS data services · Processing and compute · about 10 min read
Amazon Elastic Kubernetes Service (EKS)
Managed Kubernetes on AWS: AWS runs the control plane, and with Auto Mode the worker machines too, while you run containers on it.
In plain words
What it is
Kubernetes is an open-source system for running many containers across many machines. You describe what should run, for example three copies of an API and one copy of a worker, in files called manifests, and Kubernetes keeps reality matching that description: it places containers, called pods, onto machines, called nodes, restarts them when they fail, and scales them up and down. It is powerful and portable, and it has a lot of moving parts.
Amazon Elastic Kubernetes Service, or EKS, is Kubernetes run by AWS. AWS runs the control plane, the brain that stores the desired state and makes decisions, across several availability zones. You choose how the nodes are run: your own EC2 machines, managed node groups, Fargate, or EKS Auto Mode, where AWS also picks, patches and replaces the EC2 nodes, with each node living at most 21 days. Since December 2025, EKS Capabilities can also run Argo CD, AWS Controllers for Kubernetes and kro for you, outside your nodes.
A manufacturing example: a company's digital team already runs its dealer portal and warranty-claim APIs on EKS, with a platform team, GitOps deployments and on-call rotations. When the data team needs to move events from the plants' on-premises Kafka brokers into AWS, they deploy Kafka Connect workers on the same cluster with the Strimzi operator, write the events onto Kinesis or MSK, and let Databricks take them from there. The data team reuses a platform that already exists rather than building one.
A large automated warehouse with its own robots and rules: excellent when you already run one, a big project to build just to store a few boxes.
Big picture
Where it fits
The lit lane is where EKS works. Boxes on the left feed it; boxes on the right use what it produces.
The reference design does not need EKS. Processing, orchestration and governance live in Databricks, ingestion uses managed movers such as DMS, AppFlow, Transfer Family and Kinesis, and the fast lane uses Managed Flink and DynamoDB. Adding a Kubernetes cluster adds upgrades every few months, node patching, add-ons, networking and security policies, and the skills to run them.
Where EKS earns a place is at the edges, when it already exists. In the Move in lane it can host long-running connectors, such as Kafka Connect or Debezium workers, that land data in S3, Kinesis or MSK. In the Serve lane it can host the company's own APIs and apps that read gold tables through a Databricks SQL warehouse or Lakebase. In both cases the data work after landing, and the business logic over the tables, stays in Databricks.
Spark on EKS, through the Spark operator or EMR on EKS, is the case to resist. It looks cheap because the cluster is already there, but you take on Spark images, autoscaling, shuffle storage, a catalog, lineage and access control yourself, and the data now lives under two engines with two permission models. The site's verdict on EMR and Glue jobs applies here too: new processing runs on Databricks.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Control plane | The Kubernetes API server and its database, run by AWS across availability zones. You pay a fixed hourly fee per cluster. |
| Node, pod and deployment | A node is a machine; a pod is one or more containers that run together; a deployment keeps a set number of identical pods running. |
| Managed node groups and Karpenter | Two ways to supply EC2 nodes. Managed node groups are Auto Scaling groups EKS helps manage; Karpenter starts the right instance type for waiting pods. |
| EKS Auto Mode | AWS manages the nodes, autoscaling, load balancers, storage drivers and pod networking. Nodes use locked-down Bottlerocket images and are replaced at least every 21 days. |
| Standard and extended support | Each Kubernetes minor version gets 14 months of standard support, then 12 months of paid extended support, after which AWS upgrades the control plane for you. |
| Operator | A program running in the cluster that manages a complex application, such as Strimzi for Kafka or the Spark operator for Spark jobs. |
| EKS Pod Identity and IRSA | Ways to give a pod its own IAM role, so a Kafka Connect worker can write to one stream without the whole node having that permission. |
| EKS Capabilities | Managed Argo CD (GitOps deployments), AWS Controllers for Kubernetes (create AWS resources from manifests) and kro (group resources), run by AWS outside your nodes and billed per hour. |
| EMR on EKS | An option to run Amazon EMR's Spark on an existing EKS cluster, sharing nodes with other workloads. |
Choosing
When to use it, and when not
Use it when
- The company already runs EKS with a platform team, upgrade routine, GitOps and on-call, and offers it to the data team as a service.
- You need long-running connectors, such as Kafka Connect, Debezium or MQTT bridges, alongside other Kubernetes workloads.
- The apps that read gold data, such as a dealer portal or a store replenishment API, already live on Kubernetes.
- You must run the same container platform on several clouds or on premises, and Kubernetes is the agreed common layer.
- Workloads need operators or Kubernetes-native tools that have no simpler managed equivalent on AWS.
Avoid it when
- The only reason is to run Spark or Airflow more cheaply than Databricks or MWAA; you would be building and staffing a second platform.
- The data team would be the only user and nobody has run Kubernetes upgrades in production.
- A few containers are all you need; ECS on Fargate does that with far less to operate.
- A managed mover such as DMS, AppFlow, Firehose or MSK Connect already covers the source.
- You cannot commit to upgrading Kubernetes roughly every year, which is what avoiding extended support fees requires.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Version treadmill | A minor version leaves standard support after 14 months; the cluster moves to extended support at six times the control plane price, then is upgraded automatically. | Plan an upgrade at least once a year, test add-ons and operators on a staging cluster, and alarm on the end-of-support dates. |
| Add-on and API drift | An upgrade removes an old Kubernetes API that a Helm chart or operator still uses, and deployments fail. | Scan manifests for deprecated APIs before each upgrade and keep operators such as Strimzi current. |
| IP address exhaustion | Each pod takes a VPC IP address; busy clusters run out of addresses in small subnets and pods stay pending. | Size subnets generously, use prefix delegation or a secondary CIDR range, and watch available IPs per subnet. |
| Connector restarts and lag | Kafka Connect workers rebalance on every pod restart, and node replacement every few weeks causes lag spikes. | Set pod disruption budgets, spread workers across zones, and alarm on connector lag rather than pod health. |
| Overly broad permissions | A node role with write access to every bucket lets any pod on the node write anywhere. | Use EKS Pod Identity per service account, keep node roles minimal, and audit with CloudTrail. |
| Spark on Kubernetes sprawl | A team runs Spark jobs on the cluster; tables now have two engines, two permission models and no shared lineage. | Keep processing on Databricks; allow only movers and apps on EKS, enforced by namespace policy. |
| Hidden costs | Load balancers, NAT traffic, logs and idle nodes cost more than the control plane fee. | Tag namespaces, use Auto Mode or Karpenter consolidation, and review the bill per namespace monthly. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick EKS when |
|---|---|---|
| Amazon ECS and AWS Fargate | You need a few containers and no Kubernetes skills or ecosystem. | The company runs Kubernetes already and wants one platform for all containers. |
| Amazon MSK Connect | You only need managed Kafka Connect workers and no other containers. | Connectors need custom plugins, operators or sit next to other services on the cluster. |
| Databricks jobs and apps | The workload is processing, quality checks or a data app over Unity Catalog tables. | The workload is an always-on connector or a company app that already lives on Kubernetes. |
| Amazon EMR on EKS | You must keep EMR Spark jobs running and want them on shared nodes during a migration. | You are not running Spark at all on the cluster. |
| Azure AKS or Google GKE | The platform runs on Azure or Google Cloud; the role is the same. | You are on AWS. |
Cost
How you pay
Each cluster pays an hourly control plane fee: list price 0.10 US dollars per hour in standard support and 0.60 per hour in extended support, check your region. Nodes are billed as EC2 instances, Fargate pods or Auto Mode instances; Auto Mode adds a per-instance management fee on top of the EC2 price. EKS Capabilities bill a base rate per hour plus a usage rate per managed application or resource. An optional Provisioned Control Plane adds larger fixed hourly tiers. Load balancers, EBS, NAT and logs are billed by their own services.
What drives the bill
- The EC2 nodes, by far the biggest line; utilisation decides how much of them you waste.
- Number of clusters, each with its own control plane fee and add-ons.
- Extended support fees for clusters that are not upgraded in time.
- Load balancers, NAT gateway traffic and cross-zone data transfer.
- CloudWatch Container Insights and log volume.
Cost traps
- A cluster per team or per environment multiplies fixed fees, add-ons and upgrade work.
- Extended support starts automatically when standard support ends; a forgotten dev cluster pays six times the control plane price.
- Pods that request far more CPU and memory than they use keep large nodes alive.
- Counting only the AWS bill: people time for upgrades and incidents is the largest cost of running Kubernetes.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Keep the cluster API endpoint private, or restrict public access to known address ranges, and run nodes in private subnets without public IPs.
- Give each workload its own IAM role through EKS Pod Identity, scoped to one stream, one bucket prefix or one secret; keep the node role minimal.
- Map people to Kubernetes permissions with EKS access entries and your identity provider, and keep cluster-admin to a small group.
- Encrypt Kubernetes secrets with a KMS key, or better, read secrets from Secrets Manager at run time.
- Use network policies so connector pods can reach only their sources and AWS endpoints; reach S3, STS, ECR and Kinesis through VPC endpoints.
- Send control plane audit logs to CloudWatch and keep CloudTrail on; do not let pods log the personal data that passes through connectors.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| Pods pending and node count | Pods that cannot be placed show capacity or IP shortages before users notice. |
| Pod restarts and OOMKilled events | Crash loops or memory kills in connectors and apps. |
| Connector lag | For Kafka Connect or Debezium, how far behind the source the workers are; the real health signal for data. |
| Control plane version and end-of-support date | When the cluster will move to paid extended support. |
| Available IPs per subnet | Running out stops new pods from starting. |
| Cost per namespace | Which workloads drive the node bill, from tags and split cost allocation data. |
| Limit | Value and notes |
|---|---|
| Standard support | 14 months per Kubernetes minor version |
| Extended support | A further 12 months at a higher hourly price, then automatic control plane upgrade |
| Auto Mode node lifetime | At most 21 days, then the node is replaced |
| EKS Capabilities | One of each capability type (Argo CD, ACK, kro) per cluster |
| Pod IP addresses | Each pod takes a VPC IP address by default, so subnet size limits pod count |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Company already runs EKS; plant Kafka must reach AWS | Kafka Connect with Strimzi on the shared cluster, writing to MSK or Kinesis | Keep if you have it: reuse the platform and its people; keep the workers to moving data. |
| No Kubernetes in the company; one custom API puller needed | ECS on Fargate or a Databricks job instead of a new EKS cluster | Drop: a Kubernetes cluster for one container is all cost and no gain. |
| Team proposes Spark on EKS to cut Databricks spend | Spark operator or EMR on EKS for silver and gold | Drop: a second processing platform with no shared catalog or lineage. |
| Store replenishment API reading gold tables | Run the API on the existing EKS cluster, reading through a Databricks SQL warehouse or Lakebase | Pursue with care: fine where the app team owns EKS; keep the logic over tables in Databricks. |
FAQ
Questions people ask
Do I need Kubernetes for a Databricks data platform on AWS?
No. Databricks runs its own compute, and AWS managed movers cover most sources. EKS only makes sense if the company already runs it and offers it as a shared service.
What is EKS Auto Mode?
An EKS option where AWS manages the worker nodes, autoscaling, load balancing, storage and pod networking as well as the control plane. Nodes are replaced at least every 21 days and you pay a management fee on top of EC2.
How long is a Kubernetes version supported on EKS?
Fourteen months of standard support from its EKS release, then twelve months of extended support at a higher price. After that AWS upgrades the control plane automatically, and your nodes and add-ons still need updating.
Should I run Spark on EKS instead of Databricks?
Only if you are ready to build and staff a Spark platform: images, autoscaling, shuffle, catalog, lineage and security. For most data teams the saving disappears in engineering time, and data ends up governed in two places.
ECS or EKS for a few data containers?
ECS on Fargate. It has no control plane fee, no version upgrades and far fewer concepts to learn.
Sources
Checked against
- AWS docs: Kubernetes version lifecycle on EKS
- AWS docs: EKS Auto Mode
- AWS docs: EKS Capabilities
- Amazon EKS pricing
Checked 2026-11-11.
Related
Where it connects
Data platform on AWS and Databricks · Streaming on AWS and Databricks · Amazon ECS and AWS Fargate · Amazon EMR · Amazon MSK · All AWS data services
← ECS and Fargate · Glue ETL jobs →