Home › AWS › AWS data services › EC2
AWS data services · Processing and compute · about 12 min read
Amazon EC2 (as Databricks classic compute)
The virtual machines in your own AWS account that Databricks classic clusters run on: you pick the family, size, Spot share and pools.
In plain words
What it is
Amazon Elastic Compute Cloud, or EC2, rents out virtual machines by the second. A virtual machine, called an instance, is a slice of a physical server with a fixed number of processor cores (vCPUs) and a fixed amount of memory. You choose an instance type, such as r7g.2xlarge, and AWS starts it in one of your networks. When you stop it, the bill stops.
On this site EC2 matters for one reason: Databricks classic compute. When you create a classic cluster, Databricks asks AWS to start EC2 instances inside your own account and your own virtual network (VPC). One instance becomes the driver, which plans the work, and the others become workers, which do it. Databricks charges its own units, called DBUs, for the software, and AWS charges for the EC2 instances, their disks and the network traffic. Serverless compute is different: the machines run in a Databricks account, and one DBU price covers both.
A concrete example. A tyre maker runs a nightly job that rebuilds a year of curing-press sensor readings for a scrap-rate model: about 4 TB, three hours, every night. On classic compute, the platform team picks memory-optimised Graviton workers, lets most of them be Spot (spare AWS capacity sold at a deep discount that AWS can take back), keeps the driver on normal on-demand pricing, and starts the cluster from a pool of warm instances. Every one of those choices changes the bill and the risk of a failed night.
Serverless is a taxi you pay by the trip; classic compute on EC2 is a rented van fleet, cheaper per mile if you choose the vans well, park them when idle and accept that some are on short-notice loan.
Big picture
Where it fits
The lit lane is where EC2 works. Boxes on the left feed it; boxes on the right use what it produces.
EC2 sits in the processing lane, underneath Databricks. Nothing in the data flow names it directly: data lands in S3 from DMS, AppFlow, Transfer Family, Kinesis or Firehose, and Databricks jobs and pipelines read it into bronze, silver and gold. Whether those jobs run on serverless compute or on classic clusters is a compute policy choice, and only the classic choice puts EC2 instances in your account.
The reference platform uses serverless by default for SQL, jobs and pipelines. Classic compute stays for two cases. The first is reach: a job must call a private system, such as a plant historian or an on-premises SAP gateway, over Direct Connect or VPN that serverless cannot reach yet. The second is price: a long, steady, large job where the cost calculator and the billing tables show that Spot workers on classic are clearly cheaper than serverless.
When classic is used, the clusters live in private subnets across two availability zones with no public IP addresses, reach S3 through a gateway endpoint and Kinesis and STS through interface endpoints, and talk to the Databricks control plane over PrivateLink. Cluster policies fix instance families, Spot share, auto-termination and tags, so cost shows up per team, per data product and per hop in both the Databricks billing tables and the AWS Cost and Usage Report.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Instance family | A group of instance types with the same shape. M is general purpose (about 4 GB of memory per vCPU), R is memory optimised (about 8 GB per vCPU), C is compute optimised (about 2 GB per vCPU). A trailing d, as in r7gd, means fast local NVMe disks. |
| Graviton | AWS's own Arm processors, shown by a g in the name (m7g, r7g, c7g). AWS positions them as the best price-performance on EC2. Databricks supports them with some limits: you cannot mix Arm and x86 nodes in one cluster, and a few features are not supported. |
| Driver and workers | The driver plans a Spark job and collects results; workers run the tasks. Losing the driver kills the job, losing a worker only slows it. |
| Spot instance | Spare EC2 capacity at a deep discount that AWS can reclaim with a two-minute warning. Good for workers, never for the driver. |
| First on-demand | A cluster setting that keeps the first nodes, including the driver, on normal pricing and makes the rest Spot, with a fallback to on-demand if Spot is not available. |
| Fleet instance type | A Databricks size name such as m-fleet.xlarge that resolves at launch to whichever instance type of that size has capacity, using AWS's Spot placement score to pick the zone. It cuts capacity failures; it does not support GPUs. |
| Instance pool | A set of idle, ready instances that clusters take from and return to. It shortens start-up from minutes to seconds. Idle instances in a pool still cost EC2 money, though not DBUs. |
| Availability zone (auto-AZ) | Databricks can pick the zone with the most free IP addresses and retry other zones on a capacity error, but only at start-up. |
| Cluster policy | A Databricks rule set that limits what users may choose: instance types, Spot share, maximum size, auto-termination and required tags. |
| vCPU quota | AWS caps how many vCPUs of on-demand and of Spot instances an account may run per region. A large cluster can fail to start because of this cap, not because of Databricks. |
Choosing
When to use it, and when not
Use it when
- Use classic compute on EC2 when a job must reach a private network, such as a plant historian or an SAP gateway, that serverless compute cannot reach yet in your region.
- Use it for long, steady, large batch jobs where the calculator and a month of billing data show Spot workers are clearly cheaper than serverless.
- Use it when you need an instance shape that serverless does not offer, such as GPU instances for model training or very large memory for a one-off backfill.
- Use it when you need a custom container image, init scripts or a library that only installs on a machine you control.
- Use Graviton instances for classic Spark and Photon work once your libraries are known to run on Arm.
Avoid it when
- Avoid classic compute for bursty, short or scheduled jobs; start-up time, idle time and NAT traffic often cost more than the lower DBU price saves.
- Avoid it for interactive notebooks left open all day; serverless notebooks or short auto-termination stop that bill.
- Avoid Spot for the driver node or for short jobs with a hard deadline, where one reclaimed node can miss the window.
- Avoid picking classic everywhere just because the DBU price is lower; the site's cost page marks that choice Drop.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Spot reclaimed mid-job | Workers disappear with two minutes' notice; Spark recomputes lost tasks and the job runs late or fails at a shuffle-heavy stage. | Keep the driver and a few workers on-demand (first on-demand), allow fallback to on-demand, use fleet types to widen the choice, and keep Spot for jobs that can tolerate a rerun. |
| Insufficient capacity at start-up | The cluster fails with an insufficient-capacity error for one instance type in one zone, often at the busiest hour. | Use fleet instance types or flexible node types, let auto-AZ choose the zone, and spread nightly start times. |
| vCPU quota hit | A cluster or a backfill stops growing because the account reached its on-demand or Spot vCPU quota for standard instances. | Track the quota against peak usage, raise it through Service Quotas before big backfills, and cap cluster size with policies. |
| Idle and start-up cost | Clusters bill EC2 while booting and while idle before auto-termination; all-purpose clusters left running overnight dominate the bill. | Short auto-termination by policy, job clusters instead of all-purpose clusters for scheduled work, pools with a small minimum idle, or serverless. |
| NAT gateway charges | S3 reads from private subnets go through the NAT gateway and are charged per gigabyte, which can rival the EC2 cost. | Add the S3 gateway endpoint and interface endpoints for STS and Kinesis so data traffic never touches the NAT gateway. |
| Subnet IP exhaustion | Each node takes IP addresses; large autoscaling clusters run out of addresses in small subnets and fail to grow. | Size subnets for peak node count across both zones and watch free addresses. |
| Pools that never empty | A pool with a high minimum idle keeps instances running all night and weekend for jobs that only run in the morning. | Minimum idle of zero or a few, an idle timeout of about 20 minutes, and tags on the pool so its cost has an owner. |
| Wrong family for the work | Joins and merges spill to disk on compute-optimised nodes, or memory-heavy nodes sit idle on light parsing jobs. | Start with R or M families with local disks for merge-heavy silver jobs, check spill and CPU in the Spark UI, then resize. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick EC2 when |
|---|---|---|
| Databricks serverless compute | The work is bursty, scheduled or interactive, and you want no clusters to size, patch or keep warm. This is the default on this site. | A job needs private network reach serverless lacks, a custom image, GPUs, or a long steady run where Spot is proven cheaper. |
| Amazon EMR on EC2 or EMR Serverless | The company already runs Spark on EMR, or needs open-source engines such as Hive, Trino or HBase outside Databricks. | The processing runs in Databricks and must be governed by Unity Catalog with one engine for batch and streaming. |
| AWS Glue ETL | Small, AWS-only jobs where an existing Glue estate is already maintained. | You want Photon, Delta features, Unity Catalog lineage and full control of instance shape. |
| Azure Databricks classic compute on Azure VMs | The platform runs on Azure; the same choices (spot VMs, pools, policies) apply with Azure VM families. | The platform runs on AWS. |
Cost
How you pay
Classic compute has two bills. Databricks charges DBUs per hour for each node, at a lower DBU rate than serverless. AWS charges for each EC2 instance per second while it runs (after a one-minute minimum for Linux), plus EBS disks, NAT gateway processing and cross-zone traffic. On-demand is the full price; Spot is a variable discount on spare capacity; Savings Plans and Reserved Instances lower the on-demand price in return for a one or three year commitment. Idle pool instances cost EC2 but not DBUs.
What drives the bill
- Instance hours: node count times run time, including start-up and idle time before auto-termination.
- Instance family and generation: Graviton and newer generations usually give more work per dollar.
- Spot share and how often Spot is reclaimed, which adds rerun time.
- Network: NAT gateway per-gigabyte processing when S3 traffic has no gateway endpoint, and cross-zone traffic during shuffles.
- Pools' minimum idle instances and their idle timeout.
Cost traps
- Comparing only the DBU price of classic and serverless and forgetting the EC2, start-up, idle and NAT costs.
- All-purpose clusters used for scheduled jobs, which bill at a higher DBU rate and often stay up.
- Savings Plans sized for today's classic usage that sit unused after more jobs move to serverless.
- Untagged clusters and pools, so the EC2 lines in the Cost and Usage Report cannot be joined back to a team or data product.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Classic clusters run in private subnets with no public IP addresses; the Databricks control plane is reached through back-end PrivateLink and users reach the workspace through front-end PrivateLink.
- Each cluster gets an instance profile, an IAM role the machines assume. Keep it minimal; data access should go through Unity Catalog storage credentials, not broad S3 rights on the instance profile.
- EBS volumes and local disks are encrypted; use your own KMS key for managed disks where policy requires it, and enable encryption between worker nodes if traffic must be encrypted inside the VPC.
- Use instance metadata service version 2 (IMDSv2) and cluster policies so nobody can start an unapproved instance type, a public IP or an init script from an unknown location.
- CloudTrail records every RunInstances and TerminateInstances call made by the Databricks cross-account role; Databricks audit logs record who started which cluster.
- Personal data in shuffle files and spill sits on encrypted local disks only for the life of the cluster; short-lived job clusters reduce how long it lives there.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| Cluster events (system.compute tables) | Start-up time, resizes, Spot losses and terminations per cluster; the first place to see capacity and Spot trouble. |
| system.billing.usage joined with the CUR EC2 lines | DBUs and EC2 dollars per tag, so each team and hop sees its full classic cost. |
| Spot interruption notices (EC2 Spot Instance Interruption Warning in EventBridge) | How often workers are reclaimed and in which zones and families. |
| InsufficientInstanceCapacity errors | Start-up failures that point to a narrow instance choice or a busy zone. |
| Service Quotas usage for running on-demand and Spot standard vCPUs | Headroom before a large backfill or a new team's clusters hit the account cap. |
| NAT gateway BytesOutToDestination | A rise means S3 or other AWS traffic is bypassing the VPC endpoints. |
| Limit | Value and notes |
|---|---|
| On-demand vCPU quota | Counted in vCPUs per region for standard families (A, C, D, H, I, M, R, T, Z); new accounts start low and AWS raises it with usage or on request. |
| Spot vCPU quota | A separate vCPU quota per region for all standard Spot requests. |
| Spot interruption notice | Two minutes before AWS reclaims a Spot instance. |
| Graviton on Databricks | Graviton2 and Graviton3 documented; no mixing of Arm and x86 in one cluster; some features such as Databricks Container Services not supported. |
| Fleet instance types | No GPU support; the Spot bid percentage setting has no effect. |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Nightly 3-hour backfill of plant sensor history | Classic job cluster, Graviton R family, driver on-demand, workers mostly Spot with fallback | Pursue with care: only once a month of billing shows it is cheaper than serverless, and with a rerun plan. |
| Hourly silver merges of SAP orders | Serverless jobs | Pursue: short, frequent runs pay more in start-up and idle time on classic than they save. |
| Job that reads a plant historian over Direct Connect | Classic job cluster in private subnets | Pursue with care: needed for private reach today; recheck when serverless private connectivity covers it. |
| Classic everywhere because the DBU price is lower | All clusters on EC2 | Drop: EC2, start-up, idle and NAT costs come on top, as the cost page shows. |
| Pools for morning interactive work | One pool per common instance type, minimum idle low, idle timeout about 20 minutes | Keep if you have it: Databricks now recommends serverless instead of pools where the workload allows. |
FAQ
Questions people ask
Do Databricks clusters run on my EC2 instances?
Classic clusters do: Databricks starts EC2 instances in your account and VPC and you pay AWS for them, plus DBUs to Databricks. Serverless compute runs in a Databricks-owned account in the same region, and its DBU price includes the machines.
Should I use Graviton instances for Databricks?
Usually yes for classic Spark and Photon jobs, because Graviton gives more work per dollar. Check that your libraries run on Arm and that you don't need features unsupported on Graviton, and don't mix Arm and x86 nodes in one cluster.
Is it safe to use Spot instances for Databricks jobs?
For workers, yes, if the job can absorb a lost node and rerun. Keep the driver on on-demand, allow fallback to on-demand, and avoid Spot for short jobs with a hard deadline.
What is a Databricks fleet instance type?
A size name such as m-fleet.xlarge that resolves at launch to an available instance type of the same size, picking the zone with the best Spot placement score. It reduces capacity errors; it doesn't support GPUs.
Do idle instances in a Databricks pool cost money?
Yes. Idle pool instances are running EC2 instances, so AWS bills them, though Databricks does not charge DBUs for them. Keep the minimum idle low and the idle timeout short.
Sources
Checked against
- Databricks docs: Compute configuration reference (AWS)
- Databricks docs: Pool best practices
- AWS docs: Amazon EC2 instance type quotas
- AWS docs: Spot Instance interruptions
- AWS: Graviton processors
Checked 2026-11-11.
Related
Where it connects
Data platform on AWS and Databricks · Databricks cost calculator · VPC, endpoints and NAT · IAM and STS · Amazon EMR · All AWS data services
← Amazon EMR · ECS and Fargate →