Home › AWS › AWS data services › Glue ETL jobs

AWS data services · Processing and compute · about 12 min read

AWS Glue ETL jobs (Spark, Python shell, streaming)

Serverless Apache Spark and plain Python jobs on AWS, billed per worker-second, for moving and reshaping data between stores.

RoleProcessing and compute
StatusGenerally available. Glue 6.0 (Spark 4.1.1, Python 3.13) is the newest version and 5.1 is the default for new jobs. Glue for Ray closed to new customers on 30 Apr 2026. Glue 0.9, 1.0 and 2.0 Spark jobs reached end of life on 1 Apr 2026, and Python shell 3.6 on 1 Jun 2026.
Verdict on this siteKeep if you have it
AzureAzure Data Factory data flows, Fabric Spark
Google CloudDataproc Serverless, Dataflow
DatabricksLakeflow Jobs and Lakeflow Declarative Pipelines on serverless compute
Open sourceApache Spark on any cluster
Verdict: Keep if you have it. Do not start new Glue jobs on this platform: Databricks is the one engine for batch and streaming. Existing Glue jobs are rebuilt as Databricks pipelines one subject at a time, run side by side for two weeks until counts and sums reconcile, then switched off. The Glue Data Catalog can still matter if Athena or Redshift read the same tables.

In plain words

What it is

AWS Glue is a family of data tools. This page is about one part of it: Glue ETL jobs. ETL stands for extract, transform and load, which means reading data from somewhere, changing its shape, and writing it somewhere else. A Glue job is a script, written in Python or Scala, that AWS runs for you on a temporary group of machines. You do not create or patch servers. You say how many workers you want, AWS starts them, runs the script, stops them, and bills you for the seconds they ran.

There are three job types. A Spark job runs Apache Spark, an engine that splits a big dataset into pieces and works on the pieces in parallel across many machines. A streaming job is a Spark job that never ends; it keeps reading from a stream such as Kinesis or Kafka and writes small batches as data arrives. A Python shell job runs a single plain Python script on a small slice of a machine, for small tasks such as calling an API or running a SQL statement in a warehouse. A fourth type, Glue for Ray, is closed to new customers since 30 April 2026.

A concrete example. A tyre maker exports SAP sales orders as CSV files every night into an S3 bucket. A Glue Spark job reads the new files, fixes the date formats, joins each order to the customer master, removes duplicate lines, and writes Parquet files partitioned by order date. A Glue crawler or the job itself then updates the Glue Data Catalog so that Athena can query the result. A feature called job bookmarks remembers which files were already processed, so the next night only new files are read.

A hired kitchen crew that arrives when you call, cooks exactly the recipe you hand them, and leaves; you pay by the minute, but you still write the recipe.

Big picture

Where it fits

SourcesMove inStreamStoreProcessServeUseAcross everything: catalog · security · orchestration · observability and cost · infrastructure as codegets data fromhands data toAWS DMSAppFlowKinesis Data StreamsGlue ETL jobsProcessing and computeS3Glue Data CatalogAthena

The lit lane is where Glue ETL jobs works. Boxes on the left feed it; boxes on the right use what it produces.

In a plain AWS data platform, Glue sits in the processing lane. Movers such as DMS, AppFlow and Transfer Family drop files into an S3 landing area, and Glue jobs turn landing into bronze, silver and gold tables. Glue Workflows or Step Functions start the jobs, and Athena, Redshift Spectrum or QuickSight read the results through the Glue Data Catalog.

On this site's reference platform, that processing job belongs to Databricks. Lakeflow Declarative Pipelines and Lakeflow Jobs read the landing folder, write Delta tables in the medallion layout, check quality rules, record lineage in Unity Catalog and run both batch and streaming on one engine. Running Glue alongside would mean two Spark engines, two ways to grant access, two places to look for failures and two bills to explain. So Glue is not part of the target design.

Glue still appears in two honest ways. First, many companies already have dozens or hundreds of Glue jobs. The migration plan rebuilds them as metadata-driven Databricks pipelines one subject at a time, for example sales orders first and then inventory, runs the old and new jobs in parallel for two weeks, compares row counts and sums, and only then switches the Glue job off. Second, the Glue Data Catalog, which is a separate part of Glue, may stay as the catalog that Athena and Redshift use, kept in step with Unity Catalog through catalog federation during the move.

How it works

The ideas you need

IdeaWhat it means
DPUData processing unit, the measure Glue bills by. One DPU is 4 vCPUs and 16 GB of memory. A G.1X worker is one DPU, a G.2X worker is two.
Worker typeThe size of each machine in the job. G.025X for small streams, G.1X and G.2X for most work, G.4X to G.16X for heavy joins, and R types with extra memory for jobs that run out of memory.
Glue versionThe bundle of Spark, Python and libraries the job runs on. 5.1 is the default; 6.0 brings Spark 4.1.1. Old versions reach end of support and then end of life, when runs stop.
DynamicFrameGlue's own wrapper around a Spark DataFrame that tolerates columns whose type changes between files. Handy for messy input, but it ties the code to Glue.
Job bookmarkState that Glue keeps between runs so a job only reads files or rows it has not seen before. It is how most Glue jobs become incremental.
Flex executionA cheaper class for jobs that are not urgent. Glue runs them on spare capacity, so start times and run times can stretch. Only G.1X and G.2X workers.
Auto scalingGlue adds and removes workers during a run, up to the number you set, and bills each worker only for the time it ran.
ConnectionA stored definition of how to reach a database or a private network, including the VPC subnet and security group the workers join.

Choosing

When to use it, and when not

Use it when

  • You already run Glue jobs that work, and the plan is to keep them stable until each subject is rebuilt on Databricks.
  • A team outside the data platform needs a small, occasional Spark job in an AWS-only account where Databricks is not available.
  • You need a quick Python shell job that calls an AWS API or runs a SQL statement, and the account has no other scheduler.
  • A job must write to the Glue Data Catalog for Athena users, and the team has no plan to adopt Unity Catalog.
  • You want serverless Spark with no cluster to size, and the workload is a few jobs, not a platform.

Avoid it when

  • Avoid building new pipelines in Glue on a platform where Databricks already does processing; you would run two engines with two access models.
  • Avoid Glue streaming jobs for always-on, low-latency work; a job that never ends bills every worker for every second of the month.
  • Avoid DynamicFrame-heavy code if you may move later; plain Spark DataFrame code moves to Databricks or EMR with far fewer changes.
  • Avoid Glue for Ray; it is closed to new customers and AWS points users toward Ray on Amazon EKS instead.
  • Avoid using Glue as the place where business rules live if lineage, quality results and access must be shown in one catalog.

Production grade

What goes wrong in production

ProblemWhat breaksWhat to do
Version end of lifeWhen a Glue version reaches end of life, its jobs can no longer start. Glue 0.9, 1.0 and 2.0 Spark jobs stopped on 1 Apr 2026.Keep an inventory of jobs by GlueVersion. Upgrade one version at a time with a parallel run, or use the end-of-life date as the deadline to move the job to Databricks.
Bookmarks miss or repeat dataA bookmark is reset, a file lands late with an old timestamp, or a script lacks the transformation_ctx name, and files are skipped or read twice.Make writes idempotent with a merge on a business key. Log file names and row counts per run. Never reset a bookmark without a written backfill plan.
Out of memory on large joinsThe driver or an executor dies with an out-of-memory error, often on a skewed key such as one huge customer or plant.Use R worker types or a larger G type for that job only, repartition on the join key, and broadcast the small side. Check the skew in the Spark UI first.
Slow cold starts and capacity waitsLarger worker types and R types start more slowly, and Flex runs can wait for spare capacity, so a nightly load misses its window.Use standard execution for jobs with a deadline. Turn on job run queuing so runs wait instead of failing. Alert on time to start, not only on failure.
Default timeout cuts long jobsFrom Glue 5.0 the default timeout is 480 minutes. A job upgraded from 4.0, which defaulted to 2,880 minutes, is stopped mid-run.Set the timeout explicitly on every job, close to its expected run time plus margin. The maximum is 7 days.
VPC IP exhaustionJobs that join a VPC take one IP address per worker. A busy night with many jobs runs the subnet out of addresses and runs fail.Give Glue its own large subnets, or a secondary CIDR range. Watch free IP addresses per subnet.
Logic scattered across scriptsEach job has its own copy of cleaning rules, and nobody can say which rule changed a number.Treat the migration as the fix: one metadata-driven framework, rules in config, quality results stored with a run id.

Alternatives

What to compare it with

AlternativePick the alternative whenPick Glue ETL jobs when
Databricks Lakeflow Declarative Pipelines and JobsYou run Databricks: one engine for batch and streaming, quality rules, lineage and access in Unity Catalog, and serverless compute. This is the reference choice.The account has no Databricks and the need is a handful of small jobs.
Amazon EMR (Serverless, on EC2, on EKS)You need open-source engines beyond Spark, such as Trino, Hive or HBase, or full control of Spark settings and instance types.You want no cluster or application to manage and a simpler per-job setup.
Amazon Athena with dbtThe transformations are plain SQL over S3 tables and the team thinks in SQL, not in Spark.You need Python logic, complex parsing or machine learning steps.
AWS LambdaThe task is small, event-driven and finishes in well under 15 minutes, such as unzipping one file or calling one API.The data is large enough to need parallel processing across machines.
Azure Data Factory data flows or Fabric SparkYour platform is on Azure; these are the closest equivalents for visual and Spark-based transforms.Your data and accounts are on AWS.

Cost

How you pay

Glue bills Spark and streaming jobs by DPU-hour, charged per second with a one-minute minimum on current versions. The bill for a run is the number of DPUs times the time they ran, so a job with ten G.1X workers for six minutes uses one DPU-hour. Flex execution charges a lower rate per DPU-hour for jobs that can wait. Python shell jobs use either one-sixteenth of a DPU or one DPU. Interactive sessions are billed the same way while the session is open. The Data Catalog and crawlers are billed separately. Check the Glue pricing page for your region's list price.

What drives the bill

  • The number and size of workers you request; most jobs are over-provisioned because nobody measures them.
  • Run time, which grows with skew, small files and full re-reads when bookmarks are not used.
  • Streaming jobs, which bill every worker for every second because they never stop.
  • Retries: a job set to retry three times can cost four runs on a bad day.
  • Crawlers that rescan whole buckets on a schedule.

Cost traps

  • A streaming job left running in a dev account bills all month even with no data arriving.
  • Interactive sessions left open after a notebook is closed keep billing until their idle timeout.
  • Large worker types chosen to fix one out-of-memory error, then copied to every job as a template.
  • Running old and new pipelines in parallel during migration doubles cost for that subject; keep the window to the agreed two weeks.
  • NAT gateway charges for jobs in a VPC that reach S3 without an S3 gateway endpoint.

Prices change; check the AWS pricing page for your region before deciding.

Security and governance

Keeping it safe

  • Each job runs as an IAM role. Give each job, or each subject, its own role scoped to its own buckets and prefixes, not one shared role with S3 full access.
  • Use a Glue security configuration to encrypt job output, CloudWatch logs and job bookmarks with KMS keys you own.
  • Jobs that reach databases join your VPC through a Glue connection. Keep them in private subnets with no public IP and reach S3, STS and Glue through VPC endpoints.
  • Store database passwords in Secrets Manager and reference them from the connection; never put them in job parameters, which are visible in the console and logs.
  • Glue may create a temporary bucket for scripts and temp files that can allow public access; turn on S3 Block Public Access at the account level.
  • CloudTrail records job creation, changes and runs. Personal data such as customer names should be tokenised before it reaches any Glue output that many people can read.

Operating it

What to watch, and the limits

WatchWhy
Job run state and durationFailed, timed-out and unusually long runs; the first sign a source or volume has changed.
glue.driver.aggregate.numFailedTasksFailed Spark tasks within a run that later succeeds; a hint of skew or bad input.
glue.ALL.jvm.heap.usageMemory pressure on the driver and executors before a job dies of out-of-memory.
glue.driver.ExecutorAllocationManager.executors.numberAllExecutorsHow many executors auto scaling actually used, to right-size the requested workers.
DPU-hours per job from billing tagsCost per job per day, so that the costliest jobs are migrated or tuned first.
Streaming batch processing timeFor streaming jobs, whether each micro-batch finishes inside its window or falls behind.
LimitValue and notes
Concurrent job runs per account2,000 by default per region; adjustable.
Jobs per account2,000 by default per region; adjustable.
Concurrent DPUs per account1,000 in the largest regions and 500 elsewhere, by default; adjustable.
Job timeoutUp to 7 days (10,080 minutes); default 480 minutes from Glue 5.0.
Retries per job0 to 10 automatic retries; runs that hit the timeout are not retried.
Worker type availabilityG.12X, G.16X and R types need Glue 4.0 or later and are only in some regions; Flex supports only G.1X and G.2X.

Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.

Decisions

Situations and verdicts

SituationChoiceVerdict
New pipeline for plant sensor data on the reference platformDatabricks streaming table on serverless computeDrop Glue for this: Databricks is the one engine for batch and streaming, with quality and lineage in Unity Catalog.
200 existing Glue jobs feeding Athena reportsKeep them running; rebuild one subject at a time on Databricks, two-week parallel run, then switch offKeep if you have it: stable jobs keep the business running while the move happens in measured steps.
Glue 2.0 jobs still in the account after 1 Apr 2026They no longer run; move them straight to Databricks rather than upgrading twicePursue with care: treat end of life as the migration deadline, and check nothing downstream silently stopped.
Small API call each night to fetch exchange ratesEventBridge Scheduler and Lambda, or a small Databricks jobDrop Glue Python shell here: a whole job type for one call is more to run than it is worth.

FAQ

Questions people ask

Is AWS Glue being deprecated?

No. Glue ETL is generally available and Glue 6.0 with Spark 4.1.1 is the newest version. What ends are old versions: Glue 0.9, 1.0 and 2.0 Spark jobs reached end of life on 1 April 2026, and Glue for Ray closed to new customers on 30 April 2026.

Should I use Glue or Databricks for ETL on AWS?

If the platform runs Databricks, use Databricks for new work so that one engine, one catalog and one access model cover everything. Glue fits small, AWS-only needs, and existing Glue jobs can keep running until they are migrated.

Can Databricks read tables that Glue jobs write?

Yes. Databricks can read Parquet, Iceberg or Delta files in S3, and Unity Catalog can federate the Glue Data Catalog so Glue tables appear during a migration. That makes a side-by-side run straightforward.

What is the difference between a Glue job and a Glue workflow?

A job is one script that processes data. A workflow is a chain of jobs, crawlers and triggers that run in order. On the reference platform, Lakeflow Jobs replaces workflows, and Glue Workflows are marked Drop.

How do I make Glue jobs cheaper?

Measure actual executor use and cut the worker count, use Flex for jobs that can wait, use bookmarks so each run reads only new data, stop idle streaming jobs, and add an S3 gateway endpoint so traffic does not pay NAT charges.

Sources

Checked against

Checked 2026-11-11.

Related

Where it connects

Data platform on AWS and Databricks · Medallion architecture · AWS Glue Workflows · AWS Glue Data Catalog · Amazon EMR · Databricks cost calculator · All AWS data services

← EKS · Lambda →