Home › Data engineering › Iceberg lakehouse on AWS with Databricks
Data engineering · free guide · one page
Iceberg lakehouse on AWS with Databricks: who should own the catalog?
You want an open lakehouse on AWS, with Apache Iceberg as the table format and Databricks in the picture. AWS offers S3 Tables, Glue, Lake Formation and the SageMaker lakehouse; Databricks offers Unity Catalog. The names spin heads, but there is one decision underneath: which catalog owns the tables. This page draws the whole platform first, then the two physical designs, then compares them concern by concern and picks a winner.
Step 1 · the conceptual big picture
The whole data platform, with every concern
Before any product name, here is the platform as jobs. Data comes from sources on the left, moves along one line (ingest, bronze, silver, gold, semantic) and reaches consumers on the right. Under the line sit five layers every table needs. Along the bottom are the 24 concerns that cut across every hop; each is explained on the data platform page.
Look at the thick box. Storage, the table format and compute are easy to agree on. Who owns the catalog decides who can write, who sets permissions, who sees lineage and who cleans the tables.
Why the catalog matters so much: an Iceberg write only becomes real when the catalog swaps its pointer to the new metadata file. Whoever holds that pointer controls writes, permissions and clean-up. Every other engine is a guest. See Who maintains your tables for the picture.
Step 2 · the AWS names, decoded
Six names, three jobs
Several of these are layers of the same thing, which is why they confuse. Storage is S3 Tables. Catalog and permissions are Glue plus Lake Formation, sold together as the SageMaker lakehouse. Unity Catalog does both catalog and permissions on the Databricks side.
| Name | What it really is |
|---|---|
| Amazon S3 Tables | Storage: a table bucket that holds Iceberg tables and maintains them itself (compaction, snapshot expiry, unreferenced file removal). |
| AWS Glue Data Catalog | The catalog: which tables exist and, for Iceberg, the pointer to each table's current metadata file. Every AWS engine looks tables up here. |
| AWS Lake Formation | Permissions on the Glue catalog: table, column, row and cell grants, LF-Tags, and short-lived credentials handed to engines. |
| SageMaker lakehouse | AWS's umbrella name for Glue Data Catalog plus Lake Formation, reaching across S3, S3 Tables, Redshift storage and federated catalogs, with an Iceberg REST endpoint. |
| SageMaker Catalog | A different thing: the business catalog (glossary, data products, access requests) on top. Formerly DataZone. |
| Databricks Unity Catalog | Databricks' catalog, permissions, lineage, audit and sharing in one, with its own Iceberg REST catalog for outside engines. |
Step 3 · physical design B
Architecture B: AWS owns the catalog, Databricks is a guest
Iceberg tables live in an S3 table bucket. The Glue Data Catalog holds each table's pointer and Lake Formation holds the permissions. S3 Tables compacts, expires snapshots and removes unreferenced files by itself. AWS engines write and read natively.
Owners on the left write through Glue. AWS readers on the right read through Glue. Databricks, dashed, is a guest.
Can Databricks do bronze to silver to gold here?
This is the question most teams get wrong. Through Unity Catalog, AWS-owned Iceberg tables appear as foreign tables, which are read-only. The only way for Databricks to write them today is around Unity Catalog. Four ways, with verdicts:
| Way | How it works | What you give up | Verdict |
|---|---|---|---|
| B1. AWS engines do all the ETL; Databricks only reads | Glue ETL or EMR Spark writes bronze, silver and gold into S3 Tables. Databricks reads the tables through Unity Catalog federation for BI, ML or Genie. | Clean, but Databricks is a reader you pay for | Pursue when Databricks is a light guest |
| B2. Databricks writes through the Glue Iceberg REST endpoint | A classic Databricks cluster (dedicated access mode) loads the open-source Iceberg library and points a Spark catalog at the Glue Iceberg REST endpoint, signed with the cluster's IAM role; Lake Formation hands out credentials. Bronze to silver to gold runs as ordinary Spark jobs. | Works. But no serverless, no Lakeflow pipelines, no Unity Catalog permissions, lineage or audit on these writes, and you keep Iceberg library versions in step yourself | Pursue only for a short migration or a few jobs |
| B3. Databricks writes to foreign tables through Unity Catalog | Unity Catalog shows Glue tables as foreign Iceberg tables. Today these are read-only in Databricks, with no partition evolution. | Not available as of November 2026; check the Databricks docs, this changes | Drop for now |
| B4. Databricks owns bronze and silver; AWS owns gold | Split ownership by layer: Databricks writes Unity Catalog tables, then an AWS job copies or rebuilds gold into S3 Tables. | Two catalogs, a copy, two permission systems, lineage breaks at the seam | Drop |
Databricks' support for foreign Iceberg tables changes every few months. Check the current docs before you rule writes in or out.
Step 4 · physical design A
Architecture A: Unity Catalog owns the catalog, AWS engines are guests
Iceberg tables live in your own S3 bucket as Unity Catalog managed Iceberg tables. Unity Catalog holds the pointer, the grants, the lineage and the audit. Predictive Optimization does the chores. Databricks writes every layer on serverless compute.
Owners on the left write through Unity Catalog. Guests on the right read in place, and Iceberg engines can even write, through Unity Catalog's Iceberg REST catalog.
And the other way round: can AWS engines work here?
| Who | How it works | Limit | Verdict |
|---|---|---|---|
| Athena, Redshift, QuickSight read gold | Glue catalog federation to Unity Catalog fetches the table's current metadata from Unity Catalog's Iceberg REST endpoint; Lake Formation hands out short-lived credentials; files are read in place, no copy. | Read-only for these engines; grant Lake Formation SELECT on the target tables | Pursue |
| EMR, Glue Spark, Trino, Snowflake write managed Iceberg | They connect to Unity Catalog's Iceberg REST catalog with a Databricks service principal and write managed Iceberg tables directly, governed by Unity Catalog grants. | Their writes are not in Unity Catalog lineage; Predictive Optimization still maintains the table | Pursue only where an AWS engine really must write |
| Firehose or DMS write into the lakehouse | They land files in a landing bucket; Auto Loader or Lakeflow Connect loads them into bronze. | One extra hop of seconds to minutes | Pursue |
Step 5 · the asymmetry
Who can write where
This grid is the heart of the decision. Unity Catalog lets outside Iceberg engines write managed tables through its REST catalog, under its own grants. Glue lets Databricks write only around Unity Catalog. So A keeps the door open both ways; B closes Databricks' governed door.
Green: full, governed. Butter: works with a limit. Rose: read-only, or writes that skip governance.
Step 6 · concern by concern
A and B compared on every concern
Every one of the 24 platform concerns is here (their ids in grey), plus the serving, ingestion and strategy concerns. Green works fully, yellow works with a limit, rose does not. Checked against AWS and Databricks docs in November 2026; Iceberg support on both sides changes often.
| Concern | A: Unity Catalog owns | B: AWS owns | Better |
|---|---|---|---|
| Tables and maintenance | |||
| Storage and table format | Managed Iceberg tables in your own S3 bucket, at a path Unity Catalog picks under the managed location | Iceberg in an S3 table bucket, reached through table APIs; or Glue Iceberg in your bucket | Tie |
| Maintenance (compaction, snapshot expiry, orphan files, statistics, clustering) X19 | Predictive Optimization, automatic and required on managed Iceberg; runs on serverless, billed as DBUs | S3 Tables, automatic and on by default; billed per object and byte processed. Predictive Optimization cannot run on these tables | Tie: both automatic, one owner each |
| Layout: catalogs, schemas, names X23 | catalog.schema.table names encode environment, layer and domain; one naming scheme | Table bucket, namespace, table; seen from Databricks through a federated catalog with a second set of names | A |
| ETL | |||
| Ingestion (sources to bronze) | Auto Loader, Lakeflow Connect, Structured Streaming; AWS services land files that Auto Loader picks up | Firehose and DMS write Iceberg directly; Glue jobs; Databricks only from classic clusters | Tie: each is native on its own side |
| Bronze to silver, silver to gold | Databricks jobs, SQL and Lakeflow on serverless; EMR or Glue Spark can also write through Unity Catalog's Iceberg REST | Glue ETL or EMR Spark natively; Databricks only on classic clusters through the Glue REST endpoint, ungoverned by Unity Catalog | A |
| Orchestration X2 | Lakeflow Jobs with table-update triggers on owned tables | Step Functions or Airflow (MWAA) for AWS jobs; Databricks jobs called from them; two schedulers to watch | A |
| Metadata & control X1 | One catalog to read every table's schema, owner and tags from | Glue holds the tables; Databricks-side settings live in a second place | A |
| Replay & backfill X15 | Re-runs and MERGE on Databricks; time travel on snapshots | Re-runs on Glue or EMR; Databricks re-runs only from classic clusters | A |
| Streaming | |||
| Streaming lane X24 | Structured Streaming and Lakeflow are strong, but streaming tables and materialized views are Delta, so the fast lane is Delta even when gold is Iceberg | Firehose and Managed Flink write Iceberg into S3 Tables natively; Databricks streaming into them is not practical | B if streams are AWS-native; A if Databricks streams |
| Semantic layer | |||
| Governance & semantics: owners, glossary, metric definitions X13 | Unity Catalog metric views, tags, owners and certified tables, used by Genie and AI/BI dashboards | No lake-wide metric layer on AWS: QuickSight datasets, Redshift views, SageMaker Catalog glossary; Databricks metric views would sit over read-only foreign tables | A |
| Serving | |||
| BI and SQL | Databricks SQL warehouses and AI/BI; Athena, Redshift and QuickSight read gold through federation | Athena, Redshift Serverless and QuickSight natively; Databricks SQL reads through federation | Tie |
| Apps and APIs | Lakebase (managed Postgres) synced from Unity Catalog tables, Databricks Apps | DynamoDB or Aurora fed by Glue or EMR jobs, API Gateway and Lambda | Tie |
| AI agents and ML | Genie, Vector Search, Model Serving, MLflow, features and models governed in Unity Catalog next to the data | Bedrock and SageMaker AI read through Athena or Redshift; Databricks ML reads foreign tables but keeps models and features in Unity Catalog, so governance splits | A |
| Data sharing | Delta Sharing, open and cross-cloud; Iceberg REST for engines | Lake Formation across AWS accounts (RAM), Redshift data sharing | A outside AWS; tie inside AWS |
| Cost | |||
| Cost & FinOps X10 | One main bill (DBUs plus S3); serverless everywhere; Athena per TB for AWS readers; one FinOps view from system tables | S3 Tables storage costs a little more per GB than S3 Standard, plus monitoring and compaction fees; Glue and EMR compute; Databricks on classic clusters only; two bills, two FinOps views | A if Databricks does most work; B if Databricks is light or absent |
| Performance & scale X19 | Photon, liquid clustering and Predictive Optimization on owned tables | AWS engines tuned on their own tables; Databricks reads foreign tables without control of their layout | A |
| Governance and security | |||
| Security & access X11 | Unity Catalog grants and tag-based policies; Lake Formation only at the read door for AWS readers of gold | Lake Formation for AWS engines; Unity Catalog grants on the foreign catalog for Databricks readers; Databricks writers (B2) bypass Unity Catalog | A |
| Privacy & PII X12 | Row filters and column masks are not supported on managed Iceberg today (they are on Delta); use dynamic views, or keep PII tables in Delta with Iceberg reads | Lake Formation row and cell filters for AWS engines; outside engines with vended credentials generally cannot read filtered tables | Tie: both partial with Iceberg |
| Audit & run log X14 | Unity Catalog audit and access system tables for all Databricks work; CloudTrail for AWS reads | CloudTrail and Lake Formation for AWS; Databricks audit for its share; two trails to join | A |
| Lineage | |||
| Lineage X7 | Automatic table and column lineage for every Databricks job; writes by outside engines through Iceberg REST are not recorded | No automatic lineage in Glue or Lake Formation; SageMaker Catalog lineage needs OpenLineage set up; B2 Databricks writes are invisible to both | A |
| Trust | |||
| Contracts & schema X3 | Contract checks in code; schema enforced on write (informational keys not on managed Iceberg) | Contract checks in code; Glue schema | Tie |
| Data quality X4 | Lakeflow expectations, DQX, data quality monitoring on Unity Catalog tables | Glue Data Quality (DQDL), Deequ | Tie |
| Profiling X5 | Data profiling on Unity Catalog tables; check Iceberg support | Glue Data Quality rule recommendations | Tie |
| Reconciliation X6 | Engine-neutral checks; one catalog to compare against | Engine-neutral checks | Tie |
| See | |||
| Logs, metrics, traces X8 | System tables for jobs, queries and billing in one place | CloudWatch for AWS jobs plus Databricks system tables; two places | A |
| Freshness & SLOs X9 | Table-update triggers and freshness monitors on owned tables | EventBridge events from Glue and S3 Tables; Databricks cannot trigger on foreign table updates | A |
| Run | |||
| Environments & CI/CD X16 | Databricks bundles plus Terraform for the bucket and IAM | CloudFormation, CDK or Terraform for S3 Tables, Glue and Lake Formation, plus bundles for Databricks; two toolchains | A |
| Testing X17 | Table-driven tests, zero-copy clones for test data | Table-driven tests; Iceberg branches and tags for test data | Tie |
| Resilience & DR X18 | Time travel, UNDROP for 7 days; plan how catalog metadata is rebuilt in another region | Snapshots and AWS-managed durability; plan cross-region copies of the table bucket | Tie |
| Self | |||
| Self-healing X20 | One catalog and one run log to detect and fix from | Two systems to watch and fix | A |
| Agent assistance X21 | Agents read one catalog, lineage and metric views | Agents must read two catalogs | A |
| Simulation X22 | Engine-neutral | Engine-neutral | Tie |
| Strategy | |||
| Lock-in and portability | Unity Catalog runs on AWS, Azure and Google Cloud; but it picks managed table paths, so leaving means re-registering or rewriting | AWS only; S3 Tables data is reached only through table APIs | A |
| Team and support | One vendor for the data plane; Spark and SQL skills | AWS-native skills for the core, Databricks skills for the guest; two vendors to call | Depends on the team |
Step 7 · what each one restricts
What you accept with each
If you go with A (Unity Catalog owns)
- Athena, Redshift and QuickSight read gold through Glue federation and cannot write. EMR, Glue Spark or Trino can write managed Iceberg through Unity Catalog's Iceberg REST, but those writes are missing from Unity Catalog lineage.
- Lake Formation still appears, but only at the read door for AWS engines, on curated gold tables. Expose tables without row filters or masks there, or pre-filtered gold tables.
- Managed Iceberg today gives up row filters, column masks and informational keys; change data feed works only on Iceberg v3 and only for Databricks readers; Lakeflow streaming tables and materialized views are Delta. Keep those tables in Delta with Iceberg reads (UniForm), or use dynamic views.
- Check that managed Iceberg is generally available for your region and runtime before production; parts were still marked preview in the docs we checked.
- Firehose and DMS cannot write straight into Unity Catalog tables: they land files, and Auto Loader loads them.
- Databricks is the main bill. Predictive Optimization runs on serverless and shows up as DBUs.
- Unity Catalog picks the paths of managed tables. Leaving later means re-registering or rewriting tables.
If you go with B (AWS owns)
- Databricks sees AWS tables as foreign Iceberg tables: read-only, no partition evolution, and views defined on one side are not visible on the other.
- Databricks can do bronze to silver to gold only from classic clusters with the open-source Iceberg library pointed at the Glue REST endpoint: no serverless, no Lakeflow pipelines, no Unity Catalog permissions, lineage or audit on those writes.
- No Predictive Optimization, liquid clustering control, table-update triggers or Unity Catalog masks on these tables; Genie and metric views can only sit over read-only foreign tables.
- Permissions live in two systems: Lake Formation for AWS engines and Unity Catalog for Databricks readers. Tables with Lake Formation row or cell filters are generally not readable by outside engines.
- Lineage is fragmented: nothing automatic on the AWS side, and Databricks writes through the Glue endpoint are invisible to Unity Catalog.
- Two bills, two toolchains (CloudFormation or CDK and Databricks bundles), two monitoring places.
- AWS only. Moving to Azure, Fabric or Google Cloud later means a new catalog and new permissions.
Step 8 · evidence
Case studies: from a bad design to a good one
We found clear public write-ups of Architecture A (United Airlines, with Delta tables read as Iceberg) and of Architecture B without Databricks (Razor Group). We found no public case of B with Databricks as the main writer, which is itself a finding. Numbers come from the companies' or vendors' own blogs and are not audited.
United AirlinesArchitecture A, with Delta + Iceberg reads
Before. Before: petabytes in bronze, silver and gold, catalogued in Unity Catalog. Redshift could not see them, so tables were copied into Redshift storage with sync pipelines and a manually synced catalog, which AWS's write-up calls operationally expensive and prone to drift.
After. After: Databricks does all processing and owns the tables (Delta with UniForm). A Glue federated catalog fetches metadata from Unity Catalog's Iceberg REST endpoint, Lake Formation vends scoped credentials, and Redshift Serverless reads the files in place. About $30K a month of duplicate ETL removed; 30 tables in production, 70 rolling out, hundreds planned.
Lesson. Grant Lake Formation SELECT on the target tables, not on the resource link; roll out the busiest tables first; name external schemas so readers know the data is federated.
Razor GroupArchitecture B, without Databricks
Before. Before: more than 1,000 SQL models on always-on Redshift clusters, 98% of compute spent on batch ETL, data 4 to 6 hours old, workloads fighting for one cluster.
After. After: S3 Tables (Iceberg) in bronze, silver and gold, Glue and Lake Formation as catalog and permissions, Spark on EC2 Spot for ETL, Athena for ad-hoc SQL, Redshift Serverless only for Tableau. About 40% of compute removed by moving ingestion alone, data fresh in minutes, Iceberg time travel saved over 35 Redshift hours a week.
Lesson. About 30% of SQL models needed a person to convert; running old and new side by side with row-by-row comparison avoided downtime; S3 Tables removed compaction and snapshot chores.
HelloFreshArchitecture A, Unity Catalog as the Iceberg REST catalog for many engines
Before. A decentralised mix of tools needed one table format and one catalog.
After. Iceberg as the format, Unity Catalog serving as the Iceberg REST catalog for Spark, Flink and Snowflake.
Lesson. A conference talk: design, not audited numbers.
RiskifiedArchitecture A, native Iceberg in Unity Catalog
Before. Wanted tables any engine could read, without tying the data to Databricks.
After. Writes Iceberg tables through Unity Catalog that any Iceberg client can read, which the team describes as future-proofing the architecture.
Lesson. A press-release quote: direction, not detail.
Grab and InsiderStorage step only: Hive tables to Iceberg, on either architecture
Before. Parquet under a Hive-style Glue catalog: too many small files, unsafe concurrent writes, slow file listing.
After. Iceberg tables; Insider reports about 90% lower S3 costs.
Lesson. An in-place migration carries small files over, so rewrite or compact as you migrate; expect lock contention, timestamp mismatches and a temporary S3 cost spike (Grab).
The bad patterns these teams left behind
| Bad pattern | Why it hurts | Good pattern |
|---|---|---|
| Copying lake data into a warehouse for a second engine | Two copies, sync pipelines, drift (United before) | Let the second engine read the same Iceberg files through federation |
| The same table registered in Glue and Unity Catalog, both writing | Two pointers to one table: writes conflict, one clean-up deletes files the other needs | One authoritative catalog; the other federates it read-only |
| Permissions defined twice for the same readers | A row filter in Unity Catalog does not exist in Lake Formation, and the reverse | Fine-grained rules where the table is owned; other engines get pre-filtered gold |
| An always-on warehouse running batch ETL | Paying for idle capacity while workloads fight (Razor before) | Separate compute per workload, serverless where possible |
| Iceberg maintenance with no owner | Snapshots and orphan files pile up, queries slow, storage grows | One owner of chores: Predictive Optimization or S3 Tables |
Step 9 · the market
Who chose A, who chose B, and what the benchmarks really say
We looked for every public team running Iceberg on AWS that says which catalog owns its tables. The picture is very clean. Teams with Databricks at the centre chose A. Teams that chose B do not use Databricks at all. We found no public team running B with Databricks as a real writer, and the public moves between the two that keep Databricks (Decathlon, Outreach) all went from B to A.
| Company | Side | What they run | Where they win | Where they lose, or what to watch | Evidence |
|---|---|---|---|---|---|
| United Airlines | A | Databricks writes everything into Unity Catalog (Delta with Iceberg reads); Redshift Serverless reads gold through Glue federation and Lake Formation. | About $30K a month of copy pipelines removed; one catalog; Redshift users keep their tool. | Redshift is read-only; only Iceberg or UniForm tables are visible; the Lake Formation grant must be on the target table, not the resource link. | Detailed AWS blog |
| Mastercard | A | Unity Catalog is the single source of truth, federated into AWS Glue so AWS engines can use the same tables. | Ended the multi-catalog struggle; engineers pick the engine per job without metadata drifting. | No numbers published. | Summit talk abstract (2026) |
| Decathlon | A, moved from B | Left the AWS Glue catalog for Unity Catalog because Glue metadata management strained at thousands of tables; kept EMR, Athena, Polars and DuckDB working against Unity Catalog. | One governance layer at scale while outside engines stay open. | The only public move between the two we found, and it went from B to A; no numbers published. | Summit talk abstract |
| HelloFresh | A | Iceberg as the format, Unity Catalog as the Iceberg REST catalog for Spark, Flink and Snowflake. | One catalog across many tools. | Design talk, no numbers. | Summit talk |
| Riskified | A | Writes Iceberg through Unity Catalog so any Iceberg client can read it. | Avoids tying the data to one engine. | A press quote, no detail. | Press release |
| Indeed | B, no Databricks | Moved a 101 PB lake (18,000 datasets, 190,000 queries a day) from Hive to Iceberg on S3 Tables, linked to its existing Glue catalog; Athena and EMR Spark among four engines. | Expected 10% lower cost, 1,000+ developer hours a year back, 10x more transactions per second, team onboarding from a day to minutes. | Most numbers are labelled expected, not measured; the talk lists Lake Formation integration, query performance and different logging as the hard parts. | AWS case study |
| Razor Group | B, no Databricks | S3 Tables in bronze, silver and gold; Glue and Lake Formation; Spark on EC2 Spot; Athena; Redshift Serverless only for Tableau. | About 40% of compute removed by moving ingestion alone; data minutes old instead of 4 to 6 hours. | About 30% of 1,000+ SQL models needed a person to convert. | Detailed AWS blog |
| Pendulum | B, no Databricks | Glue streaming and Glue ETL write three S3 table buckets; Athena, EMR Serverless, OpenSearch and QuickSight read. | About 40% lower cost (mostly Glue DPUs spent on home-made compaction), daily job 1 hour to 10 minutes, about 70% faster queries. | The before state was hand-written maintenance scripts, so part of the win is simply having an owner of the chores. | AWS blog |
| Tubular Labs | B, no Databricks | Spark on EKS writes 70 TB, 25 billion rows, into S3 Tables with write-audit-publish on Iceberg branches. | Merge job 6 to 8 hours down to under 2; recovery from failures 12 to 16 hours down to about 40 minutes; about half of engineering time back. | The before state was raw Parquet with no transactions, so most of the win is Iceberg itself, not the catalog. | AWS blog |
| Comcast | B, no Databricks | GraphQL on API Gateway and Lambda reads S3 Tables through its Iceberg REST endpoint; Athena, EMR and PyIceberg read the same tables. | 80x more concurrent requests, 38% lower cost, median latency 820 to 590 ms. | Replaced per-team database copies, not another catalog. | AWS blog |
| Amica | B, no Databricks | S3 table buckets with Lake Formation grants tied to Identity Center groups; Athena reads. | Fine-grained access for analysts without copies. | No numbers published. | AWS blog |
Where A wins and loses in practice
- Wins: copy pipelines into Redshift disappear (United, about $30K a month); one catalog stays the truth for many engines (Mastercard, HelloFresh); it survived scale where Glue strained (Decathlon).
- Loses: AWS readers are read-only; only Iceberg or UniForm tables cross the door; Lake Formation grants have gotchas; almost none of these teams publish numbers, so A's case rests on design talks more than measurements.
Where B wins and loses in practice
- Wins: big, measured cost and speed gains (Razor 40% compute, Pendulum 40% cost and 6x jobs, Tubular merges 6 to 8 hours down to under 2, Comcast 38% cost).
- Loses: those gains are mostly against a bad before state (no maintenance, raw Parquet, an always-on warehouse), not against A; Lake Formation sharing and logging are the hard parts (Indeed; one team lost a month to cross-account sharing); none of them run Databricks.
Has anyone moved from one to the other, and did it pay?
Moves are rare and almost never measured. Every B-to-A move we found was a Databricks-centred team leaving the Glue catalog; the only A-to-B-like move left Databricks entirely, for reasons about ML operations rather than the catalog. Nobody published a before-and-after cost or speed number for a pure catalog move in either direction.
| Direction | Company | What moved | Gain claimed | Measured? | Verdict on the evidence |
|---|---|---|---|---|---|
| B to A | Decathlon | Catalog from AWS Glue to Unity Catalog at thousands of tables; EMR, Athena, Polars and DuckDB kept reading. | One governance layer that held up at scale; outside engines stayed open. | No numbers; a summit talk abstract. | Real move, unmeasured |
| B to A | Outreach | Glue tables brought under Unity Catalog with the one-click foreign-to-managed conversion. | "Eliminated months of migration planning" and sped up adoption of Unity Catalog and serverless. | One quote in a Databricks post; no cost or speed figures. | Real move, unmeasured |
| A to B (roughly) | Kargo | ML pipelines left Databricks notebooks and Delta for Glue ETL writing Iceberg, Athena for training slices, SageMaker for training and serving. | Lower compute cost from removing always-on clusters; pay-per-query Athena; easier CI/CD and reproducibility. | Qualitative only, in an AWS blog; the catalog is not named and Unity Catalog was not the owner before. | Weak: left Databricks for an ML-ops reason, not a catalog reason |
| A to B | Nobody else we could find | No public team moved tables from Unity Catalog ownership to S3 Tables, Glue and Lake Formation while keeping Databricks. | - | - | Absence is the finding |
On A, which AWS services do people still use?
Choosing A does not mean leaving AWS. It means AWS stops owning the table pointer. Almost every AWS service keeps a job; Glue and Lake Formation change from owner to door.
| AWS service | Its job on A | Who shows it | Verdict |
|---|---|---|---|
| Amazon S3 | Holds every file of every layer, in your own bucket | Everyone on A | Keep |
| Redshift Serverless | Reads gold in place through Glue federation; no copy into Redshift storage | United Airlines | Keep if analysts already live there |
| Athena | Ad-hoc SQL on gold through Glue federation, paid per TB scanned | Decathlon; AWS federation guide | Keep for occasional readers |
| EMR and Glue Spark | Read, and can write, managed Iceberg through Unity Catalog's Iceberg REST catalog | Decathlon; AWS EMR with Unity Catalog guide | Keep where a job must stay on AWS; writes miss Unity Catalog lineage |
| Glue Data Catalog | Becomes a door, not an owner: a federated catalog that mirrors Unity Catalog | United Airlines, Mastercard | Keep as the door for AWS engines |
| Lake Formation | Hands short-lived credentials to AWS readers of gold | United Airlines | Keep, read door only |
| IAM, KMS, PrivateLink | Roles for the bucket, encryption keys, private network path between Databricks and your VPC | Databricks and AWS networking guides | Keep, always |
| Firehose, DMS, MSK, Kinesis | Land files or streams in a landing bucket that Auto Loader picks up | Pattern in the guides; no named case | Keep for capture |
| QuickSight | Dashboards over Athena or Redshift on federated gold | Pattern in the guides; no named case | Keep if already paid for |
| Step Functions, MWAA | Only for AWS-side jobs; Lakeflow Jobs runs the Databricks side | Pattern; two schedulers is a cost | Drop where Lakeflow can do it |
Is there a benchmark that names a clear winner?
No. Every benchmark we found compares one side against a weaker version of itself, or one engine against another. None runs the same data through A and B. Read each claim for what it actually compares:
| Claim | Who measured | What it really compares | Does it pick A or B? |
|---|---|---|---|
| S3 Tables: up to 3x faster queries, up to 10x more transactions per second | AWS | S3 Tables against Iceberg in an ordinary bucket that nobody compacts | No. It measures maintained against unmaintained tables. |
| S3 Tables: about 2x faster than medium-file Iceberg and 4.6x to 37.6x faster than small-file Iceberg on Glue Spark (TPC-H, 104 GB); ingestion 1.7x to 3.1x faster; Athena was slower on S3 Tables than on plain Iceberg; storage about 17.5% dearer | Loka (AWS partner) | S3 Tables against self-managed Iceberg, same AWS engines | No. Same catalog family, different maintenance. |
| Predictive Optimization: up to 20x faster selective queries, 2x lower storage cost; Anker 2x faster queries, 50% lower storage | Databricks | Unity Catalog managed tables against unmanaged ones | No. Same idea as the S3 Tables claim, on the other side. |
| Redshift Serverless on Iceberg over 2x faster in 2025 (TPC-DS 10 TB) | AWS | Redshift this year against Redshift last year | No. Helps both A and B, since Redshift reads Iceberg either way. |
| EMR Spark 4.5x faster than open-source Spark on Iceberg (TPC-DS 3 TB) | AWS | EMR runtime against open-source Spark | No. An engine claim. |
| StarRocks 1.3x faster than Databricks Photon on catalog queries (TPC-DS 1 TB) | CelerData (vendor) | Engines; Photon ran on Delta, not managed Iceberg | No. Engines, and a vendor test. |
| A head-to-head: same data, A against B, both engines, ETL plus queries plus chores plus cost | Nobody, as of November 2026 | - | This is the benchmark that would tell the story, and it does not exist. |
The benchmark that would settle it
If a client wants numbers, run this small test on their own account before committing. It is a few days of work and answers the question for their workload, which no public benchmark does.
| Part | What to do |
|---|---|
| Data | TPC-DS at 1 TB as gold, plus a daily MERGE of 1% changed rows into silver |
| Run A | Databricks serverless writes managed Iceberg in Unity Catalog with Predictive Optimization; Athena and Redshift read gold through Glue federation |
| Run B | Glue or EMR Spark writes S3 Tables with Glue and Lake Formation; Databricks reads gold as foreign tables |
| Measure | ETL wall time and cost, query time per engine on both sides, maintenance cost (DBUs or S3 Tables fees), freshness from write to readable, and hours of setup |
| Expected result | Each side wins where its own engines write and maintain the tables; the guest side pays in features and governance, not speed. That is the claim to test. |
Step 10 · synthesis
Who wins
Architecture B wins only when AWS is the real home: Glue, EMR, Athena, Redshift, Firehose and Flink do the work, and Databricks is a light guest or absent. Then S3 Tables, Glue and Lake Formation are native, cheaper and simpler, as Razor Group shows.
| If this is true | Pick | Why |
|---|---|---|
| Databricks does most of the work (pipelines, ML, Genie, most SQL) | A | Everything Databricks offers works on owned tables; AWS engines still read gold in place. |
| AWS engines do most of the work; Databricks is used by one team, or not at all | B | S3 Tables, Glue and Lake Formation are native for Glue, EMR, Athena, Redshift, Firehose and Flink. Databricks reads through federation. |
| Streams arrive through Firehose or Managed Flink and must be queryable in Athena within seconds | B for that lane | Keep that domain AWS-owned and let Databricks read it; this is the one place a split by domain makes sense. |
| Separate teams already committed to each side, by domain | Split by domain | Each table still has exactly one owner; catalogs federate each other read-only. Highest effort to run. |
| You may move to Azure, Fabric or Google Cloud later | A | Unity Catalog is the same on every cloud; only the AWS reader door changes. |
Whichever you choose, write it down as one rule: one table, one catalog, one writer path, one owner of the chores. List the other architecture under "options considered" with its reason; that is what a reviewer will ask for.
FAQ
Questions people ask
Can Databricks write bronze to silver to gold when AWS owns the catalog?
Only from classic clusters, using the open-source Iceberg library pointed at the Glue Iceberg REST endpoint with Lake Formation credentials. It works, but those writes skip Unity Catalog permissions, lineage and audit, cannot use serverless or Lakeflow pipelines, and you maintain library versions. Through Unity Catalog itself, AWS-owned tables are read-only foreign tables.
Can AWS engines write when Unity Catalog owns the catalog?
Spark on EMR or Glue, Trino and Snowflake can write managed Iceberg tables through Unity Catalog's Iceberg REST catalog with a Databricks service principal, governed by Unity Catalog grants. Athena, Redshift and QuickSight read gold through Glue federation and do not write. Firehose and DMS land files that Auto Loader loads.
Are Glue with Lake Formation and the SageMaker lakehouse the same thing?
Mostly. Glue Data Catalog is the catalog, Lake Formation is the permissions on it, and the SageMaker lakehouse is AWS's umbrella name for the two, reaching across S3, S3 Tables, Redshift storage and federated catalogs. SageMaker Catalog (formerly DataZone) is a different, business-facing layer.
Where does Redshift fit?
As one more reader of the same Iceberg tables, through federation, if the client already uses it. Loading data into Redshift's own storage creates a second copy with its own permissions, which is the pattern the case studies moved away from.
Why not let both catalogs own tables?
Two writers on one table means two pointers, conflicting commits and clean-ups that delete each other's files. Split by domain only if teams force it, and even then every table has exactly one owner.
Is Iceberg the right format on Databricks?
It is a sound choice for openness. Inside Databricks, Delta still has features managed Iceberg lacks today (row filters, column masks, keys, Lakeflow streaming tables). Delta with Iceberg reads (UniForm) gives AWS engines Iceberg without giving those up, so many teams mix the two.
Sources
Checked against
Checked in November 2026. Federation writes, managed Iceberg status and fine-grained access for outside engines change every few months; check before you commit.
- Apache Iceberg in Databricks
- Databricks external access and the Iceberg REST catalog
- Amazon S3 Tables integration with AWS analytics
- Amazon SageMaker lakehouse architecture
- AWS: access S3 Iceberg tables from Databricks through the Glue Iceberg REST catalog
- AWS: access Databricks Unity Catalog data through Glue catalog federation
- AWS: catalog federation for Iceberg tables in the Glue Data Catalog
- AWS: United Airlines, Redshift and Glue federation to Databricks
- AWS: Razor Group's journey to a lakehouse on AWS
- Grab: our journey to Apache Iceberg
- Insider: migrating a production data lake to Iceberg
- LakeOps: cross-catalog sync for Polaris, Glue and Unity (third party)
- Onehouse: the catch in Databricks Iceberg support (critic)
- Mastercard: breaking data silos with multi-engine catalog federation (talk)
- Decathlon: race to unified governance, Glue to Unity Catalog (talk)
- AWS: Indeed moves to S3 Tables
- AWS: Pendulum, 6x faster processing with S3 Tables
- AWS: Tubular Labs rebuilds a 70 TB pipeline on S3 Tables
- AWS: Comcast scales analytics 80x with S3 Tables
- AWS: Amica on S3 Tables
- AWS: S3 Tables compaction, up to 3x faster queries
- Loka: test driving S3 Tables (TPC-H)
- Databricks: Predictive Optimization, faster queries and lower cost
- AWS: 2x faster Iceberg queries on Redshift
- AWS: EMR Spark and Iceberg 4.5x faster than open-source Spark
- CelerData: TPC-DS benchmark (vendor)
- Databricks: from the AWS Glue catalog to Unity Catalog, end to end
- Databricks: migrating foreign HMS and Glue tables to Unity Catalog (Outreach quote)
- AWS: Kargo moves ML workloads from Databricks to AWS
Related
Where it connects
The data platform, whole · Who maintains your tables · Delta vs Iceberg on Databricks · AWS + Databricks, ETL end to end · AWS data services catalog · Streaming on AWS and Databricks