Home › Data engineering › Iceberg lakehouse on AWS with Databricks

Data engineering · free guide · one page

Iceberg lakehouse on AWS with Databricks: who should own the catalog?

You want an open lakehouse on AWS, with Apache Iceberg as the table format and Databricks in the picture. AWS offers S3 Tables, Glue, Lake Formation and the SageMaker lakehouse; Databricks offers Unity Catalog. The names spin heads, but there is one decision underneath: which catalog owns the tables. This page draws the whole platform first, then the two physical designs, then compares them concern by concern and picks a winner.

Short answer: if Databricks does most of the work, let Unity Catalog own the tables (Architecture A): managed Iceberg in your own S3 bucket, Predictive Optimization doing the chores, and Athena, Redshift and QuickSight reading gold in place through Glue federation. If AWS engines do most of the work and Databricks is a light guest, let AWS own them (Architecture B): S3 Tables with Glue and Lake Formation. Never let both own the same table.

Step 1 · the conceptual big picture

The whole data platform, with every concern

Before any product name, here is the platform as jobs. Data comes from sources on the left, moves along one line (ingest, bronze, silver, gold, semantic) and reaches consumers on the right. Under the line sit five layers every table needs. Along the bottom are the 24 concerns that cut across every hop; each is explained on the data platform page.

SourcesdatabasesSaaS and APIsfilesevents and IoTmachines (OT)the data platformingestbronzesilvergoldsemanticstreaming: a second, faster speed on the same linecompute engines: batch, SQL, streaming, MLcatalog + governance: ONE owner per table (pointer, permissions, lineage)table format: Apache Icebergobject storage: S3, open Parquet filesmaintenance: compaction, snapshot expiry, orphan filesthe decision on this page is the thick box: who owns the catalogConsumersBI dashboardsAI agents, Genieapps and APIsML and featuresdata sharingother enginesacross every hop: the 24 concernsTrustcontracts & schemadata qualityprofilingreconciliationSeelineagelogs, metrics, tracesfreshness & SLOscost & FinOpsperformance & scaleGuardsecurity & accessprivacy & PIIgovernance & semanticsaudit & run logRunmetadata & controlorchestrationreplay & backfillenvironments & CI/CDtestingresilience & DRlayoutstreamingSelfself-healingagent assistancesimulation

Look at the thick box. Storage, the table format and compute are easy to agree on. Who owns the catalog decides who can write, who sets permissions, who sees lineage and who cleans the tables.

Why the catalog matters so much: an Iceberg write only becomes real when the catalog swaps its pointer to the new metadata file. Whoever holds that pointer controls writes, permissions and clean-up. Every other engine is a guest. See Who maintains your tables for the picture.

Step 2 · the AWS names, decoded

Six names, three jobs

Several of these are layers of the same thing, which is why they confuse. Storage is S3 Tables. Catalog and permissions are Glue plus Lake Formation, sold together as the SageMaker lakehouse. Unity Catalog does both catalog and permissions on the Databricks side.

NameWhat it really is
Amazon S3 TablesStorage: a table bucket that holds Iceberg tables and maintains them itself (compaction, snapshot expiry, unreferenced file removal).
AWS Glue Data CatalogThe catalog: which tables exist and, for Iceberg, the pointer to each table's current metadata file. Every AWS engine looks tables up here.
AWS Lake FormationPermissions on the Glue catalog: table, column, row and cell grants, LF-Tags, and short-lived credentials handed to engines.
SageMaker lakehouseAWS's umbrella name for Glue Data Catalog plus Lake Formation, reaching across S3, S3 Tables, Redshift storage and federated catalogs, with an Iceberg REST endpoint.
SageMaker CatalogA different thing: the business catalog (glossary, data products, access requests) on top. Formerly DataZone.
Databricks Unity CatalogDatabricks' catalog, permissions, lineage, audit and sharing in one, with its own Iceberg REST catalog for outside engines.

Step 3 · physical design B

Architecture B: AWS owns the catalog, Databricks is a guest

Iceberg tables live in an S3 table bucket. The Glue Data Catalog holds each table's pointer and Lake Formation holds the permissions. S3 Tables compacts, expires snapshots and removes unreferenced files by itself. AWS engines write and read natively.

AWS Glue Data Catalog + Lake Formation(the SageMaker lakehouse) owns the pointerpermissions: Lake Formation, LF-TagsS3 table bucketIceberg: bronze, silver, goldS3 Tables doesthe choresowners writeFirehose, Managed FlinkGlue ETL, EMR SparkAWS readersAthenaRedshift ServerlessQuickSightDatabricks, a guestreads: Unity Catalog federation,foreign tables, read-onlywrites: classic clusters only, Iceberglibrary + Glue REST endpoint,outside Unity Catalog governance

Owners on the left write through Glue. AWS readers on the right read through Glue. Databricks, dashed, is a guest.

Can Databricks do bronze to silver to gold here?

This is the question most teams get wrong. Through Unity Catalog, AWS-owned Iceberg tables appear as foreign tables, which are read-only. The only way for Databricks to write them today is around Unity Catalog. Four ways, with verdicts:

WayHow it worksWhat you give upVerdict
B1. AWS engines do all the ETL; Databricks only readsGlue ETL or EMR Spark writes bronze, silver and gold into S3 Tables. Databricks reads the tables through Unity Catalog federation for BI, ML or Genie.Clean, but Databricks is a reader you pay forPursue when Databricks is a light guest
B2. Databricks writes through the Glue Iceberg REST endpointA classic Databricks cluster (dedicated access mode) loads the open-source Iceberg library and points a Spark catalog at the Glue Iceberg REST endpoint, signed with the cluster's IAM role; Lake Formation hands out credentials. Bronze to silver to gold runs as ordinary Spark jobs.Works. But no serverless, no Lakeflow pipelines, no Unity Catalog permissions, lineage or audit on these writes, and you keep Iceberg library versions in step yourselfPursue only for a short migration or a few jobs
B3. Databricks writes to foreign tables through Unity CatalogUnity Catalog shows Glue tables as foreign Iceberg tables. Today these are read-only in Databricks, with no partition evolution.Not available as of November 2026; check the Databricks docs, this changesDrop for now
B4. Databricks owns bronze and silver; AWS owns goldSplit ownership by layer: Databricks writes Unity Catalog tables, then an AWS job copies or rebuilds gold into S3 Tables.Two catalogs, a copy, two permission systems, lineage breaks at the seamDrop

Databricks' support for foreign Iceberg tables changes every few months. Check the current docs before you rule writes in or out.

Step 4 · physical design A

Architecture A: Unity Catalog owns the catalog, AWS engines are guests

Iceberg tables live in your own S3 bucket as Unity Catalog managed Iceberg tables. Unity Catalog holds the pointer, the grants, the lineage and the audit. Predictive Optimization does the chores. Databricks writes every layer on serverless compute.

Databricks Unity Catalogowns the pointergrants, tags, lineage, audit, metric viewsyour S3 bucketmanaged Iceberg: bronze, silver, goldPredictiveOptimizationowners writeLakeflow jobs, SQLAuto Loader, ConnectAWS landing (optional)Firehose, DMS, apps drop filesin a landing bucket; Auto Loaderpicks them up into bronzeguestsAthena, Redshift, QuickSightread gold through Gluefederation + Lake FormationEMR, Trino, Snowflakeread, and can write, throughUnity Catalog's Iceberg REST

Owners on the left write through Unity Catalog. Guests on the right read in place, and Iceberg engines can even write, through Unity Catalog's Iceberg REST catalog.

And the other way round: can AWS engines work here?

WhoHow it worksLimitVerdict
Athena, Redshift, QuickSight read goldGlue catalog federation to Unity Catalog fetches the table's current metadata from Unity Catalog's Iceberg REST endpoint; Lake Formation hands out short-lived credentials; files are read in place, no copy.Read-only for these engines; grant Lake Formation SELECT on the target tablesPursue
EMR, Glue Spark, Trino, Snowflake write managed IcebergThey connect to Unity Catalog's Iceberg REST catalog with a Databricks service principal and write managed Iceberg tables directly, governed by Unity Catalog grants.Their writes are not in Unity Catalog lineage; Predictive Optimization still maintains the tablePursue only where an AWS engine really must write
Firehose or DMS write into the lakehouseThey land files in a landing bucket; Auto Loader or Lakeflow Connect loads them into bronze.One extra hop of seconds to minutesPursue

Step 5 · the asymmetry

Who can write where

This grid is the heart of the decision. Unity Catalog lets outside Iceberg engines write managed tables through its REST catalog, under its own grants. Glue lets Databricks write only around Unity Catalog. So A keeps the door open both ways; B closes Databricks' governed door.

A: Unity Catalog ownsB: AWS ownsDatabrickswrites everything: serverless,Lakeflow, SQL; fully governedreads via federation (read-only);writes only from classic clustersAWS enginesAthena, Redshift read gold;EMR, Spark can write via UC RESTGlue, EMR, Firehose, Flinkwrite everything natively

Green: full, governed. Butter: works with a limit. Rose: read-only, or writes that skip governance.

Step 6 · concern by concern

A and B compared on every concern

Every one of the 24 platform concerns is here (their ids in grey), plus the serving, ingestion and strategy concerns. Green works fully, yellow works with a limit, rose does not. Checked against AWS and Databricks docs in November 2026; Iceberg support on both sides changes often.

ConcernA: Unity Catalog ownsB: AWS ownsBetter
Tables and maintenance
Storage and table formatManaged Iceberg tables in your own S3 bucket, at a path Unity Catalog picks under the managed locationIceberg in an S3 table bucket, reached through table APIs; or Glue Iceberg in your bucketTie
Maintenance (compaction, snapshot expiry, orphan files, statistics, clustering) X19Predictive Optimization, automatic and required on managed Iceberg; runs on serverless, billed as DBUsS3 Tables, automatic and on by default; billed per object and byte processed. Predictive Optimization cannot run on these tablesTie: both automatic, one owner each
Layout: catalogs, schemas, names X23catalog.schema.table names encode environment, layer and domain; one naming schemeTable bucket, namespace, table; seen from Databricks through a federated catalog with a second set of namesA
ETL
Ingestion (sources to bronze)Auto Loader, Lakeflow Connect, Structured Streaming; AWS services land files that Auto Loader picks upFirehose and DMS write Iceberg directly; Glue jobs; Databricks only from classic clustersTie: each is native on its own side
Bronze to silver, silver to goldDatabricks jobs, SQL and Lakeflow on serverless; EMR or Glue Spark can also write through Unity Catalog's Iceberg RESTGlue ETL or EMR Spark natively; Databricks only on classic clusters through the Glue REST endpoint, ungoverned by Unity CatalogA
Orchestration X2Lakeflow Jobs with table-update triggers on owned tablesStep Functions or Airflow (MWAA) for AWS jobs; Databricks jobs called from them; two schedulers to watchA
Metadata & control X1One catalog to read every table's schema, owner and tags fromGlue holds the tables; Databricks-side settings live in a second placeA
Replay & backfill X15Re-runs and MERGE on Databricks; time travel on snapshotsRe-runs on Glue or EMR; Databricks re-runs only from classic clustersA
Streaming
Streaming lane X24Structured Streaming and Lakeflow are strong, but streaming tables and materialized views are Delta, so the fast lane is Delta even when gold is IcebergFirehose and Managed Flink write Iceberg into S3 Tables natively; Databricks streaming into them is not practicalB if streams are AWS-native; A if Databricks streams
Semantic layer
Governance & semantics: owners, glossary, metric definitions X13Unity Catalog metric views, tags, owners and certified tables, used by Genie and AI/BI dashboardsNo lake-wide metric layer on AWS: QuickSight datasets, Redshift views, SageMaker Catalog glossary; Databricks metric views would sit over read-only foreign tablesA
Serving
BI and SQLDatabricks SQL warehouses and AI/BI; Athena, Redshift and QuickSight read gold through federationAthena, Redshift Serverless and QuickSight natively; Databricks SQL reads through federationTie
Apps and APIsLakebase (managed Postgres) synced from Unity Catalog tables, Databricks AppsDynamoDB or Aurora fed by Glue or EMR jobs, API Gateway and LambdaTie
AI agents and MLGenie, Vector Search, Model Serving, MLflow, features and models governed in Unity Catalog next to the dataBedrock and SageMaker AI read through Athena or Redshift; Databricks ML reads foreign tables but keeps models and features in Unity Catalog, so governance splitsA
Data sharingDelta Sharing, open and cross-cloud; Iceberg REST for enginesLake Formation across AWS accounts (RAM), Redshift data sharingA outside AWS; tie inside AWS
Cost
Cost & FinOps X10One main bill (DBUs plus S3); serverless everywhere; Athena per TB for AWS readers; one FinOps view from system tablesS3 Tables storage costs a little more per GB than S3 Standard, plus monitoring and compaction fees; Glue and EMR compute; Databricks on classic clusters only; two bills, two FinOps viewsA if Databricks does most work; B if Databricks is light or absent
Performance & scale X19Photon, liquid clustering and Predictive Optimization on owned tablesAWS engines tuned on their own tables; Databricks reads foreign tables without control of their layoutA
Governance and security
Security & access X11Unity Catalog grants and tag-based policies; Lake Formation only at the read door for AWS readers of goldLake Formation for AWS engines; Unity Catalog grants on the foreign catalog for Databricks readers; Databricks writers (B2) bypass Unity CatalogA
Privacy & PII X12Row filters and column masks are not supported on managed Iceberg today (they are on Delta); use dynamic views, or keep PII tables in Delta with Iceberg readsLake Formation row and cell filters for AWS engines; outside engines with vended credentials generally cannot read filtered tablesTie: both partial with Iceberg
Audit & run log X14Unity Catalog audit and access system tables for all Databricks work; CloudTrail for AWS readsCloudTrail and Lake Formation for AWS; Databricks audit for its share; two trails to joinA
Lineage
Lineage X7Automatic table and column lineage for every Databricks job; writes by outside engines through Iceberg REST are not recordedNo automatic lineage in Glue or Lake Formation; SageMaker Catalog lineage needs OpenLineage set up; B2 Databricks writes are invisible to bothA
Trust
Contracts & schema X3Contract checks in code; schema enforced on write (informational keys not on managed Iceberg)Contract checks in code; Glue schemaTie
Data quality X4Lakeflow expectations, DQX, data quality monitoring on Unity Catalog tablesGlue Data Quality (DQDL), DeequTie
Profiling X5Data profiling on Unity Catalog tables; check Iceberg supportGlue Data Quality rule recommendationsTie
Reconciliation X6Engine-neutral checks; one catalog to compare againstEngine-neutral checksTie
See
Logs, metrics, traces X8System tables for jobs, queries and billing in one placeCloudWatch for AWS jobs plus Databricks system tables; two placesA
Freshness & SLOs X9Table-update triggers and freshness monitors on owned tablesEventBridge events from Glue and S3 Tables; Databricks cannot trigger on foreign table updatesA
Run
Environments & CI/CD X16Databricks bundles plus Terraform for the bucket and IAMCloudFormation, CDK or Terraform for S3 Tables, Glue and Lake Formation, plus bundles for Databricks; two toolchainsA
Testing X17Table-driven tests, zero-copy clones for test dataTable-driven tests; Iceberg branches and tags for test dataTie
Resilience & DR X18Time travel, UNDROP for 7 days; plan how catalog metadata is rebuilt in another regionSnapshots and AWS-managed durability; plan cross-region copies of the table bucketTie
Self
Self-healing X20One catalog and one run log to detect and fix fromTwo systems to watch and fixA
Agent assistance X21Agents read one catalog, lineage and metric viewsAgents must read two catalogsA
Simulation X22Engine-neutralEngine-neutralTie
Strategy
Lock-in and portabilityUnity Catalog runs on AWS, Azure and Google Cloud; but it picks managed table paths, so leaving means re-registering or rewritingAWS only; S3 Tables data is reached only through table APIsA
Team and supportOne vendor for the data plane; Spark and SQL skillsAWS-native skills for the core, Databricks skills for the guest; two vendors to callDepends on the team
Tally: A is better on 17 concerns, B on 1, they tie on 13, and 3 depend on your situation (streaming, cost, sharing, team). B's wins are real but narrow: AWS-native streaming, and cost when Databricks is light or absent.

Step 7 · what each one restricts

What you accept with each

If you go with A (Unity Catalog owns)

  • Athena, Redshift and QuickSight read gold through Glue federation and cannot write. EMR, Glue Spark or Trino can write managed Iceberg through Unity Catalog's Iceberg REST, but those writes are missing from Unity Catalog lineage.
  • Lake Formation still appears, but only at the read door for AWS engines, on curated gold tables. Expose tables without row filters or masks there, or pre-filtered gold tables.
  • Managed Iceberg today gives up row filters, column masks and informational keys; change data feed works only on Iceberg v3 and only for Databricks readers; Lakeflow streaming tables and materialized views are Delta. Keep those tables in Delta with Iceberg reads (UniForm), or use dynamic views.
  • Check that managed Iceberg is generally available for your region and runtime before production; parts were still marked preview in the docs we checked.
  • Firehose and DMS cannot write straight into Unity Catalog tables: they land files, and Auto Loader loads them.
  • Databricks is the main bill. Predictive Optimization runs on serverless and shows up as DBUs.
  • Unity Catalog picks the paths of managed tables. Leaving later means re-registering or rewriting tables.

If you go with B (AWS owns)

  • Databricks sees AWS tables as foreign Iceberg tables: read-only, no partition evolution, and views defined on one side are not visible on the other.
  • Databricks can do bronze to silver to gold only from classic clusters with the open-source Iceberg library pointed at the Glue REST endpoint: no serverless, no Lakeflow pipelines, no Unity Catalog permissions, lineage or audit on those writes.
  • No Predictive Optimization, liquid clustering control, table-update triggers or Unity Catalog masks on these tables; Genie and metric views can only sit over read-only foreign tables.
  • Permissions live in two systems: Lake Formation for AWS engines and Unity Catalog for Databricks readers. Tables with Lake Formation row or cell filters are generally not readable by outside engines.
  • Lineage is fragmented: nothing automatic on the AWS side, and Databricks writes through the Glue endpoint are invisible to Unity Catalog.
  • Two bills, two toolchains (CloudFormation or CDK and Databricks bundles), two monitoring places.
  • AWS only. Moving to Azure, Fabric or Google Cloud later means a new catalog and new permissions.

Step 8 · evidence

Case studies: from a bad design to a good one

We found clear public write-ups of Architecture A (United Airlines, with Delta tables read as Iceberg) and of Architecture B without Databricks (Razor Group). We found no public case of B with Databricks as the main writer, which is itself a finding. Numbers come from the companies' or vendors' own blogs and are not audited.

United AirlinesArchitecture A, with Delta + Iceberg reads

Before. Before: petabytes in bronze, silver and gold, catalogued in Unity Catalog. Redshift could not see them, so tables were copied into Redshift storage with sync pipelines and a manually synced catalog, which AWS's write-up calls operationally expensive and prone to drift.

After. After: Databricks does all processing and owns the tables (Delta with UniForm). A Glue federated catalog fetches metadata from Unity Catalog's Iceberg REST endpoint, Lake Formation vends scoped credentials, and Redshift Serverless reads the files in place. About $30K a month of duplicate ETL removed; 30 tables in production, 70 rolling out, hundreds planned.

Lesson. Grant Lake Formation SELECT on the target tables, not on the resource link; roll out the busiest tables first; name external schemas so readers know the data is federated.

Read the write-up

Razor GroupArchitecture B, without Databricks

Before. Before: more than 1,000 SQL models on always-on Redshift clusters, 98% of compute spent on batch ETL, data 4 to 6 hours old, workloads fighting for one cluster.

After. After: S3 Tables (Iceberg) in bronze, silver and gold, Glue and Lake Formation as catalog and permissions, Spark on EC2 Spot for ETL, Athena for ad-hoc SQL, Redshift Serverless only for Tableau. About 40% of compute removed by moving ingestion alone, data fresh in minutes, Iceberg time travel saved over 35 Redshift hours a week.

Lesson. About 30% of SQL models needed a person to convert; running old and new side by side with row-by-row comparison avoided downtime; S3 Tables removed compaction and snapshot chores.

Read the write-up

HelloFreshArchitecture A, Unity Catalog as the Iceberg REST catalog for many engines

Before. A decentralised mix of tools needed one table format and one catalog.

After. Iceberg as the format, Unity Catalog serving as the Iceberg REST catalog for Spark, Flink and Snowflake.

Lesson. A conference talk: design, not audited numbers.

Read the write-up

RiskifiedArchitecture A, native Iceberg in Unity Catalog

Before. Wanted tables any engine could read, without tying the data to Databricks.

After. Writes Iceberg tables through Unity Catalog that any Iceberg client can read, which the team describes as future-proofing the architecture.

Lesson. A press-release quote: direction, not detail.

Read the write-up

Grab and InsiderStorage step only: Hive tables to Iceberg, on either architecture

Before. Parquet under a Hive-style Glue catalog: too many small files, unsafe concurrent writes, slow file listing.

After. Iceberg tables; Insider reports about 90% lower S3 costs.

Lesson. An in-place migration carries small files over, so rewrite or compact as you migrate; expect lock contention, timestamp mismatches and a temporary S3 cost spike (Grab).

Read the write-up

The bad patterns these teams left behind

Bad patternWhy it hurtsGood pattern
Copying lake data into a warehouse for a second engineTwo copies, sync pipelines, drift (United before)Let the second engine read the same Iceberg files through federation
The same table registered in Glue and Unity Catalog, both writingTwo pointers to one table: writes conflict, one clean-up deletes files the other needsOne authoritative catalog; the other federates it read-only
Permissions defined twice for the same readersA row filter in Unity Catalog does not exist in Lake Formation, and the reverseFine-grained rules where the table is owned; other engines get pre-filtered gold
An always-on warehouse running batch ETLPaying for idle capacity while workloads fight (Razor before)Separate compute per workload, serverless where possible
Iceberg maintenance with no ownerSnapshots and orphan files pile up, queries slow, storage growsOne owner of chores: Predictive Optimization or S3 Tables

Step 9 · the market

Who chose A, who chose B, and what the benchmarks really say

We looked for every public team running Iceberg on AWS that says which catalog owns its tables. The picture is very clean. Teams with Databricks at the centre chose A. Teams that chose B do not use Databricks at all. We found no public team running B with Databricks as a real writer, and the public moves between the two that keep Databricks (Decathlon, Outreach) all went from B to A.

CompanySideWhat they runWhere they winWhere they lose, or what to watchEvidence
United AirlinesADatabricks writes everything into Unity Catalog (Delta with Iceberg reads); Redshift Serverless reads gold through Glue federation and Lake Formation.About $30K a month of copy pipelines removed; one catalog; Redshift users keep their tool.Redshift is read-only; only Iceberg or UniForm tables are visible; the Lake Formation grant must be on the target table, not the resource link.Detailed AWS blog
MastercardAUnity Catalog is the single source of truth, federated into AWS Glue so AWS engines can use the same tables.Ended the multi-catalog struggle; engineers pick the engine per job without metadata drifting.No numbers published.Summit talk abstract (2026)
DecathlonA, moved from BLeft the AWS Glue catalog for Unity Catalog because Glue metadata management strained at thousands of tables; kept EMR, Athena, Polars and DuckDB working against Unity Catalog.One governance layer at scale while outside engines stay open.The only public move between the two we found, and it went from B to A; no numbers published.Summit talk abstract
HelloFreshAIceberg as the format, Unity Catalog as the Iceberg REST catalog for Spark, Flink and Snowflake.One catalog across many tools.Design talk, no numbers.Summit talk
RiskifiedAWrites Iceberg through Unity Catalog so any Iceberg client can read it.Avoids tying the data to one engine.A press quote, no detail.Press release
IndeedB, no DatabricksMoved a 101 PB lake (18,000 datasets, 190,000 queries a day) from Hive to Iceberg on S3 Tables, linked to its existing Glue catalog; Athena and EMR Spark among four engines.Expected 10% lower cost, 1,000+ developer hours a year back, 10x more transactions per second, team onboarding from a day to minutes.Most numbers are labelled expected, not measured; the talk lists Lake Formation integration, query performance and different logging as the hard parts.AWS case study
Razor GroupB, no DatabricksS3 Tables in bronze, silver and gold; Glue and Lake Formation; Spark on EC2 Spot; Athena; Redshift Serverless only for Tableau.About 40% of compute removed by moving ingestion alone; data minutes old instead of 4 to 6 hours.About 30% of 1,000+ SQL models needed a person to convert.Detailed AWS blog
PendulumB, no DatabricksGlue streaming and Glue ETL write three S3 table buckets; Athena, EMR Serverless, OpenSearch and QuickSight read.About 40% lower cost (mostly Glue DPUs spent on home-made compaction), daily job 1 hour to 10 minutes, about 70% faster queries.The before state was hand-written maintenance scripts, so part of the win is simply having an owner of the chores.AWS blog
Tubular LabsB, no DatabricksSpark on EKS writes 70 TB, 25 billion rows, into S3 Tables with write-audit-publish on Iceberg branches.Merge job 6 to 8 hours down to under 2; recovery from failures 12 to 16 hours down to about 40 minutes; about half of engineering time back.The before state was raw Parquet with no transactions, so most of the win is Iceberg itself, not the catalog.AWS blog
ComcastB, no DatabricksGraphQL on API Gateway and Lambda reads S3 Tables through its Iceberg REST endpoint; Athena, EMR and PyIceberg read the same tables.80x more concurrent requests, 38% lower cost, median latency 820 to 590 ms.Replaced per-team database copies, not another catalog.AWS blog
AmicaB, no DatabricksS3 table buckets with Lake Formation grants tied to Identity Center groups; Athena reads.Fine-grained access for analysts without copies.No numbers published.AWS blog

Where A wins and loses in practice

  • Wins: copy pipelines into Redshift disappear (United, about $30K a month); one catalog stays the truth for many engines (Mastercard, HelloFresh); it survived scale where Glue strained (Decathlon).
  • Loses: AWS readers are read-only; only Iceberg or UniForm tables cross the door; Lake Formation grants have gotchas; almost none of these teams publish numbers, so A's case rests on design talks more than measurements.

Where B wins and loses in practice

  • Wins: big, measured cost and speed gains (Razor 40% compute, Pendulum 40% cost and 6x jobs, Tubular merges 6 to 8 hours down to under 2, Comcast 38% cost).
  • Loses: those gains are mostly against a bad before state (no maintenance, raw Parquet, an always-on warehouse), not against A; Lake Formation sharing and logging are the hard parts (Indeed; one team lost a month to cross-account sharing); none of them run Databricks.

Has anyone moved from one to the other, and did it pay?

Moves are rare and almost never measured. Every B-to-A move we found was a Databricks-centred team leaving the Glue catalog; the only A-to-B-like move left Databricks entirely, for reasons about ML operations rather than the catalog. Nobody published a before-and-after cost or speed number for a pure catalog move in either direction.

DirectionCompanyWhat movedGain claimedMeasured?Verdict on the evidence
B to ADecathlonCatalog from AWS Glue to Unity Catalog at thousands of tables; EMR, Athena, Polars and DuckDB kept reading.One governance layer that held up at scale; outside engines stayed open.No numbers; a summit talk abstract.Real move, unmeasured
B to AOutreachGlue tables brought under Unity Catalog with the one-click foreign-to-managed conversion."Eliminated months of migration planning" and sped up adoption of Unity Catalog and serverless.One quote in a Databricks post; no cost or speed figures.Real move, unmeasured
A to B (roughly)KargoML pipelines left Databricks notebooks and Delta for Glue ETL writing Iceberg, Athena for training slices, SageMaker for training and serving.Lower compute cost from removing always-on clusters; pay-per-query Athena; easier CI/CD and reproducibility.Qualitative only, in an AWS blog; the catalog is not named and Unity Catalog was not the owner before.Weak: left Databricks for an ML-ops reason, not a catalog reason
A to BNobody else we could findNo public team moved tables from Unity Catalog ownership to S3 Tables, Glue and Lake Formation while keeping Databricks.--Absence is the finding

On A, which AWS services do people still use?

Choosing A does not mean leaving AWS. It means AWS stops owning the table pointer. Almost every AWS service keeps a job; Glue and Lake Formation change from owner to door.

AWS serviceIts job on AWho shows itVerdict
Amazon S3Holds every file of every layer, in your own bucketEveryone on AKeep
Redshift ServerlessReads gold in place through Glue federation; no copy into Redshift storageUnited AirlinesKeep if analysts already live there
AthenaAd-hoc SQL on gold through Glue federation, paid per TB scannedDecathlon; AWS federation guideKeep for occasional readers
EMR and Glue SparkRead, and can write, managed Iceberg through Unity Catalog's Iceberg REST catalogDecathlon; AWS EMR with Unity Catalog guideKeep where a job must stay on AWS; writes miss Unity Catalog lineage
Glue Data CatalogBecomes a door, not an owner: a federated catalog that mirrors Unity CatalogUnited Airlines, MastercardKeep as the door for AWS engines
Lake FormationHands short-lived credentials to AWS readers of goldUnited AirlinesKeep, read door only
IAM, KMS, PrivateLinkRoles for the bucket, encryption keys, private network path between Databricks and your VPCDatabricks and AWS networking guidesKeep, always
Firehose, DMS, MSK, KinesisLand files or streams in a landing bucket that Auto Loader picks upPattern in the guides; no named caseKeep for capture
QuickSightDashboards over Athena or Redshift on federated goldPattern in the guides; no named caseKeep if already paid for
Step Functions, MWAAOnly for AWS-side jobs; Lakeflow Jobs runs the Databricks sidePattern; two schedulers is a costDrop where Lakeflow can do it

Is there a benchmark that names a clear winner?

No. Every benchmark we found compares one side against a weaker version of itself, or one engine against another. None runs the same data through A and B. Read each claim for what it actually compares:

ClaimWho measuredWhat it really comparesDoes it pick A or B?
S3 Tables: up to 3x faster queries, up to 10x more transactions per secondAWSS3 Tables against Iceberg in an ordinary bucket that nobody compactsNo. It measures maintained against unmaintained tables.
S3 Tables: about 2x faster than medium-file Iceberg and 4.6x to 37.6x faster than small-file Iceberg on Glue Spark (TPC-H, 104 GB); ingestion 1.7x to 3.1x faster; Athena was slower on S3 Tables than on plain Iceberg; storage about 17.5% dearerLoka (AWS partner)S3 Tables against self-managed Iceberg, same AWS enginesNo. Same catalog family, different maintenance.
Predictive Optimization: up to 20x faster selective queries, 2x lower storage cost; Anker 2x faster queries, 50% lower storageDatabricksUnity Catalog managed tables against unmanaged onesNo. Same idea as the S3 Tables claim, on the other side.
Redshift Serverless on Iceberg over 2x faster in 2025 (TPC-DS 10 TB)AWSRedshift this year against Redshift last yearNo. Helps both A and B, since Redshift reads Iceberg either way.
EMR Spark 4.5x faster than open-source Spark on Iceberg (TPC-DS 3 TB)AWSEMR runtime against open-source SparkNo. An engine claim.
StarRocks 1.3x faster than Databricks Photon on catalog queries (TPC-DS 1 TB)CelerData (vendor)Engines; Photon ran on Delta, not managed IcebergNo. Engines, and a vendor test.
A head-to-head: same data, A against B, both engines, ETL plus queries plus chores plus costNobody, as of November 2026-This is the benchmark that would tell the story, and it does not exist.
The clear story the numbers do tell: the big wins on both sides come from two things, giving the tables one owner of the chores (S3 Tables or Predictive Optimization) and deleting copies. Neither depends on which catalog you pick. So speed and cost do not decide A versus B; where your work runs does, because the side that writes gets every feature and the guest side gets read-only access. That is why Databricks-centred teams land on A and AWS-only teams land on B.

The benchmark that would settle it

If a client wants numbers, run this small test on their own account before committing. It is a few days of work and answers the question for their workload, which no public benchmark does.

PartWhat to do
DataTPC-DS at 1 TB as gold, plus a daily MERGE of 1% changed rows into silver
Run ADatabricks serverless writes managed Iceberg in Unity Catalog with Predictive Optimization; Athena and Redshift read gold through Glue federation
Run BGlue or EMR Spark writes S3 Tables with Glue and Lake Formation; Databricks reads gold as foreign tables
MeasureETL wall time and cost, query time per engine on both sides, maintenance cost (DBUs or S3 Tables fees), freshness from write to readable, and hours of setup
Expected resultEach side wins where its own engines write and maintain the tables; the guest side pays in features and governance, not speed. That is the claim to test.

Step 10 · synthesis

Who wins

Winner for an AWS platform where Databricks does most of the work: Architecture A. One owner for every table, every Databricks feature working on it, automatic lineage and audit, serverless everywhere, AWS engines reading gold in place, and Iceberg engines still able to write through Unity Catalog's REST catalog. It is also the design that moves to Azure, Fabric or Google Cloud with the fewest changes.

Architecture B wins only when AWS is the real home: Glue, EMR, Athena, Redshift, Firehose and Flink do the work, and Databricks is a light guest or absent. Then S3 Tables, Glue and Lake Formation are native, cheaper and simpler, as Razor Group shows.
If this is truePickWhy
Databricks does most of the work (pipelines, ML, Genie, most SQL)AEverything Databricks offers works on owned tables; AWS engines still read gold in place.
AWS engines do most of the work; Databricks is used by one team, or not at allBS3 Tables, Glue and Lake Formation are native for Glue, EMR, Athena, Redshift, Firehose and Flink. Databricks reads through federation.
Streams arrive through Firehose or Managed Flink and must be queryable in Athena within secondsB for that laneKeep that domain AWS-owned and let Databricks read it; this is the one place a split by domain makes sense.
Separate teams already committed to each side, by domainSplit by domainEach table still has exactly one owner; catalogs federate each other read-only. Highest effort to run.
You may move to Azure, Fabric or Google Cloud laterAUnity Catalog is the same on every cloud; only the AWS reader door changes.

Whichever you choose, write it down as one rule: one table, one catalog, one writer path, one owner of the chores. List the other architecture under "options considered" with its reason; that is what a reviewer will ask for.

FAQ

Questions people ask

Can Databricks write bronze to silver to gold when AWS owns the catalog?

Only from classic clusters, using the open-source Iceberg library pointed at the Glue Iceberg REST endpoint with Lake Formation credentials. It works, but those writes skip Unity Catalog permissions, lineage and audit, cannot use serverless or Lakeflow pipelines, and you maintain library versions. Through Unity Catalog itself, AWS-owned tables are read-only foreign tables.

Can AWS engines write when Unity Catalog owns the catalog?

Spark on EMR or Glue, Trino and Snowflake can write managed Iceberg tables through Unity Catalog's Iceberg REST catalog with a Databricks service principal, governed by Unity Catalog grants. Athena, Redshift and QuickSight read gold through Glue federation and do not write. Firehose and DMS land files that Auto Loader loads.

Are Glue with Lake Formation and the SageMaker lakehouse the same thing?

Mostly. Glue Data Catalog is the catalog, Lake Formation is the permissions on it, and the SageMaker lakehouse is AWS's umbrella name for the two, reaching across S3, S3 Tables, Redshift storage and federated catalogs. SageMaker Catalog (formerly DataZone) is a different, business-facing layer.

Where does Redshift fit?

As one more reader of the same Iceberg tables, through federation, if the client already uses it. Loading data into Redshift's own storage creates a second copy with its own permissions, which is the pattern the case studies moved away from.

Why not let both catalogs own tables?

Two writers on one table means two pointers, conflicting commits and clean-ups that delete each other's files. Split by domain only if teams force it, and even then every table has exactly one owner.

Is Iceberg the right format on Databricks?

It is a sound choice for openness. Inside Databricks, Delta still has features managed Iceberg lacks today (row filters, column masks, keys, Lakeflow streaming tables). Delta with Iceberg reads (UniForm) gives AWS engines Iceberg without giving those up, so many teams mix the two.

Sources

Checked against

Checked in November 2026. Federation writes, managed Iceberg status and fine-grained access for outside engines change every few months; check before you commit.

Related

Where it connects

The data platform, whole · Who maintains your tables · Delta vs Iceberg on Databricks · AWS + Databricks, ETL end to end · AWS data services catalog · Streaming on AWS and Databricks