Home › Learn › AWS-native vs Databricks lakehouse

Platform comparison · data engineering

AWS-native Iceberg vs Databricks: a like-for-like lakehouse comparison with limitations

S3 Tables, Glue and Lake Formation with EMR or Glue Spark, against Unity Catalog with Delta, UniForm, Photon and Predictive Optimization: format, compute, upkeep, pipelines, Auto CDC and SCD2, Power BI, governance, lineage, orchestration, cost and hidden cost, with a scorecard and the current documented limitations.

Prices, defaults, limits and preview status change often. No prices are quoted here. Limitations were checked against vendor documentation on 2026-11-21; confirm them on your runtime and region.

The setup

Two stacks, compared like for like

Both stacks keep data as Parquet files in S3 and add a table layer on top. The question is who supplies the table format, the catalog, the engine, the upkeep and the governance, and what each choice costs you in money and in people.

Stack A: AWS-nativeStack B: Databricks
Table formatIceberg, in S3 Tables (AWS-managed table buckets)Delta, with UniForm so other engines can read it as Iceberg
CatalogAWS Glue Data Catalog and Lake FormationUnity Catalog (UC)
ETL computeEMR (EC2, Serverless or EKS) or Glue SparkJobs compute, serverless jobs, Lakeflow declarative pipelines
Table upkeepS3 Tables built-in maintenancePredictive Optimization (PO)
BI path to Power BIAthena, Redshift or a Trino serviceDatabricks SQL warehouse

Read this first. Features, prices and limits on both sides move quickly. The comparison is from the author's knowledge, not a live price check. Use it to decide what to measure, then run the same pipeline, the same SQL and the same Power BI report on both and compare monthly cost.

This page is the wide comparison. For the narrower question of who owns the catalog, see Iceberg lakehouse on AWS and Databricks: who owns the catalog?

Format

Format and openness

QuestionWinnerWhy
Several engines must write the same tableAIceberg is writable by every engine. UniForm is read-only for outsiders.
Readers are mostly Databricks and BIBDelta is native there, with the most features.

More: Delta vs Iceberg, UniForm.

Compute

Compute, serverless and performance

NeedDatabricksAWS-native
Fewest decisionsServerless jobs and SQL warehousesEMR Serverless, Glue
Cheapest large batchJobs compute on spot, with PhotonEMR on EC2 with spot, Graviton, instance fleets
Low-priority batchSpot with fallbackGlue Flex execution
ControlPolicies, pools, access modesFull control of instances, networking, versions

Winner: A for price per hour and control, B for start-up time and effort. Raw performance is a tie until you measure. See Databricks compute and cost.

Upkeep

Table upkeep, layout and compression

TopicA: S3 TablesB: UC and PO
CompactionAutomatic binpack, newer sort and z-order options. Billed per GB and per object processed.PO runs OPTIMIZE automatically. Billed in serverless DBUs.
Snapshot and file cleanupSnapshot expiry and orphan-file removalVACUUM by PO
LayoutPartition evolution, hidden partitioning, sort orderLiquid clustering, which can choose its own keys. No Iceberg equivalent.
StatisticsIceberg column metricsFirst 32 columns by default, tunable. PO runs ANALYZE.
ScopeS3 Tables onlyUC managed tables only

Compression is not a differentiator. Both write Parquet and you can choose the same codec. File size and layout decide real speed and storage cost. Without S3 Tables, plain Iceberg on EMR means you schedule rewrite_data_files, expire_snapshots and remove_orphan_files yourself.

Winner: B for breadth (layout and automatic keys). A is simpler to predict. More: who maintains your tables, maintenance strategy, slow writes.

Write path

Optimized write and AQE

Pipelines

Declarative pipelines, Auto CDC and SCD Type 2

TopicDatabricks (Lakeflow declarative pipelines)AWS-nativeWinner
Declarative pipelinesYou define tables in SQL or Python. The engine orders them, retries, tracks lineage and refreshes incrementally. Expectations for quality.Glue jobs or EMR steps wired by Step Functions or MWAA. dbt is the closest declarative tool. Newer Glue catalog materialized views for Iceberg; check maturity.Databricks
Auto CDCGive a key and a sequence column. Late and out-of-order changes are resolved, deletes flagged, snapshots diffed.Write MERGE INTO yourself, including dedupe by sequence. DMS or Flink CDC delivers the changes.Databricks
SCD Type 2STORED AS SCD TYPE 2 closes the old row and opens the new one, with optional column-limited history.Two-step MERGE, or dbt snapshots. Correct, but you own the edge cases.Databricks (less code)

Your own PySpark gives full control. See SCD2: AUTO CDC or your own PySpark?, declarative pipelines and medallion architecture. AWS wins one thing here: zero-ETL integrations from Aurora, RDS and DynamoDB.

BI

Power BI, metric views and the semantic layer

Winner: B, clearly. More: Power BI vs Databricks AI/BI, metric views.

Security and governance

Security, lineage, governance and tagging

TopicDatabricksAWS-nativeWinner
Data-level securityOne layer: UC grants, row filters, column masks, tag-based policies. Short-lived credentials per table.Two layers: IAM and Lake Formation (table, column, row, cell filters, LF-tags). Fine-grained control from EMR needs a specific setup.Databricks for consistency
Infrastructure securityPrivate Link, IP rules, serverless egress policies, customer-managed keysVPC endpoints, KMS, GuardDuty, Macie, Config, CloudTrail, SCPsAWS for breadth
LineageAutomatic table and column lineage from notebooks, jobs, pipelines, SQL and dashboards, queryable in system tables. Only for work on UC-governed compute.No built-in column lineage for Glue or EMR. Add OpenLineage listeners and send events to SageMaker Catalog or another tool.Databricks
GovernanceOne catalog: search, tags, certification, quality monitoring, access requests, Delta Sharing, clean rooms, audit tablesSageMaker Catalog, Glue Data Quality, Macie, Lake Formation cross-account, CloudTrail: several servicesDatabricks for one place
TaggingUC tags usable in access policies. Cost tags on clusters and jobs. Budget policies tag serverless usage.LF-tags for data. Resource tags for cost, which must be activated before they appear. Organizations tag policies enforce them.Tie

More: tagging best practices, row-level security and masking.

Catalog

Unity Catalog features and what stands in for them on AWS

UC featureClosest AWS-native pieceGap
Catalogs, schemas, tables, viewsGlue Data CatalogNone
Volumes (governed files)S3 plus Lake FormationFiles are less governed
Row filters and column masksLake Formation filtersFine-grained on EMR needs setup
Tag-based access policiesLF-tagsRoughly equal
LineageOpenLineage into SageMaker CatalogYou assemble it
System tables (audit, billing, usage)CloudTrail, cost and usage reportsNot SQL-first
Metric viewsNone at catalog levelNo semantic layer in the catalog
Materialized viewsNewer Glue catalog materialized viewsLess mature
Federation to other databasesAthena federated queries, RedshiftDifferent tools
Delta Sharing and clean roomsData Exchange, Clean RoomsDifferent products
Iceberg REST endpointGlue Iceberg RESTEqual
Model and function registrySageMaker Model RegistrySeparate products

UC is one product. The AWS equivalent is four to six services that you integrate. Winner: Databricks on cohesion, AWS on how many other AWS services plug in natively.

Running it

Orchestration and Spark version

TopicDatabricksAWS-native
OrchestrationLakeflow Jobs: task DAGs, for-each, if/else, repair run, file-arrival and table-update triggers, cron, continuous, run-as service principal, Asset Bundles for codeStep Functions, MWAA (managed Airflow, billed hourly even when idle), Glue Workflows, EventBridge Scheduler. Big Airflow operator library for cross-system flows.
Spark versionDatabricks Runtime number, with LTS releases. Contains Spark plus Databricks patches, Photon and Delta features. Serverless is versionless, with environment versions you can pin.EMR release label or Glue version. Contains Spark plus the AWS runtime and bundled Iceberg libraries. You pin it.

Orchestration winner: Databricks inside the platform, Airflow or AWS across many systems. Spark version: a tie. AWS gives control, Databricks gives less upgrade work. Runtime numbers are not Spark numbers, so compare the Spark version inside each. Newer Spark can change error behaviour and defaults, so test before upgrading either side. See Asset Bundles.

Cost

Cost, system tables and cost transparency

DatabricksAWS-native
Classic computeDBUs × rate, plus VMs billed by AWSEC2 plus an EMR fee, or Glue DPU-hours
ServerlessOne DBU price, VMs includedEMR Serverless per vCPU-hour and GB-hour
DiscountsCommit-based contract discountsSavings Plans, spot, enterprise agreements
MaintenancePO serverless DBUsS3 Tables processing fees
Cost dataSystem tables: usage, list prices, compute, job runs, query history, audit. SQL-first. Attribution by job, cluster, warehouse, query and tag. Shows DBUs at list price, not your discounted price.Cost and usage reports to S3, queried with Athena, plus Cost Explorer. Real billed dollars for EC2, S3 and NAT. Tags must be activated first. No native per-job Spark cost.
Blind spotsCloud VM and storage cost sit in the AWS bill. Serverless internals are hidden.No per-query or per-job split for Spark

Observability winner: a tie, with the combination best. Databricks tells you who and what spent it. AWS tells you the whole bill. Tag everything and join Databricks usage to the AWS export from day one; that join is hidden work in the Databricks cost story.

Hidden costs on both sides

CostAB
Compute premiumEMR fee on EC2, or Glue DPU-hoursDBU premium, plus the Photon multiplier
StorageS3 Tables rate is above standard S3, plus old snapshots and a per-object monitoring feeStandard S3 in your account, plus old versions until VACUUM
RequestsMany small files mean many callsSame
NetworkNAT gateway fees without an S3 gateway endpoint, cross-AZ shuffle, cross-region readsSame on classic compute, plus serverless network charges and Delta Sharing or cross-cloud egress
Idle timeLong-lived EMR clusters, always-on MWAAWarehouses or clusters without auto-stop
BIAthena scan fees, Redshift, gatewaysSQL warehouse DBUs
PeopleCluster tuning, library versions, wiring services togetherPlatform skills and DBU forecasting

Same-region traffic between S3 and compute is free on both sides. Moving data out of the region or cloud is where bills grow. More: how billing works, system tables and what reading them costs, egress and ingress.

Hero features

What you miss if you leave Databricks out

Where AWS-native wins

Limitations

Limitations to know before you decide

Checked against vendor documentation on 21 November 2026. Where two documents disagreed, the table says so. Treat each row as "confirm this on your runtime and region before relying on it".

Databricks side

AreaLimitationSource
UniForm (Iceberg reads of Delta)Iceberg clients are read-only. Column mapping is required. Iceberg v2 reads do not work on tables with deletion vectors (Iceberg v3 supports them). Iceberg reads cannot be enabled on materialized views or streaming tables. Parquet files use Zstandard instead of Snappy. Delta Sharing recipients read the table as Delta only.Databricks: read Delta tables with Iceberg clients
Managed Iceberg tables in UCReports of general availability in 2026 conflict with pages that still say preview, so check your cloud. Listed limits: needs runtime 16.4 LTS or later, no primary or foreign keys, no partitioning by expressions such as bucket() or days(), partition evolution only through external engines on the Iceberg REST catalog, external engines cannot read UC views, Parquet only, some nested and special types unsupported, no credential vending on default storage. The documentation also says creating them needs Predictive Optimization enabled.Databricks: Apache Iceberg
Predictive OptimizationUC managed tables only. It does not run on streaming tables, materialized views, external tables or tables shared in by Delta Sharing. Needs the Premium plan and a supported region. Billed as a serverless jobs SKU. Older pages say it skips tables using Z-order or liquid clustering, newer ones describe automatic liquid clustering, so the docs differ by version.Databricks: predictive optimization
Liquid clusteringAt most four clustering keys. Cannot be combined with partitioning or Z-order. Clustering on write only happens above a size threshold (for example 64 MB with one key, 1 GB with four). Automatic key selection needs runtime 15.4 LTS or later and UC managed tables.Databricks: liquid clustering
Serverless computeOnly Spark Connect APIs: no RDD APIs, no sparkContext, no global temporary views, no compute policies. Maximum run time 7 days. Streaming triggers such as continuous and processing-time are not supported, so use available-now. Sources differ on Scala and R support, so check the current page.Databricks: serverless limitations
Declarative pipelines and Auto CDCIdentity columns are not supported on Auto CDC targets. A dataset can be defined only once across pipelines (append flows into a streaming table are the exception). pivot() is not supported. A workspace allows 1,000 concurrent pipeline updates.Databricks: pipeline limitations
LineageThe lineage system tables keep a rolling one-year window and hold only a subset of events, recorded when lineage can be inferred. Column lineage needs both source and target referenced by table name. Only events from UC-governed work (pipelines, notebooks, jobs, SQL, dashboards) appear, so mixing governed and ungoverned compute leaves gaps.Databricks: lineage system tables
Metric viewsThe main docs say metric views do not support lineage, Delta Sharing or Lakehouse Monitoring. In Power BI, the BI compatibility mode needs DirectQuery, a recent Desktop build, and cannot join a metric view to other tables. One Microsoft page says that mode was removed from the connector while a Databricks page calls it Beta, so confirm the status.Databricks: Power BI and metric views, Microsoft: BI metric view

AWS side

AreaLimitationSource
S3 Tables costYou pay for storage, requests, an object monitoring fee and optional compaction (per object and per GB processed). Compaction prices were cut by up to 90 percent from July 2025. Sort and z-order compaction can cost more than binpack. Search results disagreed by tenfold on the monitoring rate, so read the pricing page.AWS: maintenance for tables, AWS: compaction cost cut
S3 Tables compactionTarget file size must be between 64 MB and 512 MB. Snapshot management is configured per table.AWS: maintenance for tables
S3 Tables with Glue and Lake FormationColumn names are converted to lowercase. The SearchTables API cannot search S3 tables. Access errors in Athena are usually fixed with Lake Formation grants on the table-bucket catalog and database.AWS: S3 tables catalog integration limitations
Lake Formation with EMR (fine-grained access)Spark jobs only, and Hive or Iceberg tables only. A single Spark session per job. Dynamic resource allocation cannot be turned off. No custom images on EMR Serverless. Iceberg works only with the session catalog. Registered Iceberg tables expose only some metadata tables and hide sensitive columns. Write support differs by release: older pages say data-changing statements are limited, a newer page says EMR 7.12 and later supports DML and DDL through Lake Formation credentials. RDD and nested-column access control are unsupported in the version I found.AWS: EMR Serverless and Lake Formation, AWS: EMR and Lake Formation considerations
Glue materialized views for IcebergNeed Glue 5.1 or later and Spark 3.5.6 or later. Source tables must be Iceberg tables in the Glue Data Catalog. Results are Iceberg tables in S3 Tables or general-purpose buckets. I did not find the full limits list, so read the Glue page.AWS Glue: materialized views
Power BIOne community answer says Power BI has no native Iceberg connector, with OneLake virtualization as the workaround (the thread's date is unclear). The Athena connector lists DirectQuery as supported, and users report failures after publishing or with native queries. Those are user reports, not official limits.Microsoft: Athena connector

What this changes in the comparison. Four things matter most for the decision. (1) Managed Iceberg tables in UC are tied to Predictive Optimization, so "Iceberg on Databricks" is not independent of the managed stack. (2) UniForm stays read-only, so a multi-writer design must be Iceberg-native. (3) Fine-grained Lake Formation on EMR works but with a long list of conditions, so test your exact EMR release and mode. (4) Power BI on metric views is the least settled part of the Databricks story, so pilot it before promising it.

What this list cannot show. Support tickets and escalations raised by real customers are private to those customers, so neither vendor publishes them. The limits above come from public documentation and a few public user reports. Before you commit, ask each vendor's account team for known issues on your exact runtime and region, and pilot on your own data.

Verdict

Scorecard

DimensionWinnerWhy
OpennessANative Iceberg, multi-writer
Governance depthBLineage, tag-based policies, system tables
Cost per compute hourASpot, no DBU premium
Engineering effortBServerless, declarative, fewer parts
Table layoutBLiquid clustering
Upkeep simplicityTieS3 Tables and PO are both automatic
Power BIBNative connector, passes user identity
Semantic layerBMetric views
Declarative pipelines, Auto CDC, SCD2BLess code
AWS integrationAIAM, Athena, Redshift, zero-ETL
Lock-in riskAOpen format and catalog
Cost visibilityTieDatabricks per job, AWS for the whole bill

The pattern. Databricks wins where the value is one connected product: lineage, governance, declarative pipelines, Power BI, metric views. AWS-native wins where the value is cost, control and openness: compute price, multi-engine Iceberg, AWS integrations, no second vendor. The decision depends less on features and more on whether you have a platform team to do the assembly work.

If this describes youChoose
Power BI is the main consumer, the team is small, you want fast governed deliveryB: Databricks
Big steady ETL, a strong AWS platform team, several engines reading and writingA: AWS-native
Both needsHybrid: Databricks reads S3 Tables through catalog federation, and UniForm or managed Iceberg lets Athena or Redshift read Databricks tables

FAQ

Questions people ask

Does Databricks or AWS-native cost less?

Per compute hour, AWS-native on spot and Graviton is usually cheaper. Databricks adds a DBU premium but can save engineering time and run faster with Photon. Measure the same pipeline on both and compare monthly cost, including storage requests, network and idle time.

Can Athena or Redshift read Databricks tables?

Yes, through UniForm, which exposes Delta tables as Iceberg for reading. It is read-only, needs column mapping, and has limits such as deletion vectors on Iceberg v2 readers. Check the current UniForm page.

Which is better for Power BI?

Databricks, because Power BI has a native connector with DirectQuery and passes user identity. Iceberg on S3 has no native Power BI connector, so you go through Athena, Redshift or a Trino service. The metric view path in Power BI is the least settled part, so pilot it.

Is compression a reason to choose one?

No. Both write Parquet and you can choose the same codec. File size and table layout matter far more.

Next

Where to go next

Related on this site: Iceberg lakehouse: who owns the catalog? · Databricks compute and cost · Delta vs Iceberg · Who maintains your tables · SCD2: AUTO CDC or own PySpark? · Power BI vs Databricks AI/BI · Metric views · AWS data services catalog

All data engineering · Databricks on this site