Home › Learn › AWS-native vs Databricks lakehouse
Platform comparison · data engineering
AWS-native Iceberg vs Databricks: a like-for-like lakehouse comparison with limitations
S3 Tables, Glue and Lake Formation with EMR or Glue Spark, against Unity Catalog with Delta, UniForm, Photon and Predictive Optimization: format, compute, upkeep, pipelines, Auto CDC and SCD2, Power BI, governance, lineage, orchestration, cost and hidden cost, with a scorecard and the current documented limitations.
Prices, defaults, limits and preview status change often. No prices are quoted here. Limitations were checked against vendor documentation on 2026-11-21; confirm them on your runtime and region.
The setup
Two stacks, compared like for like
Both stacks keep data as Parquet files in S3 and add a table layer on top. The question is who supplies the table format, the catalog, the engine, the upkeep and the governance, and what each choice costs you in money and in people.
| Stack A: AWS-native | Stack B: Databricks | |
|---|---|---|
| Table format | Iceberg, in S3 Tables (AWS-managed table buckets) | Delta, with UniForm so other engines can read it as Iceberg |
| Catalog | AWS Glue Data Catalog and Lake Formation | Unity Catalog (UC) |
| ETL compute | EMR (EC2, Serverless or EKS) or Glue Spark | Jobs compute, serverless jobs, Lakeflow declarative pipelines |
| Table upkeep | S3 Tables built-in maintenance | Predictive Optimization (PO) |
| BI path to Power BI | Athena, Redshift or a Trino service | Databricks SQL warehouse |
Read this first. Features, prices and limits on both sides move quickly. The comparison is from the author's knowledge, not a live price check. Use it to decide what to measure, then run the same pipeline, the same SQL and the same Power BI report on both and compare monthly cost.
This page is the wide comparison. For the narrower question of who owns the catalog, see Iceberg lakehouse on AWS and Databricks: who owns the catalog?
Format
Format and openness
- A is native Iceberg. Spark, Trino, Flink, Athena, Snowflake and DuckDB can read and write it, so no single product owns the table.
- B is Delta. UniForm writes Iceberg metadata after each commit, so outside engines can read but not write. The metadata can lag a little, and some Delta features do not carry across. Databricks also offers managed Iceberg tables in UC; check how mature that is before you rely on it.
| Question | Winner | Why |
|---|---|---|
| Several engines must write the same table | A | Iceberg is writable by every engine. UniForm is read-only for outsiders. |
| Readers are mostly Databricks and BI | B | Delta is native there, with the most features. |
More: Delta vs Iceberg, UniForm.
Compute
Compute, serverless and performance
- A: EMR on spot instances and Graviton is the cheapest per hour. EMR on EC2 starts in minutes. EMR Serverless and Glue start faster. You tune memory, versions and libraries yourself.
- B: Photon (a vectorised engine) usually speeds up scans, joins and MERGE. Serverless starts in seconds. You pay a DBU premium on top of the machines.
- Each vendor publishes benchmarks that favour itself. Measure your own pipeline.
| Need | Databricks | AWS-native |
|---|---|---|
| Fewest decisions | Serverless jobs and SQL warehouses | EMR Serverless, Glue |
| Cheapest large batch | Jobs compute on spot, with Photon | EMR on EC2 with spot, Graviton, instance fleets |
| Low-priority batch | Spot with fallback | Glue Flex execution |
| Control | Policies, pools, access modes | Full control of instances, networking, versions |
Winner: A for price per hour and control, B for start-up time and effort. Raw performance is a tie until you measure. See Databricks compute and cost.
Upkeep
Table upkeep, layout and compression
| Topic | A: S3 Tables | B: UC and PO |
|---|---|---|
| Compaction | Automatic binpack, newer sort and z-order options. Billed per GB and per object processed. | PO runs OPTIMIZE automatically. Billed in serverless DBUs. |
| Snapshot and file cleanup | Snapshot expiry and orphan-file removal | VACUUM by PO |
| Layout | Partition evolution, hidden partitioning, sort order | Liquid clustering, which can choose its own keys. No Iceberg equivalent. |
| Statistics | Iceberg column metrics | First 32 columns by default, tunable. PO runs ANALYZE. |
| Scope | S3 Tables only | UC managed tables only |
Compression is not a differentiator. Both write Parquet and you can choose the same codec. File size and layout decide real speed and storage cost. Without S3 Tables, plain Iceberg on EMR means you schedule rewrite_data_files, expire_snapshots and remove_orphan_files yourself.
Winner: B for breadth (layout and automatic keys). A is simpler to predict. More: who maintains your tables, maintenance strategy, slow writes.
Write path
Optimized write and AQE
- Optimized write is an extra shuffle before the write so files come out a good size. Iceberg has the same idea as a setting, the write distribution mode (hash or range). Same mechanism, different switch. Winner: tie.
- AQE (adaptive query execution) is open-source Spark, on by default in recent versions, so EMR and Glue have it too. It merges small shuffle partitions, switches join strategy at run time and splits skewed partitions. Databricks adds some tuning on top. Winner: tie.
- Neither fixes a bad partition count before a write or a bad table layout.
Pipelines
Declarative pipelines, Auto CDC and SCD Type 2
| Topic | Databricks (Lakeflow declarative pipelines) | AWS-native | Winner |
|---|---|---|---|
| Declarative pipelines | You define tables in SQL or Python. The engine orders them, retries, tracks lineage and refreshes incrementally. Expectations for quality. | Glue jobs or EMR steps wired by Step Functions or MWAA. dbt is the closest declarative tool. Newer Glue catalog materialized views for Iceberg; check maturity. | Databricks |
| Auto CDC | Give a key and a sequence column. Late and out-of-order changes are resolved, deletes flagged, snapshots diffed. | Write MERGE INTO yourself, including dedupe by sequence. DMS or Flink CDC delivers the changes. | Databricks |
| SCD Type 2 | STORED AS SCD TYPE 2 closes the old row and opens the new one, with optional column-limited history. | Two-step MERGE, or dbt snapshots. Correct, but you own the edge cases. | Databricks (less code) |
Your own PySpark gives full control. See SCD2: AUTO CDC or your own PySpark?, declarative pipelines and medallion architecture. AWS wins one thing here: zero-ETL integrations from Aurora, RDS and DynamoDB.
BI
Power BI, metric views and the semantic layer
- B: Power BI has a native Databricks connector with Import and DirectQuery on a SQL warehouse. It passes the user's identity to UC and respects row and column rules. Photon and result caching handle concurrency well. UC metric views define measures once for SQL, dashboards and Genie (and mean more lock-in).
- A: Power BI has no good direct Iceberg connector. You go through Athena (billed per TB scanned, awkward for DirectQuery), Redshift or a Trino service, usually with a gateway. You can also shortcut S3 into Fabric OneLake, which adds a second platform. There is no catalog-level metric layer.
Winner: B, clearly. More: Power BI vs Databricks AI/BI, metric views.
Security and governance
Security, lineage, governance and tagging
| Topic | Databricks | AWS-native | Winner |
|---|---|---|---|
| Data-level security | One layer: UC grants, row filters, column masks, tag-based policies. Short-lived credentials per table. | Two layers: IAM and Lake Formation (table, column, row, cell filters, LF-tags). Fine-grained control from EMR needs a specific setup. | Databricks for consistency |
| Infrastructure security | Private Link, IP rules, serverless egress policies, customer-managed keys | VPC endpoints, KMS, GuardDuty, Macie, Config, CloudTrail, SCPs | AWS for breadth |
| Lineage | Automatic table and column lineage from notebooks, jobs, pipelines, SQL and dashboards, queryable in system tables. Only for work on UC-governed compute. | No built-in column lineage for Glue or EMR. Add OpenLineage listeners and send events to SageMaker Catalog or another tool. | Databricks |
| Governance | One catalog: search, tags, certification, quality monitoring, access requests, Delta Sharing, clean rooms, audit tables | SageMaker Catalog, Glue Data Quality, Macie, Lake Formation cross-account, CloudTrail: several services | Databricks for one place |
| Tagging | UC tags usable in access policies. Cost tags on clusters and jobs. Budget policies tag serverless usage. | LF-tags for data. Resource tags for cost, which must be activated before they appear. Organizations tag policies enforce them. | Tie |
More: tagging best practices, row-level security and masking.
Catalog
Unity Catalog features and what stands in for them on AWS
| UC feature | Closest AWS-native piece | Gap |
|---|---|---|
| Catalogs, schemas, tables, views | Glue Data Catalog | None |
| Volumes (governed files) | S3 plus Lake Formation | Files are less governed |
| Row filters and column masks | Lake Formation filters | Fine-grained on EMR needs setup |
| Tag-based access policies | LF-tags | Roughly equal |
| Lineage | OpenLineage into SageMaker Catalog | You assemble it |
| System tables (audit, billing, usage) | CloudTrail, cost and usage reports | Not SQL-first |
| Metric views | None at catalog level | No semantic layer in the catalog |
| Materialized views | Newer Glue catalog materialized views | Less mature |
| Federation to other databases | Athena federated queries, Redshift | Different tools |
| Delta Sharing and clean rooms | Data Exchange, Clean Rooms | Different products |
| Iceberg REST endpoint | Glue Iceberg REST | Equal |
| Model and function registry | SageMaker Model Registry | Separate products |
UC is one product. The AWS equivalent is four to six services that you integrate. Winner: Databricks on cohesion, AWS on how many other AWS services plug in natively.
Running it
Orchestration and Spark version
| Topic | Databricks | AWS-native |
|---|---|---|
| Orchestration | Lakeflow Jobs: task DAGs, for-each, if/else, repair run, file-arrival and table-update triggers, cron, continuous, run-as service principal, Asset Bundles for code | Step Functions, MWAA (managed Airflow, billed hourly even when idle), Glue Workflows, EventBridge Scheduler. Big Airflow operator library for cross-system flows. |
| Spark version | Databricks Runtime number, with LTS releases. Contains Spark plus Databricks patches, Photon and Delta features. Serverless is versionless, with environment versions you can pin. | EMR release label or Glue version. Contains Spark plus the AWS runtime and bundled Iceberg libraries. You pin it. |
Orchestration winner: Databricks inside the platform, Airflow or AWS across many systems. Spark version: a tie. AWS gives control, Databricks gives less upgrade work. Runtime numbers are not Spark numbers, so compare the Spark version inside each. Newer Spark can change error behaviour and defaults, so test before upgrading either side. See Asset Bundles.
Cost
Cost, system tables and cost transparency
| Databricks | AWS-native | |
|---|---|---|
| Classic compute | DBUs × rate, plus VMs billed by AWS | EC2 plus an EMR fee, or Glue DPU-hours |
| Serverless | One DBU price, VMs included | EMR Serverless per vCPU-hour and GB-hour |
| Discounts | Commit-based contract discounts | Savings Plans, spot, enterprise agreements |
| Maintenance | PO serverless DBUs | S3 Tables processing fees |
| Cost data | System tables: usage, list prices, compute, job runs, query history, audit. SQL-first. Attribution by job, cluster, warehouse, query and tag. Shows DBUs at list price, not your discounted price. | Cost and usage reports to S3, queried with Athena, plus Cost Explorer. Real billed dollars for EC2, S3 and NAT. Tags must be activated first. No native per-job Spark cost. |
| Blind spots | Cloud VM and storage cost sit in the AWS bill. Serverless internals are hidden. | No per-query or per-job split for Spark |
Observability winner: a tie, with the combination best. Databricks tells you who and what spent it. AWS tells you the whole bill. Tag everything and join Databricks usage to the AWS export from day one; that join is hidden work in the Databricks cost story.
Hidden costs on both sides
| Cost | A | B |
|---|---|---|
| Compute premium | EMR fee on EC2, or Glue DPU-hours | DBU premium, plus the Photon multiplier |
| Storage | S3 Tables rate is above standard S3, plus old snapshots and a per-object monitoring fee | Standard S3 in your account, plus old versions until VACUUM |
| Requests | Many small files mean many calls | Same |
| Network | NAT gateway fees without an S3 gateway endpoint, cross-AZ shuffle, cross-region reads | Same on classic compute, plus serverless network charges and Delta Sharing or cross-cloud egress |
| Idle time | Long-lived EMR clusters, always-on MWAA | Warehouses or clusters without auto-stop |
| BI | Athena scan fees, Redshift, gateways | SQL warehouse DBUs |
| People | Cluster tuning, library versions, wiring services together | Platform skills and DBU forecasting |
Same-region traffic between S3 and compute is free on both sides. Moving data out of the region or cloud is where bills grow. More: how billing works, system tables and what reading them costs, egress and ingress.
Hero features
What you miss if you leave Databricks out
- Photon, Predictive I/O and liquid clustering
- Lakeflow declarative pipelines, Auto Loader and Lakeflow Connect managed connectors
- UC lineage, tag-based rules and system tables for cost and audit
- Delta Sharing and clean rooms
- Genie, AI/BI dashboards and metric views
- Model serving, Vector Search and agent tools inside the same governed place
- Databricks Apps and Lakebase (Postgres)
- Serverless that starts in seconds, and repair run for failed jobs
Where AWS-native wins
- Lower compute cost for large steady ETL on spot and Graviton
- Real multi-engine Iceberg with several writers
- One vendor, one bill, existing AWS commitments, IAM everywhere
- Athena for occasional ad-hoc queries without a warehouse
- Zero-ETL from AWS databases and Managed Flink for streaming
- Less lock-in, with S3 Tables maintenance that needs no extra product
Limitations
Limitations to know before you decide
Checked against vendor documentation on 21 November 2026. Where two documents disagreed, the table says so. Treat each row as "confirm this on your runtime and region before relying on it".
Databricks side
| Area | Limitation | Source |
|---|---|---|
| UniForm (Iceberg reads of Delta) | Iceberg clients are read-only. Column mapping is required. Iceberg v2 reads do not work on tables with deletion vectors (Iceberg v3 supports them). Iceberg reads cannot be enabled on materialized views or streaming tables. Parquet files use Zstandard instead of Snappy. Delta Sharing recipients read the table as Delta only. | Databricks: read Delta tables with Iceberg clients |
| Managed Iceberg tables in UC | Reports of general availability in 2026 conflict with pages that still say preview, so check your cloud. Listed limits: needs runtime 16.4 LTS or later, no primary or foreign keys, no partitioning by expressions such as bucket() or days(), partition evolution only through external engines on the Iceberg REST catalog, external engines cannot read UC views, Parquet only, some nested and special types unsupported, no credential vending on default storage. The documentation also says creating them needs Predictive Optimization enabled. | Databricks: Apache Iceberg |
| Predictive Optimization | UC managed tables only. It does not run on streaming tables, materialized views, external tables or tables shared in by Delta Sharing. Needs the Premium plan and a supported region. Billed as a serverless jobs SKU. Older pages say it skips tables using Z-order or liquid clustering, newer ones describe automatic liquid clustering, so the docs differ by version. | Databricks: predictive optimization |
| Liquid clustering | At most four clustering keys. Cannot be combined with partitioning or Z-order. Clustering on write only happens above a size threshold (for example 64 MB with one key, 1 GB with four). Automatic key selection needs runtime 15.4 LTS or later and UC managed tables. | Databricks: liquid clustering |
| Serverless compute | Only Spark Connect APIs: no RDD APIs, no sparkContext, no global temporary views, no compute policies. Maximum run time 7 days. Streaming triggers such as continuous and processing-time are not supported, so use available-now. Sources differ on Scala and R support, so check the current page. | Databricks: serverless limitations |
| Declarative pipelines and Auto CDC | Identity columns are not supported on Auto CDC targets. A dataset can be defined only once across pipelines (append flows into a streaming table are the exception). pivot() is not supported. A workspace allows 1,000 concurrent pipeline updates. | Databricks: pipeline limitations |
| Lineage | The lineage system tables keep a rolling one-year window and hold only a subset of events, recorded when lineage can be inferred. Column lineage needs both source and target referenced by table name. Only events from UC-governed work (pipelines, notebooks, jobs, SQL, dashboards) appear, so mixing governed and ungoverned compute leaves gaps. | Databricks: lineage system tables |
| Metric views | The main docs say metric views do not support lineage, Delta Sharing or Lakehouse Monitoring. In Power BI, the BI compatibility mode needs DirectQuery, a recent Desktop build, and cannot join a metric view to other tables. One Microsoft page says that mode was removed from the connector while a Databricks page calls it Beta, so confirm the status. | Databricks: Power BI and metric views, Microsoft: BI metric view |
AWS side
| Area | Limitation | Source |
|---|---|---|
| S3 Tables cost | You pay for storage, requests, an object monitoring fee and optional compaction (per object and per GB processed). Compaction prices were cut by up to 90 percent from July 2025. Sort and z-order compaction can cost more than binpack. Search results disagreed by tenfold on the monitoring rate, so read the pricing page. | AWS: maintenance for tables, AWS: compaction cost cut |
| S3 Tables compaction | Target file size must be between 64 MB and 512 MB. Snapshot management is configured per table. | AWS: maintenance for tables |
| S3 Tables with Glue and Lake Formation | Column names are converted to lowercase. The SearchTables API cannot search S3 tables. Access errors in Athena are usually fixed with Lake Formation grants on the table-bucket catalog and database. | AWS: S3 tables catalog integration limitations |
| Lake Formation with EMR (fine-grained access) | Spark jobs only, and Hive or Iceberg tables only. A single Spark session per job. Dynamic resource allocation cannot be turned off. No custom images on EMR Serverless. Iceberg works only with the session catalog. Registered Iceberg tables expose only some metadata tables and hide sensitive columns. Write support differs by release: older pages say data-changing statements are limited, a newer page says EMR 7.12 and later supports DML and DDL through Lake Formation credentials. RDD and nested-column access control are unsupported in the version I found. | AWS: EMR Serverless and Lake Formation, AWS: EMR and Lake Formation considerations |
| Glue materialized views for Iceberg | Need Glue 5.1 or later and Spark 3.5.6 or later. Source tables must be Iceberg tables in the Glue Data Catalog. Results are Iceberg tables in S3 Tables or general-purpose buckets. I did not find the full limits list, so read the Glue page. | AWS Glue: materialized views |
| Power BI | One community answer says Power BI has no native Iceberg connector, with OneLake virtualization as the workaround (the thread's date is unclear). The Athena connector lists DirectQuery as supported, and users report failures after publishing or with native queries. Those are user reports, not official limits. | Microsoft: Athena connector |
What this changes in the comparison. Four things matter most for the decision. (1) Managed Iceberg tables in UC are tied to Predictive Optimization, so "Iceberg on Databricks" is not independent of the managed stack. (2) UniForm stays read-only, so a multi-writer design must be Iceberg-native. (3) Fine-grained Lake Formation on EMR works but with a long list of conditions, so test your exact EMR release and mode. (4) Power BI on metric views is the least settled part of the Databricks story, so pilot it before promising it.
What this list cannot show. Support tickets and escalations raised by real customers are private to those customers, so neither vendor publishes them. The limits above come from public documentation and a few public user reports. Before you commit, ask each vendor's account team for known issues on your exact runtime and region, and pilot on your own data.
Verdict
Scorecard
| Dimension | Winner | Why |
|---|---|---|
| Openness | A | Native Iceberg, multi-writer |
| Governance depth | B | Lineage, tag-based policies, system tables |
| Cost per compute hour | A | Spot, no DBU premium |
| Engineering effort | B | Serverless, declarative, fewer parts |
| Table layout | B | Liquid clustering |
| Upkeep simplicity | Tie | S3 Tables and PO are both automatic |
| Power BI | B | Native connector, passes user identity |
| Semantic layer | B | Metric views |
| Declarative pipelines, Auto CDC, SCD2 | B | Less code |
| AWS integration | A | IAM, Athena, Redshift, zero-ETL |
| Lock-in risk | A | Open format and catalog |
| Cost visibility | Tie | Databricks per job, AWS for the whole bill |
The pattern. Databricks wins where the value is one connected product: lineage, governance, declarative pipelines, Power BI, metric views. AWS-native wins where the value is cost, control and openness: compute price, multi-engine Iceberg, AWS integrations, no second vendor. The decision depends less on features and more on whether you have a platform team to do the assembly work.
| If this describes you | Choose |
|---|---|
| Power BI is the main consumer, the team is small, you want fast governed delivery | B: Databricks |
| Big steady ETL, a strong AWS platform team, several engines reading and writing | A: AWS-native |
| Both needs | Hybrid: Databricks reads S3 Tables through catalog federation, and UniForm or managed Iceberg lets Athena or Redshift read Databricks tables |
FAQ
Questions people ask
Does Databricks or AWS-native cost less?
Per compute hour, AWS-native on spot and Graviton is usually cheaper. Databricks adds a DBU premium but can save engineering time and run faster with Photon. Measure the same pipeline on both and compare monthly cost, including storage requests, network and idle time.
Can Athena or Redshift read Databricks tables?
Yes, through UniForm, which exposes Delta tables as Iceberg for reading. It is read-only, needs column mapping, and has limits such as deletion vectors on Iceberg v2 readers. Check the current UniForm page.
Which is better for Power BI?
Databricks, because Power BI has a native connector with DirectQuery and passes user identity. Iceberg on S3 has no native Power BI connector, so you go through Athena, Redshift or a Trino service. The metric view path in Power BI is the least settled part, so pilot it.
Is compression a reason to choose one?
No. Both write Parquet and you can choose the same codec. File size and table layout matter far more.
Next
Where to go next
Related on this site: Iceberg lakehouse: who owns the catalog? · Databricks compute and cost · Delta vs Iceberg · Who maintains your tables · SCD2: AUTO CDC or own PySpark? · Power BI vs Databricks AI/BI · Metric views · AWS data services catalog