Home › Data engineering › Who maintains your tables
Data engineering · free guide · one page
Who maintains your tables? Managed, external and AWS-managed Iceberg
A question on almost every AWS project with Databricks: "Can we keep tables in Iceberg, let AWS do the maintenance, and use Databricks only for analytics, like a Snowflake virtual warehouse?" It mixes two questions. Are storage and compute separate? And who does the chores on each table? This page answers both, shows how Delta and Iceberg commit and clean up inside, and gives the setup we recommend.
Question one
Storage and compute are already separate
Snowflake made "separate storage and compute" famous: the data sits in one place, and you start as many virtual warehouses as you like to query it, paying only while they run. Databricks works the same way. A SQL warehouse is the direct match for a Snowflake virtual warehouse, and job and notebook clusters are more compute of the same kind. The difference is the storage: with Databricks it is your own S3 bucket, holding open Parquet files.
Both split storage from compute. Databricks' storage is your bucket, in open files.
So you don't need AWS to run maintenance to get the split. You already have it. The real question is the second one.
Question two
Every table has chores
An open table gets messier with every write. Small writes leave many small files. Updates and deletes never edit a file; they write new files and mark the old ones as removed, and the old files stay on disk so you can read the table as it was last week. Someone has to tidy up. The two formats name the chores differently, but they are the same four jobs.
Skip the chores and queries slow down while storage grows. Do them too eagerly and time travel breaks.
The answer
Who does the chores depends on who owns the table
Three kinds of table you meet on AWS with Databricks, and who holds the broom for each.
| Kind of table | Who owns the files and catalog | Who does the chores | What DROP does | Best for |
|---|---|---|---|---|
| Unity Catalog managed (Delta or managed Iceberg) | Unity Catalog picks the folder, inside a bucket you own and set as the catalog's or schema's managed location | Databricks. Predictive Optimization runs OPTIMIZE, VACUUM and ANALYZE when a table needs it, on serverless compute billed to you | Data can be brought back with UNDROP for 7 days, then is deleted | Almost every table on a Databricks platform |
| Unity Catalog external | You choose the path; Unity Catalog only registers it and controls access | You. Predictive Optimization does not run on external tables, so schedule OPTIMIZE and VACUUM yourself | Removes the registration; files stay in place | Files written by another system, such as a vendor drop zone |
| Glue Data Catalog Iceberg | AWS Glue holds the pointer to the current metadata file; files in your S3 bucket | AWS, if you switch on Glue's table optimizers: compaction, snapshot retention, orphan file deletion. Billed as Glue compute | Depends on the engine and settings; check before dropping | AWS-only platforms using Glue, EMR and Athena |
| S3 Tables (table buckets) | A special S3 bucket that holds Iceberg tables and has its own catalog | AWS, built in and on by default: compaction, snapshot management, unreferenced file removal. Billed per object and byte processed | Deletes the table | AWS-only platforms that want the AWS twin of a managed table |
Managed versus external, in one line
Both keep open files in your bucket. Managed means Unity Catalog runs the table's whole life, chores included. External means you keep the path and the chores.
Predictive Optimization
Databricks' automatic upkeep for managed tables. It watches how each table is used and runs compaction, VACUUM and statistics only when they pay off. It is on by default for newer accounts and can be set per account, catalog or schema.
Under the hood
How Delta and Iceberg work inside
Both formats are the same Parquet files with a contents page on top. Delta keeps the contents page as an ordered diary of changes next to the data. Iceberg keeps it as a tree of metadata files, and a catalog holds the pointer to the top of the tree.
Same Parquet files, two different contents pages. For Iceberg, the catalog's pointer decides which version is current.
| Delta Lake | Apache Iceberg | |
|---|---|---|
| Where the contents page lives | A _delta_log folder beside the data: one numbered JSON file per commit | A tree of metadata files; a catalog holds the pointer to the top one |
| How a commit becomes real | The writer creates the next numbered file. Only one writer can create it; the loser retries. On S3 this uses conditional writes, or Unity Catalog coordinates commits | The catalog swaps its pointer from the old metadata file to the new one, only if nobody swapped it first |
| How readers find the current table | Start from the newest checkpoint (written every 10 commits) and replay the newer JSON files | Ask the catalog for the pointer, then follow metadata, manifest list, manifests, data files |
| What an update does | Writes new files and marks old ones removed in the log (or marks deleted rows with deletion vectors) | Writes new files, or delete files, and a new snapshot that lists them |
| How old files go away | VACUUM deletes removed files older than the retention (7 days by default) | Expire snapshots drops old versions and the files only they used; remove orphan files deletes leftovers |
| What breaks if you clean too early | Time travel to older versions, and long queries still reading old files, fail with file-not-found | Same: readers of an expired snapshot fail |
For the full feature comparison on Databricks, see Delta Lake vs Apache Iceberg. To watch a Delta table commit, VACUUM and break time travel step by step, see Delta Lake internals, recorded.
Why it matters
Whoever holds the pointer owns the table
In Iceberg, a write only becomes real when the catalog swaps its pointer to the new metadata file. Delta is the same in spirit: whoever owns the log folder owns the commits. So whoever holds the pointer controls writes, permissions and clean-up. Every other engine is a guest.
One owner commits and cleans. Guests read through federation.
The AWS wrapper
What AWS adds around Iceberg
| AWS part | What it does |
|---|---|
| Glue Data Catalog | A catalog. For Iceberg it stores the location of each table's current metadata file. Lake Formation adds permissions on top. |
| Glue table optimizers | Managed jobs you switch on per table or catalog: compaction, snapshot retention and orphan file deletion, run on AWS compute. |
| S3 Tables | A table bucket: S3 storage plus an Iceberg catalog plus maintenance in one service, reached through its own API, the Glue catalog integration or an Iceberg REST endpoint. |
| Athena, EMR, Redshift, Glue jobs | The AWS engines that read and write those tables through the Glue catalog. |
With Glue or S3 Tables holding the pointer, AWS is the owner and does the chores. Databricks then reaches those tables through catalog federation: Unity Catalog shows them as foreign tables that another catalog owns. That works for reading. As the main engine, it costs you:
- Writes from Databricks to foreign Iceberg tables are limited; reads work well. This changes often, so check the Databricks docs before relying on writes.
- Predictive Optimization, row filters, column masks and features built on tables (metric views, Lakeflow pipelines) need tables Unity Catalog owns.
- Unity Catalog lineage only sees work done on Databricks; AWS jobs that write the table are invisible to it.
- Permissions live in two places, Lake Formation and Unity Catalog, and must be kept in step by hand.
- Upkeep is billed twice over: AWS maintenance on one bill, Databricks compute on the other, watched by two teams.
- The workaround, writing Glue Iceberg tables from Databricks with the open-source Iceberg library on a classic cluster, bypasses Unity Catalog governance entirely.
Choosing
What we recommend
One copy, one owner doing the chores, many engines reading.
| Situation | Setup | Verdict |
|---|---|---|
| Databricks is the main engine (this site's AWS default) | Unity Catalog managed tables in your bucket, Predictive Optimization on, Iceberg reads through UniForm for the tables outside engines need | Pursue |
| Another engine must also write the same table | Managed Iceberg in Unity Catalog, written by others through its Iceberg REST catalog. See the limits on the Delta vs Iceberg page | Pursue only where needed |
| Something outside Databricks drops the files | Unity Catalog external table, plus a scheduled OPTIMIZE and VACUUM job | Pursue, with an owner for the chores |
| AWS-only platform (Firehose, Glue, EMR, Athena; no Databricks) | S3 Tables, or Glue Iceberg with optimizers on | Pursue |
| Tables another team already owns in Glue or S3 Tables | Read them in place through catalog federation; don't copy | Pursue for reading |
| AWS owns and maintains the tables, Databricks does most of the work as a guest | Two owners, two permission systems, limited writes, no Predictive Optimization or masks | Drop |
| External tables everywhere 'to stay open' | Managed tables are already open files in your bucket; external only hands the chores back to you | Drop |
What goes wrong
Mistakes we see often
- Calling external tables more open. Managed tables are also Parquet in your bucket. External only means you took the chores back.
- Nobody owns the chores. External tables with no OPTIMIZE or VACUUM job, or Glue tables with optimizers off. Small files pile up quietly and queries slow down.
- Two catalogs claiming one path. Glue and Unity Catalog both register the same folder and both write. One day a clean-up job deletes files the other still needs.
- Retention set by accident. Version removal runs on the default, then an auditor asks for the table as it was 30 days ago.
- Copying tables for a second engine. A nightly job copies Delta into Iceberg: double storage, a lag, two truths. Let readers in through the catalog instead.
Your turn
Five questions to check your own setup
- For each important table, which catalog holds its pointer, and does exactly one catalog write to it?
- Is each table managed or external, and was that a choice made on purpose?
- Who runs compaction and version removal for each table, and when did they last run?
- Does history retention match how far back people need to travel in time?
- Can Athena or another engine read your tables today without a copy job?
FAQ
Questions people ask
Does AWS run VACUUM on Unity Catalog tables?
No. AWS has no say over tables Unity Catalog owns. For managed tables Databricks' Predictive Optimization does it; for external tables you schedule it yourself. AWS maintains only tables its own catalog owns: Glue Iceberg tables with optimizers on, and S3 Tables.
Is Databricks like a Snowflake virtual warehouse?
Yes, for compute. A Databricks SQL warehouse starts, queries and stops like a Snowflake warehouse, and storage is already separate. The difference is that the storage is your own S3 bucket in open formats, not Snowflake-managed storage.
Do I need Iceberg managed by AWS to keep storage and compute separate?
No. Unity Catalog managed tables already keep data in your bucket, separate from compute. Who maintains a table is a separate question from where it is stored.
Does Predictive Optimization work on external tables?
No. It runs only on Unity Catalog managed tables, Delta and managed Iceberg. External tables need your own OPTIMIZE and VACUUM jobs.
Can Databricks write to S3 Tables or Glue Iceberg tables?
It can read them in place through catalog federation. Writing to foreign Iceberg tables is limited, and support is changing; check the current Databricks docs. Writing with the open-source Iceberg library on a classic cluster works but skips Unity Catalog governance.
How do other engines read Unity Catalog managed tables?
Through Unity Catalog's Iceberg REST catalog. Delta tables need Iceberg reads (UniForm) switched on; managed Iceberg tables are readable, and writable, by Iceberg engines directly.
Sources
Checked against
AWS and Databricks change Iceberg support every few months. Check the docs below before relying on federation writes or preview features.
Related
Where it connects
Delta vs Iceberg on Databricks · Data platform on AWS + Databricks · Layout: catalogs and schemas · Cost and FinOps · All data engineering