Home › Data engineering › Who maintains your tables

Data engineering · free guide · one page

Who maintains your tables? Managed, external and AWS-managed Iceberg

A question on almost every AWS project with Databricks: "Can we keep tables in Iceberg, let AWS do the maintenance, and use Databricks only for analytics, like a Snowflake virtual warehouse?" It mixes two questions. Are storage and compute separate? And who does the chores on each table? This page answers both, shows how Delta and Iceberg commit and clean up inside, and gives the setup we recommend.

Short answer: Databricks already separates storage and compute, with your data in your own S3 bucket. Who cleans a table depends on which catalog owns it, not on the file format. On a Databricks platform, keep tables as Unity Catalog managed tables, let Predictive Optimization do the chores, and let other engines read them as Iceberg. AWS-managed Iceberg is right for an AWS-only platform. Mixing the two, with AWS owning tables that Databricks mostly works on, is the setup to avoid.

Question one

Storage and compute are already separate

Snowflake made "separate storage and compute" famous: the data sits in one place, and you start as many virtual warehouses as you like to query it, paying only while they run. Databricks works the same way. A SQL warehouse is the direct match for a Snowflake virtual warehouse, and job and notebook clusters are more compute of the same kind. The difference is the storage: with Databricks it is your own S3 bucket, holding open Parquet files.

SnowflakeDatabricks on AWSwarehouse Swarehouse Mwarehouse LSQL warehousejob clusternotebookcompute: start, query, stopcompute: start, query, stopSnowflake's storageits own format, its accountyour S3 bucketopen Parquet, your account

Both split storage from compute. Databricks' storage is your bucket, in open files.

So you don't need AWS to run maintenance to get the split. You already have it. The real question is the second one.

Question two

Every table has chores

An open table gets messier with every write. Small writes leave many small files. Updates and deletes never edit a file; they write new files and mark the old ones as removed, and the old files stay on disk so you can read the table as it was last week. Someone has to tidy up. The two formats name the chores differently, but they are the same four jobs.

the chore chart every open table needs1 Compactionmerge small filesDelta: OPTIMIZEIceberg: rewrite data files2 Old versionsdelete files no kept version needsDelta: VACUUM (7 days kept)Iceberg: expire snapshots3 Orphansfiles a crashed write left behindDelta: VACUUM covers itIceberg: remove orphan files4 Metadatakeep the contents page smallDelta: checkpointsIceberg: rewrite manifests

Skip the chores and queries slow down while storage grows. Do them too eagerly and time travel breaks.

The answer

Who does the chores depends on who owns the table

Unity Catalog managedDatabricksUC picks the folder in your bucketPredictive Optimization, automaticUnity Catalog externalYouyou pick the path; UC registers ityour scheduled OPTIMIZE / VACUUM jobsAWS-managed IcebergAWSGlue catalog or an S3 table bucketGlue optimizers or S3 Tables, automatic

Three kinds of table you meet on AWS with Databricks, and who holds the broom for each.

Kind of tableWho owns the files and catalogWho does the choresWhat DROP doesBest for
Unity Catalog managed (Delta or managed Iceberg)Unity Catalog picks the folder, inside a bucket you own and set as the catalog's or schema's managed locationDatabricks. Predictive Optimization runs OPTIMIZE, VACUUM and ANALYZE when a table needs it, on serverless compute billed to youData can be brought back with UNDROP for 7 days, then is deletedAlmost every table on a Databricks platform
Unity Catalog externalYou choose the path; Unity Catalog only registers it and controls accessYou. Predictive Optimization does not run on external tables, so schedule OPTIMIZE and VACUUM yourselfRemoves the registration; files stay in placeFiles written by another system, such as a vendor drop zone
Glue Data Catalog IcebergAWS Glue holds the pointer to the current metadata file; files in your S3 bucketAWS, if you switch on Glue's table optimizers: compaction, snapshot retention, orphan file deletion. Billed as Glue computeDepends on the engine and settings; check before droppingAWS-only platforms using Glue, EMR and Athena
S3 Tables (table buckets)A special S3 bucket that holds Iceberg tables and has its own catalogAWS, built in and on by default: compaction, snapshot management, unreferenced file removal. Billed per object and byte processedDeletes the tableAWS-only platforms that want the AWS twin of a managed table

Managed versus external, in one line

Both keep open files in your bucket. Managed means Unity Catalog runs the table's whole life, chores included. External means you keep the path and the chores.

Predictive Optimization

Databricks' automatic upkeep for managed tables. It watches how each table is used and runs compaction, VACUUM and statistics only when they pay off. It is on by default for newer accounts and can be set per account, catalog or schema.

Under the hood

How Delta and Iceberg work inside

Both formats are the same Parquet files with a contents page on top. Delta keeps the contents page as an ordered diary of changes next to the data. Iceberg keeps it as a tree of metadata files, and a catalog holds the pointer to the top of the tree.

Delta LakeApache Iceberg Catalog (Unity Catalog) _delta_log/ 00000.json 00001.json 00002.json … one file per commit, in order; a checkpoint every so often the newest log file is the truth Catalog (REST, UC, Glue) metadata.json: schema, snapshotsmanifest list (one snapshot) manifests: which data files, with column stats Parquet data files (+ deletion vectors)the same kind of files in both formats Parquet data files (+ delete files or deletion vectors)the catalog's pointer to metadata.json is the truth

Same Parquet files, two different contents pages. For Iceberg, the catalog's pointer decides which version is current.

Delta LakeApache Iceberg
Where the contents page livesA _delta_log folder beside the data: one numbered JSON file per commitA tree of metadata files; a catalog holds the pointer to the top one
How a commit becomes realThe writer creates the next numbered file. Only one writer can create it; the loser retries. On S3 this uses conditional writes, or Unity Catalog coordinates commitsThe catalog swaps its pointer from the old metadata file to the new one, only if nobody swapped it first
How readers find the current tableStart from the newest checkpoint (written every 10 commits) and replay the newer JSON filesAsk the catalog for the pointer, then follow metadata, manifest list, manifests, data files
What an update doesWrites new files and marks old ones removed in the log (or marks deleted rows with deletion vectors)Writes new files, or delete files, and a new snapshot that lists them
How old files go awayVACUUM deletes removed files older than the retention (7 days by default)Expire snapshots drops old versions and the files only they used; remove orphan files deletes leftovers
What breaks if you clean too earlyTime travel to older versions, and long queries still reading old files, fail with file-not-foundSame: readers of an expired snapshot fail

For the full feature comparison on Databricks, see Delta Lake vs Apache Iceberg. To watch a Delta table commit, VACUUM and break time travel step by step, see Delta Lake internals, recorded.

Why it matters

Whoever holds the pointer owns the table

In Iceberg, a write only becomes real when the catalog swaps its pointer to the new metadata file. Delta is the same in spirit: whoever owns the log folder owns the commits. So whoever holds the pointer controls writes, permissions and clean-up. Every other engine is a guest.

the catalogholds one pointer: "current = metadata v42"metadata v42Parquet files in your bucketthe ownercommits through this catalogwrites, permissions, choresswaps ita guest enginereaches it by federationreads well, writes littlereads itwhoever holds the pointer owns the table, whatever the file format

One owner commits and cleans. Guests read through federation.

The AWS wrapper

What AWS adds around Iceberg

AWS partWhat it does
Glue Data CatalogA catalog. For Iceberg it stores the location of each table's current metadata file. Lake Formation adds permissions on top.
Glue table optimizersManaged jobs you switch on per table or catalog: compaction, snapshot retention and orphan file deletion, run on AWS compute.
S3 TablesA table bucket: S3 storage plus an Iceberg catalog plus maintenance in one service, reached through its own API, the Glue catalog integration or an Iceberg REST endpoint.
Athena, EMR, Redshift, Glue jobsThe AWS engines that read and write those tables through the Glue catalog.

With Glue or S3 Tables holding the pointer, AWS is the owner and does the chores. Databricks then reaches those tables through catalog federation: Unity Catalog shows them as foreign tables that another catalog owns. That works for reading. As the main engine, it costs you:

  • Writes from Databricks to foreign Iceberg tables are limited; reads work well. This changes often, so check the Databricks docs before relying on writes.
  • Predictive Optimization, row filters, column masks and features built on tables (metric views, Lakeflow pipelines) need tables Unity Catalog owns.
  • Unity Catalog lineage only sees work done on Databricks; AWS jobs that write the table are invisible to it.
  • Permissions live in two places, Lake Formation and Unity Catalog, and must be kept in step by hand.
  • Upkeep is billed twice over: AWS maintenance on one bill, Databricks compute on the other, watched by two teams.
  • The workaround, writing Glue Iceberg tables from Databricks with the open-source Iceberg library on a classic cluster, bypasses Unity Catalog governance entirely.

Choosing

What we recommend

Databricks jobsSQL warehouseUnity Catalog managed tablesDelta with Iceberg reads (UniForm),or managed Iceberg where others must writePredictive Optimizationyour S3 bucketone copy, open filesAthena / EMRTrinoSnowflakeread as Iceberg throughthe Iceberg REST catalogwrite and query

One copy, one owner doing the chores, many engines reading.

SituationSetupVerdict
Databricks is the main engine (this site's AWS default)Unity Catalog managed tables in your bucket, Predictive Optimization on, Iceberg reads through UniForm for the tables outside engines needPursue
Another engine must also write the same tableManaged Iceberg in Unity Catalog, written by others through its Iceberg REST catalog. See the limits on the Delta vs Iceberg pagePursue only where needed
Something outside Databricks drops the filesUnity Catalog external table, plus a scheduled OPTIMIZE and VACUUM jobPursue, with an owner for the chores
AWS-only platform (Firehose, Glue, EMR, Athena; no Databricks)S3 Tables, or Glue Iceberg with optimizers onPursue
Tables another team already owns in Glue or S3 TablesRead them in place through catalog federation; don't copyPursue for reading
AWS owns and maintains the tables, Databricks does most of the work as a guestTwo owners, two permission systems, limited writes, no Predictive Optimization or masksDrop
External tables everywhere 'to stay open'Managed tables are already open files in your bucket; external only hands the chores back to youDrop

What goes wrong

Mistakes we see often

  1. Calling external tables more open. Managed tables are also Parquet in your bucket. External only means you took the chores back.
  2. Nobody owns the chores. External tables with no OPTIMIZE or VACUUM job, or Glue tables with optimizers off. Small files pile up quietly and queries slow down.
  3. Two catalogs claiming one path. Glue and Unity Catalog both register the same folder and both write. One day a clean-up job deletes files the other still needs.
  4. Retention set by accident. Version removal runs on the default, then an auditor asks for the table as it was 30 days ago.
  5. Copying tables for a second engine. A nightly job copies Delta into Iceberg: double storage, a lag, two truths. Let readers in through the catalog instead.

Your turn

Five questions to check your own setup

  1. For each important table, which catalog holds its pointer, and does exactly one catalog write to it?
  2. Is each table managed or external, and was that a choice made on purpose?
  3. Who runs compaction and version removal for each table, and when did they last run?
  4. Does history retention match how far back people need to travel in time?
  5. Can Athena or another engine read your tables today without a copy job?

FAQ

Questions people ask

Does AWS run VACUUM on Unity Catalog tables?

No. AWS has no say over tables Unity Catalog owns. For managed tables Databricks' Predictive Optimization does it; for external tables you schedule it yourself. AWS maintains only tables its own catalog owns: Glue Iceberg tables with optimizers on, and S3 Tables.

Is Databricks like a Snowflake virtual warehouse?

Yes, for compute. A Databricks SQL warehouse starts, queries and stops like a Snowflake warehouse, and storage is already separate. The difference is that the storage is your own S3 bucket in open formats, not Snowflake-managed storage.

Do I need Iceberg managed by AWS to keep storage and compute separate?

No. Unity Catalog managed tables already keep data in your bucket, separate from compute. Who maintains a table is a separate question from where it is stored.

Does Predictive Optimization work on external tables?

No. It runs only on Unity Catalog managed tables, Delta and managed Iceberg. External tables need your own OPTIMIZE and VACUUM jobs.

Can Databricks write to S3 Tables or Glue Iceberg tables?

It can read them in place through catalog federation. Writing to foreign Iceberg tables is limited, and support is changing; check the current Databricks docs. Writing with the open-source Iceberg library on a classic cluster works but skips Unity Catalog governance.

How do other engines read Unity Catalog managed tables?

Through Unity Catalog's Iceberg REST catalog. Delta tables need Iceberg reads (UniForm) switched on; managed Iceberg tables are readable, and writable, by Iceberg engines directly.

Sources

Checked against

AWS and Databricks change Iceberg support every few months. Check the docs below before relying on federation writes or preview features.

Related

Where it connects

Delta vs Iceberg on Databricks · Data platform on AWS + Databricks · Layout: catalogs and schemas · Cost and FinOps · All data engineering