Home › AWS › AWS data services › S3 Tables
AWS data services · Storage and table formats · about 11 min read
Amazon S3 Tables
A special S3 bucket type that stores Apache Iceberg tables, holds their catalog pointer and runs compaction and clean-up for you.
In plain words
What it is
A table in a data lake is not one file. It is many Parquet data files plus a set of metadata files that say which data files make up the current version of the table. Apache Iceberg is one open format for that metadata. Someone has to keep a pointer to the newest metadata file (that is the job of a catalog), and someone has to tidy up: merge small files into bigger ones, drop old versions and delete files nothing points to any more. On plain S3 you assemble all of that yourself.
Amazon S3 Tables packages those three jobs into one AWS service. You create a table bucket, which is a different kind of bucket from the general purpose buckets most people know. Inside it you create namespaces (folders for tables, like schemas in a database) and tables. Every table is an Iceberg table. S3 keeps the catalog pointer, and it runs compaction, snapshot management and unreferenced file removal in the background, switched on by default.
Picture a retailer with no Databricks. Orders from the web shop arrive through Amazon Data Firehose every few minutes and land as a table called sales.orders in a table bucket. Analysts query it from Athena, a nightly Glue job adds returns, and Redshift reads it for finance reports. Nobody writes a compaction job: the thousands of small files Firehose leaves behind are merged in the background, and versions older than the retention window disappear on their own.
Because the tables live inside S3, they keep S3 durability and scale, but they are reached through their own API (the s3tables namespace), through the AWS Glue Data Catalog integration, or through an Iceberg REST endpoint that open-source engines can call.
A filing cabinet that comes with its own index card box and a night cleaner who staples loose pages together and shreds drafts nobody needs.
Big picture
Where it fits
The lit lane is where S3 Tables works. Boxes on the left feed it; boxes on the right use what it produces.
S3 Tables sits in the store lane, next to general purpose S3 buckets. On the site's reference platform, the medallion tables (bronze, silver, gold) are Unity Catalog managed tables in ordinary S3 buckets, one bucket and one KMS key per catalog. Databricks owns them and Predictive Optimization does the chores. S3 Tables is not part of that main path.
It shows up in two situations. First, an AWS-only team, or an older part of the estate, already keeps Iceberg tables in a table bucket, fed by Firehose, Glue or EMR and read by Athena or Redshift. Databricks then reads those tables in place through Unity Catalog catalog federation, as foreign tables another catalog owns. Copying them is the wrong answer, because it doubles storage and creates two truths.
Second, a team without Databricks needs a lakehouse of its own. There S3 Tables is a good default, because it gives the AWS engines a managed table with maintenance built in. The site's table maintenance page puts it simply: whoever holds the pointer owns the table, and the owner holds the broom. Pick one owner per table and let every other engine be a guest.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Table bucket | A bucket type made only for tables. Block Public Access is always on and cannot be turned off. It is separate from general purpose and directory buckets. |
| Namespace | A logical folder inside a table bucket that groups tables, much like a schema in a database. |
| Apache Iceberg | The open table format every S3 table uses. Data stays in Parquet files; Iceberg metadata files describe each version of the table. |
| Snapshot | One version of an Iceberg table. Every write makes a new snapshot; old snapshots allow time travel until they are expired. |
| Compaction | Rewriting many small data files into fewer large ones so queries open fewer files. S3 Tables runs it automatically per table. |
| Unreferenced file removal | Deleting files that no snapshot points to, such as leftovers from failed writes. Set at the table bucket level. |
| Glue Data Catalog integration | A switch that publishes table buckets into the Glue Data Catalog so Athena, Redshift, EMR and Quick can find the tables, with Lake Formation for permissions. |
| Iceberg REST endpoint | A standard Iceberg catalog API that S3 Tables exposes, so Spark, Trino, PyIceberg and others can reach tables without AWS-specific plugins. |
| Catalog federation | How Unity Catalog shows tables owned by another catalog, such as Glue or S3 Tables, as foreign tables it can read and govern without copying. |
Choosing
When to use it, and when not
Use it when
- You run an AWS-only lakehouse with Athena, Glue, EMR and Redshift, and want Iceberg tables whose upkeep AWS runs for you.
- Firehose or a streaming job writes many small files into an Iceberg table and nobody wants to own a compaction schedule.
- Another team already owns Iceberg tables in a table bucket and your Databricks platform only needs to read them through catalog federation.
- You want access rules per table or per namespace in IAM, using the separate s3tables actions, instead of prefix rules in a bucket policy.
- A small analytics team without a platform team needs managed tables quickly and is happy to stay inside AWS engines.
Avoid it when
- Databricks is your main engine. Unity Catalog managed tables already keep open files in your bucket, and Predictive Optimization, row filters, column masks and lineage need tables Unity Catalog owns.
- You want Delta Lake. S3 Tables stores only Iceberg, so Delta pipelines, change data feed and Lakeflow streaming tables do not fit.
- You plan to write the same table from Databricks and from AWS engines. Two writers through two catalogs means two permission systems and clean-up jobs that can delete files the other still needs.
- You need every file to sit at a path you choose in a general purpose bucket, for example for an existing replication or Object Lock setup built on prefixes.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Two owners for one table | Databricks writes a table through the open-source Iceberg library while S3 Tables also maintains it. Unity Catalog governance is bypassed, and lineage and audit miss half the writes. | Give every table one owner. On this platform, Databricks-written tables are Unity Catalog managed tables; S3 Tables are read through federation. |
| Permissions in two places | Lake Formation grants on the Glue side and Unity Catalog grants on the Databricks side drift apart, so a user is blocked in one engine and allowed in the other. | Treat federated S3 Tables as read-only in Databricks, review both grant sets together and record who owns each table in the metadata store. |
| Federation writes are limited | Teams expect Databricks to write into foreign Iceberg tables as it writes its own, then find the feature limited or in preview. | Plan for reads only and check the current Databricks catalog federation docs before promising writes. |
| Retention set by default | Snapshot management expires old versions on default settings, then an auditor asks for the table as it was 30 days ago. | Set snapshot retention per table on purpose, match it to time-travel needs and write the setting down. |
| Maintenance bill surprises | Compaction is billed per object and per byte processed. A table rewritten every few minutes by a streaming job can cost more in compaction than in storage. | Watch maintenance charges in the Cost and Usage Report per table bucket; batch writes into larger commits where you can. |
| Copying tables for a second engine | A nightly job copies Delta gold tables into S3 Tables so Athena can read them: double storage, a lag and two answers. | Let Athena read Unity Catalog tables as Iceberg through UniForm and the Iceberg REST catalog instead of copying. |
| Quota surprises in multi-tenant designs | A design with one table bucket per team or per plant hits the default of 100 table buckets per Region. | Use namespaces for separation inside fewer buckets, or request a quota increase early. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick S3 Tables when |
|---|---|---|
| Unity Catalog managed tables (Delta or managed Iceberg) | Databricks is the main engine. You want Predictive Optimization, masks, row filters, lineage and one permission model. This is the site's default. | No Databricks at all; then S3 Tables is the simpler choice. |
| Glue Data Catalog Iceberg tables with table optimizers | You want Iceberg in your own general purpose bucket, at paths you choose, with AWS running compaction and retention as Glue compute. | You prefer one service that bundles storage, catalog and maintenance with less setup. |
| Iceberg on plain S3 with your own jobs | You need full control over file layout and maintenance timing, or you run engines that the AWS catalog cannot serve. | Nobody has time to own compaction, snapshot expiry and orphan clean-up. |
| Azure: Iceberg or Delta on ADLS with Fabric OneLake | You are on Azure; OneLake keeps Delta and serves Iceberg readers. | Your platform is on AWS. |
| Google Cloud: BigLake Iceberg tables | You are on Google Cloud and want BigQuery to manage Iceberg tables in Cloud Storage. | Your platform is on AWS. |
Cost
How you pay
You pay for table storage per GB-month, as with S3 but at a table bucket rate; for requests against the tables; for a small monitoring fee per object; and for maintenance, which is billed per object processed and per GB processed by compaction. Intelligent-Tiering can now move table data that is not read to cheaper access tiers. Query engines such as Athena, Redshift or EMR bill separately for reading. Check the S3 pricing page for your Region; rates differ from general purpose buckets.
What drives the bill
- How much data the tables hold after compaction and snapshot expiry.
- How often the table is written: many small commits mean more compaction work.
- How long snapshots are kept, because old versions keep old files alive.
- Number of objects, which drives the monitoring fee and per-object compaction charges.
Cost traps
- Streaming writers making a commit every few seconds, so compaction rewrites the same data again and again.
- Long snapshot retention left on by default for tables that nobody time-travels.
- Copying tables from Databricks into S3 Tables, which pays for storage twice plus a copy job.
- Forgetting that replication of tables to another Region bills storage and requests in the destination too.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Access uses its own IAM namespace, s3tables, so policies can grant per table, per namespace or per table bucket, and Service Control Policies can block whole actions across accounts.
- Block Public Access is always on for table buckets and cannot be switched off.
- Encrypt with SSE-KMS using a customer managed key per table bucket where the data is sensitive, in line with the site's one key per catalog rule.
- Reach S3 Tables privately from your VPC with AWS PrivateLink interface endpoints, so traffic never needs a public route.
- Record management and data access in AWS CloudTrail. When tables are published to the Glue Data Catalog, Lake Formation grants and audit apply on top.
- Personal data should be tokenised before it reaches any table, as on the rest of the platform; S3 Tables does not mask columns for Databricks readers.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| Maintenance job status | Whether compaction, snapshot management and unreferenced file removal last ran and succeeded, per table, from the S3 Tables maintenance status API or console. |
| Files per table and average file size | Read from the Iceberg metadata tables; many small files mean compaction is behind. |
| Snapshot count and oldest snapshot | Shows whether retention is doing its job and whether time travel reaches as far back as promised. |
| CloudTrail s3tables events | Who created, changed or deleted tables and maintenance settings. |
| Cost and Usage Report lines for S3 Tables | Storage, request, monitoring and compaction charges per table bucket, to spot a runaway table. |
| Federated reads from Databricks | Unity Catalog audit system tables for queries against the foreign catalog, to see who uses the tables and how often. |
| Limit | Value and notes |
|---|---|
| Table buckets | 100 per AWS Region per account by default; increase through Support |
| Namespaces | 10,000 per table bucket by default; adjustable |
| Tables | 10,000 per table bucket by default; adjustable |
| Table format | Apache Iceberg only, including Iceberg v3 features such as deletion vectors and row lineage; no Delta Lake |
| Table migration | Up to 1 billion source objects and 100,000 partitions per migration; 10 concurrent migrations per account |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Databricks platform writing bronze, silver and gold for plant sensor data and SAP orders | Unity Catalog managed tables in your own S3 buckets, Iceberg reads through UniForm for Athena | Drop S3 Tables here: Databricks must own the tables it writes so Predictive Optimization, masks and lineage work. |
| A retail analytics team already keeps web orders in a table bucket, and the Databricks platform needs them for a margin report | Read the table in place through Unity Catalog catalog federation | Keep if you have it: no copy, one owner, and Databricks stays a guest that reads. |
| AWS-only platform: Firehose, Glue, Athena and Redshift, no Databricks | S3 Tables with the Glue Data Catalog integration | Pursue: managed Iceberg with maintenance built in is the least work on that stack. |
| Proposal to move Databricks gold tables into S3 Tables so 'AWS maintains Iceberg and Databricks only computes' | Keep Unity Catalog as owner | Drop: two owners, two permission systems, limited writes and no Predictive Optimization. |
FAQ
Questions people ask
Is S3 Tables the same as Iceberg on S3?
It is Iceberg on S3 with the catalog and the maintenance included. You can build the same thing yourself with Iceberg files in a general purpose bucket, a catalog and scheduled jobs; S3 Tables runs those parts for you.
Can Databricks write to S3 Tables?
Databricks reads S3 Tables in place through Unity Catalog catalog federation. Writes to foreign Iceberg tables are limited and change often, so check the current Databricks docs. Writing with the open-source Iceberg library on a classic cluster works but bypasses Unity Catalog governance.
Does S3 Tables support Delta Lake?
No. Every table in a table bucket is an Apache Iceberg table. Delta tables stay in general purpose buckets, and Iceberg engines can read them through UniForm.
Should we move our Databricks tables to S3 Tables to stay open?
No. Unity Catalog managed tables are already open Parquet files in your own bucket, and other engines can read them as Iceberg. Moving them only hands ownership and chores to a second catalog.
How do Athena and Redshift find S3 Tables?
Switch on the integration with the AWS Glue Data Catalog. The tables then appear as a catalog that Athena, Redshift, EMR and Quick can query, with Lake Formation for permissions.
Sources
Checked against
- AWS docs: Working with Amazon S3 Tables and table buckets
- AWS docs: S3 Tables maintenance
- AWS docs: S3 Tables Regions, endpoints and quotas
- AWS docs: Lake Formation and S3 Tables integration
Checked 2026-11-11.
Related
Where it connects
Who maintains your tables · Delta vs Iceberg on Databricks · Data platform on AWS and Databricks · AWS Glue Data Catalog · Amazon S3 · Amazon Athena · All AWS data services