Home › AWS › AWS data services › Zero-ETL
AWS data services · Ingestion and CDC · about 10 min read
Zero-ETL integrations (Aurora, RDS, DynamoDB into Redshift and SageMaker lakehouse)
AWS-managed replication from Aurora, RDS, DynamoDB and some SaaS apps into Redshift or the SageMaker lakehouse, with no pipeline to build.
In plain words
What it is
ETL stands for extract, transform and load: the pipeline a team writes to copy data from an application database into an analytics store. Zero-ETL is the AWS name for a set of integrations where AWS runs that copy for you. You pick a source, such as an Aurora database, and a target, such as a Redshift data warehouse, and create an integration. AWS seeds the target with the existing rows and then keeps applying changes, usually within seconds to a minute or so, with no job, schedule or code on your side.
A retail example: the online shop's orders live in Aurora PostgreSQL. Finance already uses Redshift for its dashboards and wants today's orders there, not last night's. With a zero-ETL integration the orders, order lines and customers tables appear in a Redshift database and stay current. Finance queries them next to the history already in Redshift. Nobody wrote a pipeline.
The family has grown beyond Aurora. Today the documented sources into Redshift are Aurora MySQL, Aurora PostgreSQL, RDS for MySQL, RDS for PostgreSQL, RDS for Oracle, Oracle Database@AWS, DynamoDB, several SaaS apps (Salesforce, SAP OData, ServiceNow, Zendesk and others, run through AWS Glue) and self-managed MySQL, PostgreSQL, SQL Server and Oracle (run through DMS). Some sources can also target the lakehouse architecture of Amazon SageMaker, which stores data as Apache Iceberg tables in S3, S3 Tables or Redshift managed storage.
A mirror bolted to the shop counter that always shows the warehouse office what is on the counter right now, but only the warehouse office can see it.
Big picture
Where it fits
The lit lane is where Zero-ETL works. Boxes on the left feed it; boxes on the right use what it produces.
Zero-ETL sits in the move-in lane, but it skips the rest of the reference platform's path. It does not write files into the S3 landing bucket, so Auto Loader, the lander's tokenising and the run log never see the data. It writes into Redshift or into the SageMaker lakehouse catalog, which are AWS-native targets.
On this platform Redshift stays for existing BI and reads gold tables through UniForm or Delta Sharing. If a team already uses Redshift and also needs near real-time rows from an Aurora or DynamoDB application, a zero-ETL integration into that Redshift is a reasonable shortcut. The site's real-time analytics page lists Aurora zero-ETL into Redshift as the near real-time AWS option.
For anything that must join silver and gold, be governed in Unity Catalog or be tokenised before analysts see it, the platform's answer remains DMS into S3 landing, or Lakeflow Connect where it supports the source. The lakehouse target writes Iceberg tables to an AWS Glue catalog, which Databricks can read as foreign tables through catalog federation, but that is a second governance surface to manage, so treat it as an exception with an owner.
How it works
The ideas you need
| Idea | What it means |
|---|---|
| Integration | The managed replication pipeline between one source and one target. You create, modify and delete it as one resource. |
| Source | The database or app the data comes from, such as an Aurora cluster, an RDS instance, a DynamoDB table or a SaaS app. |
| Target | Where data lands: a Redshift provisioned cluster or Serverless namespace, or a SageMaker lakehouse catalog in S3, S3 Tables or Redshift managed storage. |
| Destination database | The Redshift database you create from the integration; the replicated tables appear there and are read-only. |
| Data filter | Include and exclude patterns choosing which databases, schemas and tables are replicated. Aurora PostgreSQL requires at least one. |
| Seeding and resync | The initial copy of all rows, and the re-copy of one table after some schema changes. The table cannot be queried while it resyncs. |
| History mode | A Redshift option that keeps old versions of rows instead of overwriting them, giving a change history in the target. |
| Enhanced binlog | An Aurora MySQL setting that zero-ETL relies on to capture changes; it adds some I/O and storage cost on the source. |
Choosing
When to use it, and when not
Use it when
- Redshift already serves BI and a team needs current rows from an Aurora, RDS or DynamoDB application next to it.
- The tables need no tokenising or reshaping before analysts query them, and Redshift permissions are enough to govern them.
- You want near real-time copies without running DMS instances, Kafka connectors or jobs.
- A DynamoDB table needs SQL analytics and you want to avoid exporting it to S3 by hand.
- An ML team on SageMaker wants Iceberg copies of operational tables in the SageMaker lakehouse.
Avoid it when
- The data must enter the Databricks platform through S3 landing and the shared lander; zero-ETL does not write landing files.
- Columns hold personal data that must be tokenised before anyone can query them; the copy arrives raw in the target.
- The source schema changes often; DDL on a source table can trigger a resync that makes the table unavailable for a while.
- Tables lack primary keys; on Aurora PostgreSQL such tables go into a failed state.
- The source and target are in different Regions; integrations need both in the same Region.
Production grade
What goes wrong in production
| Problem | What breaks | What to do |
|---|---|---|
| Data skips the platform's checks | Rows land in Redshift with no tokenising, data contract or run log, so two copies of the truth appear with different rules. | Register each integration as a source with an owner; keep it for Redshift-only use cases; route platform data through DMS or Lakeflow Connect. |
| Table resyncs after DDL | An ALTER TABLE on the source can force a resync, and the table disappears from queries for minutes or longer on big tables. | Agree a change window with the app team; watch IntegrationNumTablesFailedReplication and table status in SVV_INTEGRATION_TABLE_STATE. |
| Failed tables | Tables with unsupported data types, no primary key, or cascading foreign keys on MySQL go to a failed state while the rest replicate. | Check the limits list before creating; alarm on failed tables; fix the source or exclude the table with a filter. |
| Blue/green and failover breaks the link | A blue/green switchover needs the integration deleted first; a global database failover makes it inactive. | Put integration delete and recreate steps into the release runbook; expect a fresh seed afterwards. |
| Source load and lag | Seeding a large database takes 20 to 25 minutes or more and can raise replica lag; busy periods raise IntegrationLag. | Create integrations off-peak; alarm on IntegrationLag; size the Redshift target so it can apply changes. |
| Hidden cost on both ends | The integration has no line item for Aurora, so teams assume it is free, then see more Aurora I/O, Redshift compute and storage. | Tag source and target; watch Redshift Serverless RPU use and Aurora I/O after enabling; filter to only the tables needed. |
| Governance split | The lakehouse target lives in the Glue Data Catalog and Lake Formation, not in Unity Catalog, so permissions are set twice. | Federate the Glue catalog into Unity Catalog read-only, or keep lakehouse targets for SageMaker-only users. |
Alternatives
What to compare it with
| Alternative | Pick the alternative when | Pick Zero-ETL when |
|---|---|---|
| AWS DMS into S3 landing | Data must pass the platform's lander, tokenising and run log before Databricks users see it. This is the site's default. | You only need current rows inside Redshift and want nothing to run. |
| Lakeflow Connect (Databricks) | You want changes straight into Unity Catalog tables with Databricks governance. | The consumer is Redshift and Databricks is not involved. |
| Redshift federated query | You need occasional live reads of Aurora or RDS from Redshift and can tolerate load on the source. | Queries are frequent or heavy; a replicated copy protects the source. |
| Fabric Mirroring (Azure) | The platform is on Azure; Mirroring copies databases into OneLake as Delta. | You are on AWS. |
| Datastream into BigQuery (Google Cloud) | The platform is on Google Cloud. | You are on AWS with Databricks. |
| Debezium on MSK Connect | You want an open, portable change stream that many readers can use. | Nobody wants to run Kafka and connectors for a single Redshift copy. |
Cost
How you pay
For Aurora and RDS sources AWS states that ongoing processing of changes by the integration has no additional charge. You still pay for what it uses: extra source I/O and storage for enhanced binlog or logical replication, snapshot export for the initial seed, storage of the copy in Redshift or S3, Redshift compute to apply changes, and cross-AZ transfer. Integrations run through AWS Glue (DynamoDB into the lakehouse and SaaS sources) and through DMS (self-managed sources) have their own usage charges on those services' pricing pages. Check current list prices for your region.
What drives the bill
- Redshift compute applying changes; on Redshift Serverless a busy integration keeps RPUs running.
- Size of the replicated copy in Redshift managed storage or S3.
- Source-side I/O and log retention needed for change capture.
- Data volume ingested through Glue-based integrations for DynamoDB and SaaS apps.
Cost traps
- Reading no additional charge as free; the Redshift and source costs are real and continuous.
- Replicating whole databases when two tables are needed; filters cut every cost line.
- Redshift Serverless never going idle because changes arrive all day, so the minimum base capacity bills around the clock.
- Re-seeding large tables after every DDL change, paying for snapshot export and compute again.
Prices change; check the AWS pricing page for your region before deciding.
Security and governance
Keeping it safe
- Encryption is set on the source, the integration and the target; use customer-managed KMS keys and give the integration only the key grants it needs.
- The target needs a resource policy that authorises the source as an integration source; scope it to the exact source ARN and account.
- Replicated tables are read-only in Redshift, but anyone with access to the destination database sees raw columns; apply Redshift row-level security or masking, or exclude personal-data tables.
- For the lakehouse target, permissions live in Lake Formation and the Glue catalog; review them alongside Unity Catalog grants.
- CloudTrail records integration create, modify and delete calls; EventBridge integration events give a change history.
Operating it
What to watch, and the limits
| Watch | Why |
|---|---|
| IntegrationLag | Seconds from commit on the source to queryable in Redshift. The main freshness signal; alarm on it. |
| IntegrationState | A number for the state: replicating, error, stopped, recovering. Alarm when it is not replicating. |
| IntegrationNumTablesFailedReplication | Count of tables that failed. Any value above zero needs a look. |
| IntegrationAutoRemediationTriggered | Tables that auto-remediation sent back to resync; frequent values point at DDL or bad data on the source. |
| SVV_INTEGRATION and SVV_INTEGRATION_TABLE_STATE | Redshift system views with per-database change times and per-table status (Synced, Failed, Resync Required). |
| Limit | Value and notes |
|---|---|
| Integrations per account (Aurora) | 100 per Region by default |
| Integrations per target | 50 sending to one Redshift or lakehouse target |
| Integrations per source cluster (Aurora) | 5 |
| Region | Source and target must be in the same Region |
| Lakehouse catalog name | 19 characters at most for Aurora integrations |
Limits and quotas checked 2026-11-11; many can be raised through Service Quotas.
Decisions
Situations and verdicts
| Situation | Choice | Verdict |
|---|---|---|
| Finance in Redshift wants live Aurora orders | Aurora zero-ETL into the existing Redshift | Keep if you have it: low effort while Redshift serves BI; register it as a source with an owner. |
| MES and LIMS data for the lakehouse | Zero-ETL into the SageMaker lakehouse, read from Databricks | Drop: skips S3 landing and the lander; use DMS into S3 landing. |
| DynamoDB device state needed in SQL | DynamoDB zero-ETL into Redshift or the lakehouse | Pursue with care: fine for analysts on Redshift; for Databricks, DynamoDB export to S3 into landing keeps one path. |
| Salesforce or SAP OData into Redshift | Glue zero-ETL from the SaaS app | Pursue with care: only where Redshift is the consumer; AppFlow into S3 landing is the platform route. |
FAQ
Questions people ask
Is zero-ETL really free?
For Aurora and RDS sources AWS charges nothing extra for processing changes, but you pay for source I/O and storage, the initial snapshot export, Redshift compute and storage of the copy. Glue- and DMS-based integrations have their own charges.
Can zero-ETL feed Databricks?
Not directly. Its targets are Redshift and the SageMaker lakehouse. Databricks can read lakehouse Iceberg tables through catalog federation, but for a governed Databricks platform DMS into S3 or Lakeflow Connect is the cleaner path.
Which databases support zero-ETL into Redshift?
Aurora MySQL, Aurora PostgreSQL, RDS for MySQL, RDS for PostgreSQL, RDS for Oracle, Oracle Database@AWS and DynamoDB, plus SaaS apps through AWS Glue and self-managed MySQL, PostgreSQL, SQL Server and Oracle through DMS. Region and engine version support varies, so check the supported Regions page.
How fresh is zero-ETL data?
Usually seconds to a minute or so after commit, shown by the IntegrationLag metric. Seeding and resyncs take much longer and depend on table size.
Can I transform data in a zero-ETL integration?
No, apart from choosing tables with filters. You transform afterwards, for example with Redshift materialized views or SQL jobs on the copy.
Sources
Checked against
- AWS docs: Zero-ETL integrations (Redshift)
- AWS docs: Aurora zero-ETL integrations, limits and quotas
- AWS docs: Zero-ETL integrations in AWS Glue
- AWS docs: Metrics for zero-ETL integrations
- AWS: Aurora pricing (zero-ETL section)
Checked 2026-11-11.
Related
Where it connects
Real-time analytics · Data platform on AWS and Databricks · Common Ingestion Framework · AWS DMS · Amazon Redshift · Amazon Aurora PostgreSQL · All AWS data services
← Transfer Family · Amazon MSK →