Home › Learn › Real-time analytics › Change capture tools
Real-time analytics · page 4 of 11
Change capture: who turns database changes into events
Change data capture tools by source database: Debezium, Qlik Replicate, GoldenGate, Fivetran, Striim, cloud services, what each database needs, and the gotchas.
In plain words
Read what the database already writes
Every database keeps a log of its own changes, so it can recover after a crash. A change data capture (CDC) tool reads that log and turns each change into an event. The app does not change, and the database does very little extra work.
The tools
Who does change capture
| Tool | Type | Strongest sources | Writes to | Verdict |
|---|---|---|---|---|
| Debezium | Open source | Postgres, MySQL, MariaDB, SQL Server, MongoDB, Oracle, Db2, Cassandra, Vitess, Spanner, Informix | Kafka through Kafka Connect; Debezium Server writes to Kinesis, Pub/Sub, Event Hubs, Redis and more | Pursue: the open-source default; you run it |
| Qlik Replicate (formerly Attunity) | Commercial | Very broad: Oracle, SQL Server, Db2 including mainframe, IMS, VSAM, SAP, Postgres, MySQL | Kafka, Event Hubs, Snowflake, Databricks, Fabric, ADLS, S3 and more | Pursue for many mixed or older sources; little code |
| Oracle GoldenGate | Commercial | Oracle first; also SQL Server, Db2, MySQL, Postgres | Kafka, big-data targets, databases | Pursue for heavy Oracle; expensive |
| Fivetran (with HVR) | Commercial service | Many databases and SAP; HVR for high-volume Oracle and SAP | Warehouses and lakehouses, mostly directly | Pursue for near real time with no operations work |
| Striim | Commercial | Oracle, SQL Server, Postgres, MySQL | Kafka, clouds, warehouses; reshapes data on the way | Pursue when you want capture and light processing in one |
| IBM Data Replication (IIDR) | Commercial | Db2 for z/OS, IBM i, IMS | Kafka, databases | Pursue for IBM mainframe and IBM i |
| Precisely Connect, tcVISION | Commercial | Mainframe (VSAM, IMS, Db2), IBM i | Kafka, cloud | Alternatives for mainframe |
| AWS DMS | Cloud service | Oracle, SQL Server, Postgres, MySQL, Db2, MongoDB | Kinesis, MSK, S3, Redshift | Pursue inside AWS; simpler than the others |
| Google Datastream | Cloud service | Oracle, MySQL, Postgres, SQL Server | BigQuery, Cloud Storage | Pursue inside Google Cloud |
| Fabric Mirroring | Cloud service | Azure SQL, SQL Server, Cosmos DB, Postgres, Snowflake, Oracle (newer) | OneLake as Delta tables | Pursue for near real time on Azure |
| Fabric Eventstream CDC connectors | Cloud service | Azure SQL, Postgres, MySQL, Cosmos DB | Eventstream, Eventhouse | Pursue for real time on Azure |
| Databricks Lakeflow Connect | Platform | SQL Server, Postgres, MySQL, SaaS apps | Delta tables | Pursue on Databricks |
| Snowflake Openflow and connectors | Platform | Postgres, MySQL, SQL Server | Snowflake | Pursue on Snowflake |
| ClickPipes / PeerDB | Managed by ClickHouse | Postgres (strongest), MySQL, MongoDB | ClickHouse directly | Pursue for Postgres straight into ClickHouse |
| Flink CDC | Open source | MySQL, Postgres, Oracle, SQL Server, MongoDB, Db2 | Kafka, StarRocks, Doris, Paimon, Iceberg | Pursue if Flink is already central |
| Estuary Flow | Commercial service, open core | Postgres, MySQL, SQL Server, Oracle, MongoDB | Kafka-compatible output, ClickHouse, Snowflake, Databricks and more | Pursue as managed real time for small teams |
| Airbyte (Debezium inside) | Open source and service | Postgres, MySQL, SQL Server, MongoDB | Warehouses | Minutes, not seconds; drop for real time |
| Confluent managed CDC connectors | Commercial | Debezium-based for Postgres, MySQL, SQL Server, MongoDB; Oracle XStream (premium) | Confluent Kafka | Pursue on Confluent, so you do not run Connect yourself |
By source
What each database needs, and the gotcha
| Source | What must be switched on | Best options | Gotcha |
|---|---|---|---|
| PostgreSQL | wal_level = logical, a replication slot, a publication | Debezium, ClickPipes/PeerDB, Estuary, DMS, Datastream, Fabric Mirroring | If the reader stops, the slot holds the log and the disk fills up. Watch slot lag. |
| MySQL, MariaDB | Binlog in ROW format with full row images; GTID on | Debezium, DMS, Datastream, Flink CDC, ClickPipes | Binlog retention must outlast any reader outage. |
| SQL Server, Azure SQL | CDC switched on per database and per table | Debezium, Qlik, Fivetran, Fabric Mirroring, Lakeflow Connect, Striim | The change tables add load; small Azure SQL tiers have limits. |
| Oracle | ARCHIVELOG mode and supplemental logging | GoldenGate, Qlik, Fivetran HVR, Striim, Debezium (LogMiner or OpenLogReplicator) | Licensing: XStream needs a GoldenGate licence; LogMiner is free but struggles at high volume. Check early. |
| Db2 (Linux, Unix, Windows) | Replication capture switched on | Debezium, Qlik, IBM IIDR | |
| Mainframe (Db2 z/OS, IMS, VSAM) | Log or exit-based capture on the mainframe | Qlik, IBM IIDR, Precisely, tcVISION | Mainframe CPU is billed; capture close to the log. |
| IBM i (AS/400) | Journaling | IBM IIDR, Precisely, Qlik | |
| SAP (ECC, S/4HANA) | SLT or ODP extractors | SAP Datasphere replication flows, SAP SLT, Qlik, Fivetran HVR, Theobald | SAP licence terms limit reading the log directly; use approved paths. |
| MongoDB | A replica set (change streams) | Debezium, MongoDB Kafka Connector, Estuary, ClickPipes | A single node has no change stream. |
| Cosmos DB, DynamoDB, Spanner | The built-in change feed or stream | The native feed with Functions or Lambda; Fabric Mirroring for Cosmos DB; Debezium for Spanner | Use the native feed before a third-party tool. |
FAQ
Questions people ask
What is log-based change data capture?
A tool reads the change log a database already keeps for crash recovery and turns each insert, update and delete into an event, with no change to the app and little extra load.
Debezium or Qlik Replicate?
Debezium is free and open source and covers the common open-source databases, but you run it. Qlik Replicate is commercial, covers mainframes and SAP, and needs little code. Pick Qlik for many mixed or older sources, Debezium otherwise.
What is the best CDC tool for Oracle?
GoldenGate is the strongest, then Qlik Replicate, Fivetran HVR and Striim. Debezium works through LogMiner or OpenLogReplicator. Check Oracle licensing early: XStream needs a GoldenGate licence.
Next
Where to go next
Next page: Which route: straight to ClickHouse, straight to BI, or through a platform
↑ All of real-time analytics · Data platform architecture · Key concepts