Home › Learn › Real-time analytics › Change capture tools

Real-time analytics · page 4 of 11

Change capture: who turns database changes into events

Change data capture tools by source database: Debezium, Qlik Replicate, GoldenGate, Fivetran, Striim, cloud services, what each database needs, and the gotchas.

↑ All of real-time analyticsReal time, explainedLambda vs KappaComponents, left to rightChange capture toolsWhich route?Eight designsDesigns on each cloudContracts and bindingsPush to the screenWhat it costsEstablished technologies

In plain words

Read what the database already writes

Every database keeps a log of its own changes, so it can recover after a crash. A change data capture (CDC) tool reads that log and turns each change into an event. The app does not change, and the database does very little extra work.

Databaseits change logWAL, binlog, redoreads the logCDC toolDebezium, Qlik,GoldenGate …Event log (Kafka)many consumersServing store directlye.g. ClickHouseLakehouse directlyDelta, Icebergthe app is untouched: the tool reads what the database already writes
One CDC tool can feed an event log, or write straight to a serving store or a lakehouse.

The tools

Who does change capture

ToolTypeStrongest sourcesWrites toVerdict
DebeziumOpen sourcePostgres, MySQL, MariaDB, SQL Server, MongoDB, Oracle, Db2, Cassandra, Vitess, Spanner, InformixKafka through Kafka Connect; Debezium Server writes to Kinesis, Pub/Sub, Event Hubs, Redis and morePursue: the open-source default; you run it
Qlik Replicate (formerly Attunity)CommercialVery broad: Oracle, SQL Server, Db2 including mainframe, IMS, VSAM, SAP, Postgres, MySQLKafka, Event Hubs, Snowflake, Databricks, Fabric, ADLS, S3 and morePursue for many mixed or older sources; little code
Oracle GoldenGateCommercialOracle first; also SQL Server, Db2, MySQL, PostgresKafka, big-data targets, databasesPursue for heavy Oracle; expensive
Fivetran (with HVR)Commercial serviceMany databases and SAP; HVR for high-volume Oracle and SAPWarehouses and lakehouses, mostly directlyPursue for near real time with no operations work
StriimCommercialOracle, SQL Server, Postgres, MySQLKafka, clouds, warehouses; reshapes data on the wayPursue when you want capture and light processing in one
IBM Data Replication (IIDR)CommercialDb2 for z/OS, IBM i, IMSKafka, databasesPursue for IBM mainframe and IBM i
Precisely Connect, tcVISIONCommercialMainframe (VSAM, IMS, Db2), IBM iKafka, cloudAlternatives for mainframe
AWS DMSCloud serviceOracle, SQL Server, Postgres, MySQL, Db2, MongoDBKinesis, MSK, S3, RedshiftPursue inside AWS; simpler than the others
Google DatastreamCloud serviceOracle, MySQL, Postgres, SQL ServerBigQuery, Cloud StoragePursue inside Google Cloud
Fabric MirroringCloud serviceAzure SQL, SQL Server, Cosmos DB, Postgres, Snowflake, Oracle (newer)OneLake as Delta tablesPursue for near real time on Azure
Fabric Eventstream CDC connectorsCloud serviceAzure SQL, Postgres, MySQL, Cosmos DBEventstream, EventhousePursue for real time on Azure
Databricks Lakeflow ConnectPlatformSQL Server, Postgres, MySQL, SaaS appsDelta tablesPursue on Databricks
Snowflake Openflow and connectorsPlatformPostgres, MySQL, SQL ServerSnowflakePursue on Snowflake
ClickPipes / PeerDBManaged by ClickHousePostgres (strongest), MySQL, MongoDBClickHouse directlyPursue for Postgres straight into ClickHouse
Flink CDCOpen sourceMySQL, Postgres, Oracle, SQL Server, MongoDB, Db2Kafka, StarRocks, Doris, Paimon, IcebergPursue if Flink is already central
Estuary FlowCommercial service, open corePostgres, MySQL, SQL Server, Oracle, MongoDBKafka-compatible output, ClickHouse, Snowflake, Databricks and morePursue as managed real time for small teams
Airbyte (Debezium inside)Open source and servicePostgres, MySQL, SQL Server, MongoDBWarehousesMinutes, not seconds; drop for real time
Confluent managed CDC connectorsCommercialDebezium-based for Postgres, MySQL, SQL Server, MongoDB; Oracle XStream (premium)Confluent KafkaPursue on Confluent, so you do not run Connect yourself

By source

What each database needs, and the gotcha

SourceWhat must be switched onBest optionsGotcha
PostgreSQLwal_level = logical, a replication slot, a publicationDebezium, ClickPipes/PeerDB, Estuary, DMS, Datastream, Fabric MirroringIf the reader stops, the slot holds the log and the disk fills up. Watch slot lag.
MySQL, MariaDBBinlog in ROW format with full row images; GTID onDebezium, DMS, Datastream, Flink CDC, ClickPipesBinlog retention must outlast any reader outage.
SQL Server, Azure SQLCDC switched on per database and per tableDebezium, Qlik, Fivetran, Fabric Mirroring, Lakeflow Connect, StriimThe change tables add load; small Azure SQL tiers have limits.
OracleARCHIVELOG mode and supplemental loggingGoldenGate, Qlik, Fivetran HVR, Striim, Debezium (LogMiner or OpenLogReplicator)Licensing: XStream needs a GoldenGate licence; LogMiner is free but struggles at high volume. Check early.
Db2 (Linux, Unix, Windows)Replication capture switched onDebezium, Qlik, IBM IIDR
Mainframe (Db2 z/OS, IMS, VSAM)Log or exit-based capture on the mainframeQlik, IBM IIDR, Precisely, tcVISIONMainframe CPU is billed; capture close to the log.
IBM i (AS/400)JournalingIBM IIDR, Precisely, Qlik
SAP (ECC, S/4HANA)SLT or ODP extractorsSAP Datasphere replication flows, SAP SLT, Qlik, Fivetran HVR, TheobaldSAP licence terms limit reading the log directly; use approved paths.
MongoDBA replica set (change streams)Debezium, MongoDB Kafka Connector, Estuary, ClickPipesA single node has no change stream.
Cosmos DB, DynamoDB, SpannerThe built-in change feed or streamThe native feed with Functions or Lambda; Fabric Mirroring for Cosmos DB; Debezium for SpannerUse the native feed before a third-party tool.

FAQ

Questions people ask

What is log-based change data capture?

A tool reads the change log a database already keeps for crash recovery and turns each insert, update and delete into an event, with no change to the app and little extra load.

Debezium or Qlik Replicate?

Debezium is free and open source and covers the common open-source databases, but you run it. Qlik Replicate is commercial, covers mainframes and SAP, and needs little code. Pick Qlik for many mixed or older sources, Debezium otherwise.

What is the best CDC tool for Oracle?

GoldenGate is the strongest, then Qlik Replicate, Fivetran HVR and Striim. Debezium works through LogMiner or OpenLogReplicator. Check Oracle licensing early: XStream needs a GoldenGate licence.

Next

Where to go next

Next page: Which route: straight to ClickHouse, straight to BI, or through a platform

↑ All of real-time analytics · Data platform architecture · Key concepts