Home › Key concepts › Change data capture (CDC)
Key concept · Walk (intermediate) · the same on every platform
Change data capture (CDC)
Copy only what changed in a database, by reading its change log.
What it is
Every database keeps a log of inserts, updates and deletes. Change data capture reads that log and sends each change downstream. Instead of copying a whole table again, you copy only the rows that changed, including deletes.
See it happen: Problem Lab: hard deletes that incremental loads never see (simulation), recorded from a real run.
Why it matters
It is fast, light on the source database, and it catches deletes that a simple 'new rows since yesterday' query misses.
An example
A customer changes their address in the app. Within a minute the change lands in the bronze table as an update event.
The common mistake
Forgetting deletes: a customer closes their account, but the analytics copy keeps them forever.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Ingestioncopy data in from apps, databases and APIs | AWS DMS / AppFlow | Data Factory copy / Fabric mirroring | Datastream | Lakeflow Connect / Auto Loader | dlt or Airbyte |
| Streaminghandle events as they happen, not once a night | Kinesis / Amazon MSK | Event Hubs / Fabric Eventstream | Pub/Sub + Dataflow | Structured Streaming | Kafka or Redpanda |
Related ideas
Where it shows up
Part of the Walk level on these paths: Data Engineering (moving data).