Home › Learn › Real-time analytics
Data engineering · free guide · 11 short pages
Real-time analytics, drawn simply
How a change in a transaction database becomes a number on a screen within a second: the words people mix up, the components left to right, eight designs without product names, then the products on each cloud, the contracts at every hop, push to the screen, cost, and which technologies are safe.
In plain words
A shop window that updates itself
Imagine a shop where every sale is written in the till's notebook. Someone copies each new line onto a moving belt the moment it is written. Further down, people take lines off the belt, add them up, and update a big board in the window. Customers outside see the board change within a second, without asking.
The notebook is the database's change log. The copier is change capture. The belt is the event log (Kafka). The people adding up are the stream processor. The board's memory is the serving store, and the window is the screen that receives push updates.
The picture
Nine roles, left to right
The short answer: real-time analytics is not the same as streaming. For new builds the usual design is Kappa: every change becomes an event in a replayable log, and everything downstream reads from it. But many screens need less: one change capture tool straight into ClickHouse, or into a lakehouse when a few minutes is fine.
The pages
Read in order, or jump in
FAQ
Questions people ask
What is the right architecture for real-time analytics?
Usually Kappa: every change becomes an event in a replayable log such as Kafka, and everything downstream reads from it. Start smaller when you can: one change capture tool straight into ClickHouse, or into a lakehouse if a few minutes is fine.
Where should I start on this site?
Start with real time explained, then the components left to right, then the eight designs. Cost and change capture tools are there when you need them.
Related
Where it connects
Data platform architecture shows where streaming sits in a whole platform, and the medallion layers show how the same layers run at two speeds. The ideas behind the boxes are in key concepts, such as change data capture.