Home › Learn › Real-time analytics

Data engineering · free guide · 11 short pages

Real-time analytics, drawn simply

How a change in a transaction database becomes a number on a screen within a second: the words people mix up, the components left to right, eight designs without product names, then the products on each cloud, the contracts at every hop, push to the screen, cost, and which technologies are safe.

In plain words

A shop window that updates itself

Imagine a shop where every sale is written in the till's notebook. Someone copies each new line onto a moving belt the moment it is written. Further down, people take lines off the belt, add them up, and update a big board in the window. Customers outside see the board change within a second, without asking.

The notebook is the database's change log. The copier is change capture. The belt is the event log (Kafka). The people adding up are the stream processor. The board's memory is the serving store, and the window is the screen that receives push updates.

The picture

Nine roles, left to right

Sourcethe app andits databaseCaptureturns changesinto eventsEvent logkeeps events,allows replayProcessclean, join,count windowsServefast answers,many usersAPImetrics,securityPushsends changesto screensScreendashboardor appHistory (lakehouse)cheap, full history; also the replay sourcedata moves left to right; the history store sits underneath, fed by the log
Every real-time design uses some of these nine roles. Which ones you keep is the design.

The short answer: real-time analytics is not the same as streaming. For new builds the usual design is Kappa: every change becomes an event in a replayable log, and everything downstream reads from it. But many screens need less: one change capture tool straight into ClickHouse, or into a lakehouse when a few minutes is fine.

The pages

Read in order, or jump in

FAQ

Questions people ask

What is the right architecture for real-time analytics?

Usually Kappa: every change becomes an event in a replayable log such as Kafka, and everything downstream reads from it. Start smaller when you can: one change capture tool straight into ClickHouse, or into a lakehouse if a few minutes is fine.

Where should I start on this site?

Start with real time explained, then the components left to right, then the eight designs. Cost and change capture tools are there when you need them.

Related

Where it connects

Data platform architecture shows where streaming sits in a whole platform, and the medallion layers show how the same layers run at two speeds. The ideas behind the boxes are in key concepts, such as change data capture.