Home › Learn › Real-time analytics › Real time, explained

Real-time analytics · page 1 of 11

Real time vs streaming vs near real time

Real-time analytics is a result; streaming is one way to move data. Real time (under a second) and near real time (seconds to minutes) need different designs and budgets.

↑ All of real-time analyticsReal time, explainedLambda vs KappaComponents, left to rightChange capture toolsWhich route?Eight designsDesigns on each cloudContracts and bindingsPush to the screenWhat it costsEstablished technologies

In plain words

The result vs the way of moving data

Streaming is a way of moving and processing data: one event at a time, all the time. Real-time analytics is a result: questions about very fresh data, answered in under a second, for many people at once. Streaming is usually how the data gets there, but you also need a store built for fast questions and a screen that receives updates.

You can have either without the other. A stream that feeds a report printed every night is still streaming. And you can send raw events straight into a fast analytics database, such as ClickHouse, with no stream-processing engine at all, and still get real-time answers.

StreamingReal-time analytics
What it isA way of moving data, event by event, continuouslyA result: fresh answers in under a second, for many users
Question it answersHow does data move?How fresh and how fast are the answers?
Can it exist alone?Yes: streaming into a nightly reportYes: raw events straight into ClickHouse, no processing engine

Freshness

Real time and near real time are different

Hard real timemicroseconds,guaranteedReal timeunder1 secondNear real time5 seconds to15 minutesBatchhours toa dayslower answers, but simpler and cheapermachines, not analyticsfraud, bidding, live opsdashboards people watchreports, planning
Four tiers. Each step to the right is slower, but simpler and cheaper.
TierDelay, end to endWho acts on itExamplesTypical design
Hard real timeMicroseconds to milliseconds, guaranteedMachinesFactory control, car brakes, high-frequency tradingEmbedded systems and special hardware. Not Kafka or analytics platforms; out of scope here.
Real timeUnder about 1 secondMachines, or a person who must react right nowBlocking a fraudulent payment, ad bidding, a live operations roomEvent log, stream processor, fast analytics store, push to the screen
Near real timeAbout 5 seconds to 15 minutesPeople watching dashboardsToday's sales, stock levels, call-centre boardsSmall batches into a warehouse or lakehouse; the dashboard refreshes every 30 to 60 seconds
BatchHours to a dayPeople planningFinance reports, monthly cost reportsA nightly job

The one question

Who acts, and how fast can they act?

Ask what decision changes, who makes it, and how quickly they can act on it. A person watching a dashboard cannot act on a change that is half a second old, so near real time is almost always enough for people. A program that blocks a payment can act in milliseconds, so it needs real time.

Most requests for a "real-time dashboard" turn out to be near real time once you ask this question. That matters, because real time usually costs three to ten times as much as the batch version of the same report (see what it costs).

The cost line: near real time stays cheap only while compute can switch off between batches. Below about a five-minute interval, warehouse compute never gets to pause, so you pay for always-on compute and still have a batch design. At that point a proper streaming design is often cheaper.

FAQ

Questions people ask

Is real-time analytics the same as streaming?

No. Streaming is a way of moving data, event by event. Real-time analytics is the result: fresh answers in under a second for many users. Streaming is usually the transport, but you also need a fast analytics store and a screen that receives updates.

What is the difference between real time and near real time?

Real time is under about one second end to end and is needed when a machine acts on the data. Near real time is seconds to about fifteen minutes and is enough when people watch a dashboard. Near real time is usually much cheaper.

Is Kafka real time?

Kafka itself moves events in milliseconds, but it is soft real time: fast on average, with no guaranteed deadline. Hard real-time systems, such as brakes, use special hardware, not Kafka.

Next

Where to go next

Next page: Lambda vs Kappa, drawn simply

↑ All of real-time analytics · Data platform architecture · Key concepts