Home › Learn › Real-time analytics › Eight designs

Real-time analytics · page 6 of 11

Eight real-time designs, without product names

Eight abstract real-time analytics designs built from nine roles, with no product names: when to use each, delay, cost, effort and a verdict.

↑ All of real-time analyticsReal time, explainedLambda vs KappaComponents, left to rightChange capture toolsWhich route?Eight designsDesigns on each cloudContracts and bindingsPush to the screenWhat it costsEstablished technologies

The building blocks

Nine roles, no product names

Every design on this page uses the same nine roles. A design is just a choice of which roles to use. Products come later, on the physical designs page.

#RoleWhat it does
R1Sourcethe app and its database
R2Captureturns changes into events
R3Event logkeeps events, allows replay
R4Processclean, join, count windows
R5Servefast answers, many users
R6Historycheap, full history
R7APImetrics, security
R8Pushsends changes to screens
R9Screendashboard or app

At a glance

Eight designs

DesignTypical delaySmall monthly costRunning effortVerdict
L1 Direct querySeconds (replica lag)LowestLowPursue for one system, small data
L2 Direct capture into the serving store1 to 5 s$500–2k a monthLow to mediumPursue: the simplest real-time design
L3 Log-centred streaming (Kappa)Under 1 to 3 s$2–6k a monthHighPursue with several consumers
L4 Lakehouse, near real time1 to 15 min$0.5–3k a monthMediumPursue when people are the ones acting
L5 Kappa plus lakehouseUnder 1 s live, plus full history$5k+ a monthHighPursue as the end state
L6 Streaming databaseUnder 1 s$1–4k a monthMediumPursue for continuous joins plus push
L7 HTAP: one database for bothSecondsVariesLowPursue only for one system on a capable database
L8 LambdaMixedHighVery highDrop: legacy

Designs L1, L2, L6 and L7 skip the log; they are fine when one source feeds one consumer. L3 and L5 put the log at the centre, which you need once there are several consumers or a need to replay. L4 gives up speed for history and breadth. Costs are rough; see what it costs.

Each design

The eight, one by one

L1 · Direct query

The BI tool reads a replica of the source database.

SourceCaptureEvent logProcessServeHistoryAPIPushScreen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Pursue for one system, small data

Use it when: One system, small data, few users. Avoid when: Joins across systems, many users. Products for L1

L2 · Direct capture into the serving store

A change capture tool writes straight into ClickHouse or similar; reshaping happens inside the store.

SourceCaptureEvent logProcessServeHistoryAPIPush ?Screen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Pursue: the simplest real-time design

Use it when: One or two sources, one consumer, sub-second answers. Avoid when: Other systems need the same changes. Products for L2

L3 · Log-centred streaming (Kappa)

Every change flows through an event log; a processor builds curated topics; a fast store serves; changes are pushed.

SourceCaptureEvent logProcessServeHistory ?APIPushScreen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Pursue with several consumers

Use it when: Several consumers, replay, enrichment in flight. Avoid when: One source, one target. Products for L3

L4 · Lakehouse, near real time

Change capture into Databricks, Fabric or Snowflake in small batches; the warehouse serves the dashboard.

SourceCaptureEvent log ?ProcessServeHistoryAPIPushScreen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Pursue when people are the ones acting

Use it when: A few minutes is fine; history and joins across many systems. Avoid when: Sub-second answers for many users. Products for L4

L5 · Kappa plus lakehouse

One log feeds a live store and a full history; the mature enterprise design.

SourceCaptureEvent logProcessServeHistoryAPIPushScreen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Pursue as the end state

Use it when: Live screens and deep analytics together. Avoid when: A tight budget or a small team. Products for L5

L6 · Streaming database

One engine keeps query results up to date as data changes, and pushes the changes.

SourceCaptureEvent log ?ProcessServeHistory ?APIPushScreen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Pursue for continuous joins plus push

Use it when: A live screen built on joins that must stay up to date. Avoid when: You need wide tool support; the products are still maturing. Products for L6

L7 · HTAP: one database for both

The transaction database also answers analytical questions, using a column engine inside it.

SourceCaptureEvent logProcessServeHistoryAPIPush ?Screen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Pursue only for one system on a capable database

Use it when: One system whose database has a column engine. Avoid when: Several sources; analytics would slow the app. Products for L7

L8 · Lambda

A batch path and a speed path, merged in a serving layer.

SourceCaptureEvent logProcessServeHistoryAPIPushScreen
Strong boxes are used; dashed boxes are optional; faded boxes are left out.

Drop: legacy

Use it when: Only when you inherit it. Avoid when: Any new build. Products for L8

Choosing

Which one?

IfThen
One system, small data, few usersL1
One or two sources, sub-second answers, only one consumerL2
A few minutes is fine; history and joins across many systemsL4
Several consumers, replay, enrichment in flightL3
All of that, plus live screens and deep analyticsL5, usually reached by growing from L2 or L4
A live screen built on joins that stay up to date, with pushL6

A sensible path is to start with L2 or L4 and add the log when a second consumer appears. If each role sits behind its own interface, with a contract per topic and the product named only in a binding, moving between designs means swapping products, not rewriting. See contracts and bindings.

FAQ

Questions people ask

What are the main real-time analytics architectures?

Direct query on a replica, direct capture into a serving store, log-centred streaming (Kappa), lakehouse near real time, Kappa plus lakehouse, a streaming database, HTAP, and the older Lambda.

Which real-time architecture should I start with?

Usually direct capture into a serving store such as ClickHouse (real time) or a lakehouse with small batches (near real time). Add an event log such as Kafka when a second consumer appears.

What is HTAP?

Hybrid transactional and analytical processing: one database handles both the app's transactions and analytical queries, using a column engine inside it. Examples: SingleStore, TiDB, AlloyDB, SQL Server columnstore.

Next

Where to go next

Next page: The eight designs on Azure, AWS, Google Cloud and open source

↑ All of real-time analytics · Data platform architecture · Key concepts