Home › Learn › Real-time analytics › Components, left to right

Real-time analytics · page 3 of 11

The components, left to right

Every component of a real-time analytics pipeline, from the transaction database through change capture, Kafka, stream processing and ClickHouse to push on the screen.

↑ All of real-time analyticsReal time, explainedLambda vs KappaComponents, left to rightChange capture toolsWhich route?Eight designsDesigns on each cloudContracts and bindingsPush to the screenWhat it costsEstablished technologies

The whole line

Eleven components, from the transaction to the screen

1. Transaction apporders, payments2. Databasewrites a changelog3. Change captureCDC or outbox4. Event logKafka, EventHubs, Kinesis5. Stream processoptional: clean,join, windows6. Curated topicsback intothe log7. Serving storeClickHouse,Pinot, Druid8. APImetrics,security9. Push gatewaySSE orWebSocket10. Screensnapshot, thenchanges11. HistoryIceberg orDeltathe serving store and the history store both read the curated topics
Read the top row left to right, then the bottom row. Stream processing (5) is optional; history (11) sits beside the live path.

Each box

What each one does, and the products

#ComponentWhat it doesOptionsOn Azure
1Transaction systemThe business app: orders, payments, point of sale, bookingsYour servicesApp Service, AKS
2Transaction databaseStores the truth and records every change in its own logPostgres (WAL), MySQL (binlog), Oracle (redo log), SQL Server (transaction log), MongoDB (oplog)Azure SQL, Cosmos DB change feed, Azure Database for PostgreSQL
3Change captureTurns a change into an eventLog-based CDC (Debezium and others), outbox pattern, SDKs, IoT bridgesDebezium, Fabric Eventstream CDC connectors
4Event logKeeps events durably, in order per key, and lets consumers replayKafka, Redpanda, Pulsar, Kinesis, Google Pub/Sub, plus a schema registryEvent Hubs (speaks the Kafka protocol), Fabric Eventstream
5Stream processing (optional)Cleans, removes duplicates, joins reference data, counts per time window, handles late eventsFlink, Kafka Streams, Spark Structured Streaming, RisingWave, MaterializeStream Analytics, Databricks, Fabric Eventstream
6Curated topicsProcessed output goes back into the log as new topics, each with its own contractSame log as 4Event Hubs
7Serving storeAnswers analytical questions fast, for many usersClickHouse, Pinot, Druid, StarRocks; Redis for single-key lookups; OpenSearch for searchFabric Eventhouse, Azure Data Explorer, Cosmos DB
8Query / API layerOne definition per metric, security, cachingCube, or your own REST or GraphQL serviceAPI Management plus your API
9Push gatewaySends changes to browsersServer-Sent Events, WebSocket, GraphQL subscriptions; Socket.IO, Ably, PusherAzure SignalR, Web PubSub, Fabric Activator
10ScreenShows a snapshot, then applies small changes as they arriveReact with a live-data client, or a BI toolFabric Real-Time Dashboard, Power BI
11History storeCheap long-term copy, and the replay sourceIceberg or Delta tablesADLS with Delta, OneLake

Step 3, closer

Who can create an event?

SourceHow it worksWatch out forVerdict
Log-based CDCA tool reads the database's own change log and emits each row change, with before and after valuesEvents describe table rows, not business meaningPursue as the default for databases
Outbox patternIn the same transaction, the app writes a business event (OrderPlaced) to an outbox table; CDC ships itNeeds one small change in the appPursue for business events
App writes to Kafka directlyThe app writes to its database and to Kafka separatelyIf one write fails, the two disagree (the dual-write problem)Drop, unless the data never touches a database
Clickstream SDKA script in the browser or app sends user actionsAd blockers, consent rulesPursue for user behaviour
IoT and devicesDevices publish over MQTT; a bridge writes to the logBursty, out-of-order arrivalsPursue where relevant
Logs and metricsAgents such as Fluent Bit, Vector or the OpenTelemetry Collector ship themVolumePursue for operational analytics
SaaS webhooksAn HTTP endpoint receives the call and writes it to the logRetries send duplicatesPursue
Files dropped in batchesA file-arrival trigger turns each file into eventsIt is batch underneathUse only as a bridge

The tools that do log-based CDC, by source database, are on their own page: change capture tools.

Step 4, closer

A queue is not a log

RabbitMQ and SQS are queues: a message disappears once it has been read, so nothing can replay it. Kafka, Event Hubs, Kinesis and Pulsar are logs: they keep events for a set time, and any number of consumers can read, and re-read, at their own pace. Kappa needs a log.

Step 5, closer

Three kinds of receiver

  1. A stream processor (Flink and similar) reads raw topics, transforms them and writes curated topics.
  2. A sink connector (Kafka Connect, ClickPipes, Pinot's real-time ingestion) copies a topic as it is into a store.
  3. A lakehouse writer (Flink, Spark, or Confluent Tableflow) writes Iceberg or Delta tables.

The processor's output is another topic with its own contract, so the same pattern repeats: producer, topic, consumer. See contracts and bindings.

FAQ

Questions people ask

What are the components of a real-time analytics architecture?

A transaction system and database, change capture, an event log such as Kafka, optional stream processing, curated topics, a fast serving store such as ClickHouse, a lakehouse for history, an API layer, a push gateway and the screen.

Who creates the events in a streaming pipeline?

Usually a change data capture tool that reads the database's change log, or the app itself through the outbox pattern. Browser SDKs, IoT devices, log shippers and webhooks create events too.

Can I use RabbitMQ or SQS instead of Kafka?

Not for analytics that need replay. Queues delete a message once it is read; Kafka-style logs keep events so any consumer can re-read them.

Next

Where to go next

Next page: Change capture: who turns database changes into events

↑ All of real-time analytics · Data platform architecture · Key concepts