Home › Learn › Real-time analytics › Components, left to right
Real-time analytics · page 3 of 11
The components, left to right
Every component of a real-time analytics pipeline, from the transaction database through change capture, Kafka, stream processing and ClickHouse to push on the screen.
The whole line
Eleven components, from the transaction to the screen
Each box
What each one does, and the products
| # | Component | What it does | Options | On Azure |
|---|---|---|---|---|
| 1 | Transaction system | The business app: orders, payments, point of sale, bookings | Your services | App Service, AKS |
| 2 | Transaction database | Stores the truth and records every change in its own log | Postgres (WAL), MySQL (binlog), Oracle (redo log), SQL Server (transaction log), MongoDB (oplog) | Azure SQL, Cosmos DB change feed, Azure Database for PostgreSQL |
| 3 | Change capture | Turns a change into an event | Log-based CDC (Debezium and others), outbox pattern, SDKs, IoT bridges | Debezium, Fabric Eventstream CDC connectors |
| 4 | Event log | Keeps events durably, in order per key, and lets consumers replay | Kafka, Redpanda, Pulsar, Kinesis, Google Pub/Sub, plus a schema registry | Event Hubs (speaks the Kafka protocol), Fabric Eventstream |
| 5 | Stream processing (optional) | Cleans, removes duplicates, joins reference data, counts per time window, handles late events | Flink, Kafka Streams, Spark Structured Streaming, RisingWave, Materialize | Stream Analytics, Databricks, Fabric Eventstream |
| 6 | Curated topics | Processed output goes back into the log as new topics, each with its own contract | Same log as 4 | Event Hubs |
| 7 | Serving store | Answers analytical questions fast, for many users | ClickHouse, Pinot, Druid, StarRocks; Redis for single-key lookups; OpenSearch for search | Fabric Eventhouse, Azure Data Explorer, Cosmos DB |
| 8 | Query / API layer | One definition per metric, security, caching | Cube, or your own REST or GraphQL service | API Management plus your API |
| 9 | Push gateway | Sends changes to browsers | Server-Sent Events, WebSocket, GraphQL subscriptions; Socket.IO, Ably, Pusher | Azure SignalR, Web PubSub, Fabric Activator |
| 10 | Screen | Shows a snapshot, then applies small changes as they arrive | React with a live-data client, or a BI tool | Fabric Real-Time Dashboard, Power BI |
| 11 | History store | Cheap long-term copy, and the replay source | Iceberg or Delta tables | ADLS with Delta, OneLake |
Step 3, closer
Who can create an event?
| Source | How it works | Watch out for | Verdict |
|---|---|---|---|
| Log-based CDC | A tool reads the database's own change log and emits each row change, with before and after values | Events describe table rows, not business meaning | Pursue as the default for databases |
| Outbox pattern | In the same transaction, the app writes a business event (OrderPlaced) to an outbox table; CDC ships it | Needs one small change in the app | Pursue for business events |
| App writes to Kafka directly | The app writes to its database and to Kafka separately | If one write fails, the two disagree (the dual-write problem) | Drop, unless the data never touches a database |
| Clickstream SDK | A script in the browser or app sends user actions | Ad blockers, consent rules | Pursue for user behaviour |
| IoT and devices | Devices publish over MQTT; a bridge writes to the log | Bursty, out-of-order arrivals | Pursue where relevant |
| Logs and metrics | Agents such as Fluent Bit, Vector or the OpenTelemetry Collector ship them | Volume | Pursue for operational analytics |
| SaaS webhooks | An HTTP endpoint receives the call and writes it to the log | Retries send duplicates | Pursue |
| Files dropped in batches | A file-arrival trigger turns each file into events | It is batch underneath | Use only as a bridge |
The tools that do log-based CDC, by source database, are on their own page: change capture tools.
Step 4, closer
A queue is not a log
RabbitMQ and SQS are queues: a message disappears once it has been read, so nothing can replay it. Kafka, Event Hubs, Kinesis and Pulsar are logs: they keep events for a set time, and any number of consumers can read, and re-read, at their own pace. Kappa needs a log.
Step 5, closer
Three kinds of receiver
- A stream processor (Flink and similar) reads raw topics, transforms them and writes curated topics.
- A sink connector (Kafka Connect, ClickPipes, Pinot's real-time ingestion) copies a topic as it is into a store.
- A lakehouse writer (Flink, Spark, or Confluent Tableflow) writes Iceberg or Delta tables.
The processor's output is another topic with its own contract, so the same pattern repeats: producer, topic, consumer. See contracts and bindings.
FAQ
Questions people ask
What are the components of a real-time analytics architecture?
A transaction system and database, change capture, an event log such as Kafka, optional stream processing, curated topics, a fast serving store such as ClickHouse, a lakehouse for history, an API layer, a push gateway and the screen.
Who creates the events in a streaming pipeline?
Usually a change data capture tool that reads the database's change log, or the app itself through the outbox pattern. Browser SDKs, IoT devices, log shippers and webhooks create events too.
Can I use RabbitMQ or SQS instead of Kafka?
Not for analytics that need replay. Queues delete a message once it is read; Kafka-style logs keep events so any consumer can re-read them.
Next
Where to go next
Next page: Change capture: who turns database changes into events
↑ All of real-time analytics · Data platform architecture · Key concepts