Home › Learn › Real-time analytics › Established technologies

Real-time analytics · page 11 of 11

Which real-time technologies are established

Which real-time analytics technologies are established, growing or fading, layer by layer: CDC, event logs, contracts, processing, serving, history and push.

↑ All of real-time analyticsReal time, explainedLambda vs KappaComponents, left to rightChange capture toolsWhich route?Eight designsDesigns on each cloudContracts and bindingsPush to the screenWhat it costsEstablished technologies

At a glance

Safe, growing, or drop

A technology counts as established here when it is widely used in production, offered by more than one vendor or cloud, and easy to hire for. Growing ones are fine when you have a reason. The last column is for recognising older systems, not for new builds.

EstablishedKafka, Event HubsDebeziumFlink, SparkClickHouse, Pinot, DruidWebSocket, SSEGrowingRedpandaKafka on object storageRisingWave, MaterializeAsyncAPI, ODCSFabric Real-TimeDropStorm, SamzaSpark DStreamsksqlDBLambda serving layerslong-polling
Established, growing, and drop, in brief.

Layer by layer

The full list

LayerEstablished (safe)Growing (fine with a reason)Legacy or fading (drop)
Change captureDebezium, GoldenGate, AWS DMS, native change feeds (Cosmos DB, SQL Server CDC)Estuary, Flink CDCTrigger-based capture, polling timestamp columns
Event logApache Kafka (no ZooKeeper since 4.0), Kinesis, Event Hubs, Pub/SubRedpanda; Kafka on object storage (WarpStream, AutoMQ, diskless Kafka)RabbitMQ or SQS for analytics (no replay); Pulsar is losing momentum
ContractsAvro or Protobuf with a schema registry, CloudEventsAsyncAPI 3, ODCS data contractsJSON with no schema
ProcessingApache Flink, Spark Structured Streaming, Kafka StreamsRisingWave, Materialize, Spark real-time modeStorm, Samza, Spark DStreams, ksqlDB (Confluent moved to Flink)
ServingClickHouse, Apache Pinot, Apache Druid, Elasticsearch or OpenSearch, Redis, Azure Data ExplorerStarRocks, Fabric EventhouseLambda-style batch-plus-speed serving layers
HistoryDelta, IcebergConfluent Tableflow (topics written straight to tables)Hive tables
PushWebSocket, SSE, Azure SignalR, Socket.IOGraphQL subscriptions at scaleLong-polling
All-in-one platformsConfluent Cloud, DatabricksMicrosoft Fabric Real-Time Intelligence (generally available since 2024)HDInsight

The safe default

If you have to pick today

Debezium for change capture, Kafka or Event Hubs for the log, Flink only where you need joins or time windows, ClickHouse for serving, Iceberg or Delta for history, and SSE to the screen. Every piece is widely used in production, offered by more than one vendor, and easy to hire for. To place them, see the components and the eight designs.

FAQ

Questions people ask

What is the standard technology stack for real-time analytics?

Debezium for change capture, Kafka (or a cloud log such as Event Hubs or Kinesis), Flink or Spark for processing, ClickHouse, Pinot or Druid for serving, Iceberg or Delta for history, and WebSocket or SSE to the screen.

Is ksqlDB still a good choice?

Not for new builds. Confluent moved its stream processing to Flink, and ksqlDB is no longer where the investment goes.

Kafka or Redpanda?

Kafka is the established choice with the widest support. Redpanda speaks the Kafka protocol and is simpler to run; pick it with a reason, such as a small team or lower latency.

Next

Where to go next

↑ All of real-time analytics · Data platform architecture · Key concepts