Home › Learn › Real-time analytics › Designs on each cloud

Real-time analytics · page 7 of 11

The eight designs on Azure, AWS, Google Cloud and open source

The eight real-time analytics designs with products for each role on Azure, Fabric, AWS, Google Cloud, Databricks, Snowflake and open source.

↑ All of real-time analyticsReal time, explainedLambda vs KappaComponents, left to rightChange capture toolsWhich route?Eight designsDesigns on each cloudContracts and bindingsPush to the screenWhat it costsEstablished technologies

Concept vs product

Same roles, different products

The eight designs name no product. This page gives each role a product, on each cloud. The design is the logic; the product is the language you write it in. Changing cloud means a new set of products for the same roles, not a new design.

Product names change often; check the current docs before you build.

Design by design

Products for each role

L1 · Direct query

The BI tool reads a replica of the source database. The abstract design

SourceCaptureEvent logProcessServeHistoryAPIPushScreen
RoleAzureAWSOpen source
Source and replicaAzure SQL or Hyperscale named replica, Postgres Flexible read replicaRDS or Aurora read replicaPostgres or MySQL streaming replica
ScreenPower BI DirectQuery with automatic page refreshQuickSight, GrafanaGrafana, Metabase, Superset

L2 · Direct capture into the serving store

A change capture tool writes straight into ClickHouse or similar; reshaping happens inside the store. The abstract design

SourceCaptureEvent logProcessServeHistoryAPIPush ?Screen
RoleAzureAWSGoogle CloudOpen source
CaptureFabric Eventstream CDC connectorsClickPipes, or Aurora zero-ETLDatastreamPeerDB, ClickPipes, Estuary
ServeFabric Eventhouse, or ClickHouse Cloud on AzureClickHouse Cloud, or Redshift via zero-ETLBigQuery with continuous queriesClickHouse
ReshapingKQL update policiesClickHouse materialized viewsBigQuery SQLClickHouse materialized views
Screen and pushReal-Time Dashboard, Power BI; Activator for alertsGrafana, or a small SSE serviceLookerGrafana, or a service that polls the store and pushes only changes

L3 · Log-centred streaming (Kappa)

Every change flows through an event log; a processor builds curated topics; a fast store serves; changes are pushed. The abstract design

SourceCaptureEvent logProcessServeHistory ?APIPushScreen
RoleAzureAWSOpen source and neutral
CaptureDebezium on Container Apps or AKS, writing to Event Hubs; or QlikDMS, or Debezium on MSK ConnectDebezium
Event log and schemasEvent Hubs Premium with its schema registryMSK with Glue Schema RegistryConfluent Cloud with Schema Registry, or Redpanda
ProcessStream Analytics, Databricks Structured Streaming, or Confluent Flink on AzureManaged Service for Apache FlinkConfluent Flink, or Flink you run
ServeEventhouse or Azure Data Explorer, or ClickHouse CloudClickHouse Cloud, Pinot (StarTree), OpenSearchClickHouse, Pinot, Druid
API and pushAPI on Container Apps with Azure Web PubSub or SignalRAPI Gateway WebSocket APIs, or AppSync subscriptionsA Node or Go gateway that reads a results topic and sends SSE
ScreenReactReactReact

L4 · Lakehouse, near real time

Change capture into Databricks, Fabric or Snowflake in small batches; the warehouse serves the dashboard. The abstract design

SourceCaptureEvent log ?ProcessServeHistoryAPIPushScreen
RoleFabricDatabricksSnowflakeAWSGoogle Cloud
CaptureFabric MirroringLakeflow ConnectOpenflow, Snowpipe StreamingDMS, or Aurora zero-ETLDatastream
Store and reshapeOneLake Delta tables, materialized lake viewsLakeflow Declarative Pipelines (streaming tables)Dynamic Tables with a target lag of about a minuteS3 Tables (Iceberg) or RedshiftBigQuery
ScreenPower BI in Direct Lake modePower BI, AI/BI dashboardsPower BI, SigmaQuickSightLooker

L5 · Kappa plus lakehouse

One log feeds a live store and a full history; the mature enterprise design. The abstract design

SourceCaptureEvent logProcessServeHistoryAPIPushScreen
RoleAzure with DatabricksAzure, all FabricAWSOpen source and neutral
CaptureDebezium or QlikEventstream CDCDMS or DebeziumDebezium
Event logEvent HubsEventstreamMSKConfluent Kafka
ProcessDatabricks Structured StreamingEventstream transforms, KQLManaged FlinkFlink
Live serveEventhouse, Azure Data Explorer, or ClickHouseEventhouseClickHouse, PinotClickHouse
HistoryDelta on ADLS (bronze, silver, gold)Eventhouse OneLake availability (the same data as Delta)Iceberg on S3 TablesIceberg through Confluent Tableflow, read by Trino, Databricks or Snowflake
ScreenPower BI for analytics, your own SSE app for liveReal-Time Dashboard, Power BIQuickSight and your own appSuperset and your own app

L6 · Streaming database

One engine keeps query results up to date as data changes, and pushes the changes. The abstract design

SourceCaptureEvent log ?ProcessServeHistory ?APIPushScreen
RoleOptions
CaptureThe engine's own Postgres or MySQL CDC, or reading from Kafka
EngineRisingWave (its cloud, or run on AKS or EKS), Materialize
PushSUBSCRIBE in the engine feeds a small SSE or WebSocket gateway
On AzureNo native service does this. The closest is Eventhouse materialized views plus Activator; otherwise the vendor's cloud or AKS.

L7 · HTAP: one database for both

The transaction database also answers analytical questions, using a column engine inside it. The abstract design

SourceCaptureEvent logProcessServeHistoryAPIPush ?Screen
DatabaseHow it does analyticsPush
Azure SQL, SQL ServerColumnstore indexes on the transaction tablesPower BI page refresh
AlloyDB for PostgresA built-in column engineNone built in
SingleStore, TiDB (TiFlash)Row and column storage in one databaseNone built in
Oracle Database In-MemoryAn in-memory column storeNone built in
Postgres with pg_duckdb or pg_mooncakeThe DuckDB engine inside PostgresSupabase Realtime, or LISTEN/NOTIFY
Cosmos DBFabric Mirroring (which replaces Synapse Link)Change feed

L8 · Lambda

A batch path and a speed path, merged in a serving layer. The abstract design

SourceCaptureEvent logProcessServeHistoryAPIPushScreen
RoleWhat it used to be
Batch pathHadoop or Spark batch jobs
Speed pathStorm or Spark Streaming
ServingHBase or Cassandra
VerdictDrop; for recognising older systems only

FAQ

Questions people ask

What is the Azure architecture for real-time analytics?

For real time: Debezium or Fabric Eventstream CDC, Event Hubs, Stream Analytics or Databricks, Fabric Eventhouse or Azure Data Explorer, and SignalR or Web PubSub for push. For near real time: Fabric Mirroring into OneLake and Power BI in Direct Lake mode.

What is the AWS architecture for real-time analytics?

DMS or Debezium, MSK or Kinesis, Managed Service for Apache Flink, ClickHouse Cloud or Pinot, and API Gateway WebSocket APIs. For near real time, Aurora zero-ETL into Redshift.

Can the same design run on any cloud?

Yes, if every role sits behind its own interface. Only the binding, which product plays each role, changes.

Next

Where to go next

Next page: Contracts and bindings at every hop

↑ All of real-time analytics · Data platform architecture · Key concepts