Home › Data engineering › Streaming
Data engineering · free guide · one page
Streaming architecture on AWS and Databricks
IoT and ordinary sources, seconds and minutes, AWS alone or with Databricks: the architecture as roles first, then the options for every part with pros, cons and a verdict.
How live data gets from factory machines (and from databases, apps and files) to screens, alerts and reports. The guide starts with the architecture as roles with no product names, then gives options on AWS alone and on AWS with Databricks for every role, each with pros, cons and a verdict. A tyre maker is the worked example throughout.
The worked example
A tyre maker with five plants. Curing presses, mixers and extruders report temperature, pressure and cycle state every few seconds. Supervisors want alerts and live boards within seconds. Planners and quality teams want hourly numbers they can trust. SAP, MES, a lab system (LIMS) and a dealer portal also produce data that has to join with the sensor data.
Design rules used throughout
- One event stream feeds two lanes: a live lane (seconds, for acting now) and a platform lane (minutes, the number of record).
- The platform lane does not run continuously: streaming tables run once every 15 minutes and stop.
- The live lane never writes gold. Gold is the number of record.
- One contract per stream: Avro in a schema registry; key =
press_id; every record carriesreading_idand event time. The stream keeps 7 days. - Gold facts are built by recomputing changed time windows, with as-of joins to history dimensions, not by joining inside the stream.
What a stream is, in plain words
A batch is a box of data delivered at a time: "here are yesterday's orders". A stream is a conveyor belt: small messages keep arriving, one after another, and never "finish". Streaming is a way of delivering data, not a kind of data. The same orders can arrive as a nightly file or as one message per order.
Three things make streams harder than batches:
- Time has two clocks. Event time is when it happened at the press. Arrival time is when the message reached us. A gateway that was offline for 20 minutes delivers old readings late, so reports must group by event time.
- Nothing ever "ends". To say "the 10:00–11:00 average is final", you need a rule for how long to wait for stragglers. That rule is a watermark, for example "wait 10 minutes after the hour".
- Messages can arrive twice. Senders retry when unsure. Every reading carries a unique
reading_idso duplicates can be removed and totals never double.
Where a tyre company gets streams from
Machines (IoT)
Curing presses, mixers and extruders report temperature, pressure and cycle state every few seconds.Connected tyres (IoT, maybe later)
Sensors inside tyres on fleet vehicles report pressure and temperature. Millions of small senders.Database changes
SAP orders and MES production records. These are not events by nature; change capture (CDC) turns every insert or update into an event.App events
Dealer portal clicks, warranty-claim app submissions.Webhooks
Salesforce or a payment provider calls our URL when something changes.Files and logs
LIMS lab results, partner files, application logs. Files arrive in bursts but can be read "as a stream" of files.The key question for every source is the same: who must act on this data, and how fast? If a supervisor must stop a press within a minute, it needs the live lane. If a planner reads it tomorrow, a 15-minute or daily load is cheaper and simpler.
Conceptual architecture (no products)
Twelve roles. Every physical design later is just a choice of product for each role.
Solid arrows: the stream. Dashed arrows: sources that usually do not need a stream. Dotted: optional commands back to devices.
| Role | Job, in one sentence | Why it must exist |
|---|---|---|
| R1 Edge and gateway | Collects machine signals on the factory floor, buffers them when the network is down, and sends them in batches. | Factory networks drop; sending each reading alone is slow and expensive. |
| R2 Front door | Checks who is sending, with one identity per device or app, and accepts messages. | One stolen device can be switched off without touching the rest. |
| R3 Event log | Keeps messages in order per key (press) for days, so many readers can read at their own pace and replay. | Separates senders from readers; makes faults recoverable. |
| R4 Contract | Defines the message shape and versions it in a registry. | A firmware update must not silently break every reader. |
| R5 Live processor | Computes 1-minute windows, limit and drift checks, and emits only changes. | Someone must act within minutes. |
| R6 Live store | Holds latest values and the last 48 hours for live screens. | A new screen needs a snapshot before changes start flowing. |
| R7 Push and alerts | Sends a snapshot, then changes, to screens; sends alerts to people and tickets. | Screens should not keep asking "anything new?". |
| R8 Lake ingest | Copies the log, or files, into bronze tables every 15 minutes. | Reports and ML need history, not a belt. |
| R9 Lake tables | Bronze keeps everything forever; silver cleans; gold joins and aggregates. | One number of record, rebuildable from bronze. |
| R10 Use | Dashboards, Genie, ML features, APIs. | Where value shows up. |
| R11 Commands back | Changes settings on devices, such as reporting rate. | Optional; only if the business needs control, not just watching. |
| R12 Cross-cutting | Security, monitoring, quality, catalog, cost, tests. | Same concerns as batch, plus lag and "silent stream". |
Seconds or minutes: two service levels, end to end
"Real time" means two different promises. Each needs its own path and has its own cost. End to end means from the moment something happens at the press to the moment a person can see it.
| Tier | Promise (end to end) | Who acts | Path | Rough cost a month (production) | Verdict |
|---|---|---|---|---|---|
| T0 · under a second | Machine stops itself | The machine | Plant control system (PLC, SCADA), not the cloud | — | Drop for the cloud |
| T1 · seconds | 99% of alerts on screen within 5 s; window values within 90 s | Operator, supervisor | Live lane: Kinesis → Flink (Lambda for simple limits) → DynamoDB → WebSocket push | ~$530 | Pursue |
| T2 · about a minute | Data in a table within 1–3 min | Rarely anyone | Always-on streaming table, or Auto Loader in continuous mode | +$700–1,500 (always on) | Drop unless a named user needs it |
| T3 · minutes | 99% of readings in silver within 20 min of arrival; gold hour complete within 75 min of hour end | Shift lead, planner, quality | Platform lane: streaming tables trigger-once every 15 min → silver → hourly gold | ~$420–1,120 | Pursue |
| T4 · daily | Ready by 06:00 | Management, finance | Batch | part of batch | Pursue (existing) |
End-to-end total for T1 + T3: about $950–1,650 a month at production scale. The hop-by-hop budget is in Performance and sizing. The report that proves the promises is described under SLA report.
Three speeds, two lanes: where they meet
There are three speeds: seconds (1–5 s), near real time (15–75 min) and batch (daily). There are only two lanes. Near real time and batch are the same pipeline (same code, same tables) run at a different cadence. The seconds lane is a separate, light, disposable lane.
(1) One raw stream, read separately by each lane. (2) Reference data is copied from the platform lane into the seconds lane on a schedule. (3) Near real time and batch meet in gold. (4) The screen shows live and final numbers side by side, labelled.
Four questions people ask
| Question | Answer | Verdict |
|---|---|---|
| Does the 1–5 s lane only do light transforms, never call Databricks? | Yes. Allowed: validate against the contract, short-window dedupe, unit conversion, limit checks, 1-minute windows per press, change detection, lookup of small reference data held inside the lane. Not allowed: calls to Databricks or any warehouse at runtime, multi-source joins, SCD2 as-of joins, history beyond 48 hours. A call to Databricks would add seconds and tie the lane's uptime to the platform's. | Pursue |
| Should the fast lane send its data to the slow lane? | Not its readings. The platform lane reads the raw stream from Kinesis itself, so a bug in the fast lane can never corrupt the number of record. The fast lane sends only what it creates: alerts raised and acknowledged (a new fact, "alert events"), and its one-minute SLA summaries. Both go via Firehose to bronze. | Pursue (raw from Kinesis; alerts as their own dataset) |
| (the alternative) Platform lane reads the fast lane's output | Couples the two lanes; a Flink bug or a dropped late reading flows into gold. | Drop |
| Do the speeds meet? | In four places, and only these:
| Pursue |
| Is near real time joined only with near real time, or with batch too? | With batch too. Most joins in gold are near real time to batch: sensor hours joined to the SAP material dimension (daily) and to LIMS scrap (hours late). The rule: a gold row is only as fresh as its slowest input. Each hour carries a "complete" flag that turns true only when every input's watermark has passed it. Until then the row shows with unknown members (−1), and window recompute fixes it when the batch input lands. Each fact declares its freshness from its slowest input. | Pursue |
| Can one dataset exist in all three speeds? | Yes, as one dataset with one contract and one owner, served at different speeds. Press readings: seconds = latest values and 1-minute windows for 48 hours (disposable); near real time = silver readings and press-hour gold; batch = daily OEE, monthly reports, ML training from the same gold. Only one copy is the record (bronze, silver and gold); the live copy expires. Expect small differences between live and gold (late data, duplicates); the SLA report shows that drift and alerts above a set level (say 2%). | Pursue (one dataset, many speeds, one record) |
Kappa, Lambda, and where we sit
Lambda: two paths, one answer, merged. Kappa: one path, history rebuilt by replay. Ours: Kappa where bronze plays the long log; the live lane is a different product.
How Kappa fits, in four points
- Kappa's idea: everything is a stream, there is one code path, and to fix or change logic you replay the log from the start. No separate batch code.
- The catch: Kappa needs a log that keeps everything. Kinesis can keep up to 365 days and Kafka can keep forever with tiered storage, but paying a log to hold years of readings is expensive and awkward to query.
- Our answer: the log keeps 7 days (for outages); bronze keeps everything forever on S3, cheaply. Bronze is Kappa's long log. Spark reads bronze with the same code as a stream or as a full replay, so there is still one code path.
- Why our two lanes are not Lambda: Lambda computes the same number twice and merges. Our live lane computes "act now" signals that are never reported as the number of record; gold comes only from the platform lane. So the two lanes never disagree about a reported number.
Where Kappa does not fit, and what we do instead
| Case | Why pure Kappa struggles | What we do | Verdict |
|---|---|---|---|
| Sensor readings to bronze and silver | Fits well: event-native, append-only, keyed by press. | Stream (trigger-once, 15 min); replay from bronze when logic changes. | Pursue Kappa |
| Gold facts that join several inputs (OEE, scrap per press-hour) | Inputs arrive at different speeds (LIMS results hours later); a pure stream join would hold state for hours and still miss corrections. | Window recompute: find changed hours, rebuild them fully. | Pursue window recompute |
| SAP and MES tables | Not event-native. Making every table a stream adds CDC tooling and a log for data nobody needs in seconds. | CDC lands as files every few minutes; the same incremental pipeline reads them. | Drop Kappa here |
| Live lane after a logic change | Flink state must be rebuilt; Kinesis has only 7 days. | Fine: the live lane only needs 48 hours. Replay 2 days from Kinesis. | Pursue (7 days is enough) |
One-line summary: Kappa for event-native sources, with bronze as the long log; incremental batch for databases and files; window recompute for multi-input gold. This is "one medallion, two speeds", with Kappa made explicit.
What happens in bronze, silver and gold
The same three layers as batch, run incrementally. Each layer has one job, so faults are easy to find.
| Layer | Sensor stream (event) | Database changes (CDC from SAP, MES) | Never do here |
|---|---|---|---|
| Bronze keep everything | Append every record as it came, plus envelope: shard, sequence number, Kinesis arrival time, ingest run id, schema version. Payload kept raw (or as VARIANT). Clustered on ingest date. Kept forever. | Append every change row from DMS files (insert, update, delete, commit time, change sequence). Kept forever. | Deduplicate, filter, fix values |
| Silver one clean truth per source | The 13 standard stages: decode by schema version, types, event time in UTC plus plant local, quality checks (range, stuck sensor, clock skew), deduplicate on reading_id, conform press_id, append. Bad rows go to quarantine with a reason. | Same stages, then history: master data (press, material, dealer) as SCD type 2; transactions (orders, claims) as SCD type 1 (type 2 where the business needs the past state). Changes applied in source order, not arrival order. | Join across sources (silver is source-aligned) |
| Gold business numbers | Stream-hour job: press-minute and press-hour aggregates; facts such as OEE and scrap per press-hour by window recompute; each hour carries a "complete" flag. | Conformed dimensions with surrogate keys built from silver SCD2 (−1 unknown, −2 not applicable); facts join dimensions as of event time. | Read the live store; inner-join to dimensions |
Where SCD type 1 and type 2 happen
- In silver. The dataset kind sets the history type. Master data is SCD2, transactions are SCD1, events are append-only, and snapshots are kept as snapshots. Only the listed tracked columns open a new version.
- Sensor readings get no SCD. They are facts that never change. A "correction" is a new reading, and deduplication keeps the first.
- Gold does not redo SCD. It builds dimensions from silver's SCD2 history, adds surrogate keys, and joins facts as of event time.
- The streaming catch for SCD2: changes can arrive out of order, and a late change must be slotted into the middle of history. Two ways to do it:
- DLT's AUTO CDC (formerly APPLY CHANGES) handles this with a sequence column.
- Plain Spark needs our own MERGE in
foreachBatchthat re-cuts the valid-from and valid-to dates. This is one reason to consider DLT for CDC tables (see Databricks choices).
- Changing a table from type 1 to type 2 is a replay from bronze, which still holds every change, so the past gets history too.
Joining data in a stream
In batch, a join sees both tables complete. In a stream, one or both sides are still arriving, so every join has to answer: how long do I remember the other side, and what if it is late? Memory held between messages is called state. State costs RAM, and lost state means a wrong answer.
Example: link each curing cycle to the tyre serial scanned at the press. The processor keeps unmatched cycles until the watermark passes, then gives up and routes them to an "unmatched" output.
Five kinds of join, with tyre examples
| Join kind | Tyre example | How it works | Gotcha | Engine support | Where we do it | Verdict |
|---|---|---|---|---|---|---|
| Stream to static (lookup) | Reading + press master (plant, line, mould) | Each message looks up the current row of a small table. | Uses the dimension as it is now. A late or changed dimension row does not fix messages already joined. Missing key: keep the row, mark it unknown. | Spark: stream-static join, table re-read each micro-batch. Flink: lookup join (JDBC, DynamoDB) or broadcast state. | Live lane only, to label screens. | Pursue (live only) |
| Stream to stream, time-bounded | Press cycle start + barcode scan at the press (picture above) | Both sides wait in state until a match or the watermark passes. | State size = message rate × window. No watermark means state grows forever. Unmatched rows must go somewhere visible. | Spark: inner and outer joins with watermarks on both sides. Flink: interval join. | Silver, when the link is needed within the hour; otherwise gold. | Pursue, with a declared window |
| As-of (temporal) join | Reading at 10:04 + the curing recipe valid at 10:04 (SCD2) | Picks the dimension version valid at the event time, not today's. | Needs the dimension's history. Late recipe changes mean re-joining. | Flink: temporal table join, native. Spark streaming: not native; done in batch over Delta. | Gold, by window recompute. | Pursue in gold |
| Window first, then join | Per press-hour sensor averages + scrap from LIMS, which arrives hours later | Aggregate each side to the same grain (press-hour), then join the aggregates. | One side is hours late, so the result changes after it was first shown. | Any engine; easiest in batch. | Gold: the stream-hour job, then window recompute. | Pursue in gold |
| Stream to changelog (CDC) table | Dealer app events + SAP customer table kept up to date by CDC | Both sides are changing; the join must re-emit when either side changes. | Unbounded state unless a time-to-live is set; results get retracted and re-emitted, which confuses most sinks. | Flink: regular join on changelogs. Spark: not as a stream; use MERGE in batch. | Gold, batch. | Drop in stream |
Five join hazards and what handles each
- Late dimension (a new press appears in readings before it appears in the master table): keep the reading with an unknown key (−1). Window recompute fixes it when the master row lands.
- Fan-out (duplicate key on the "one" side multiplies rows): count duplicate keys on the one side separately; one net count hides mistakes that cancel out.
- Clock skew (a press clock that is 3 minutes wrong): the gateway stamps its own time too; when the two differ by more than a limit, the reading is flagged.
- State loss (a processor restart without its checkpoint): checkpoints on S3; the live lane can rebuild from 2 days of Kinesis.
- Re-emitted results: sinks are idempotent (MERGE on key, or upsert by press and minute), so a retry never double counts.
Getting to the UI: direct, via a fast layer, or via gold
The question is not "how fast can data reach a screen" but "what does a screen need when it opens": the latest value, a chart of the last hours, or the ability to slice and filter.
| Route | Path | Delay | Pros | Cons | Rough cost | Verdict |
|---|---|---|---|---|---|---|
| A · Direct push, no store | Flink → Lambda → API Gateway WebSocket → screen | ~1 s | Fewest parts | A screen that opens or reconnects sees nothing until the next change; no chart of the last hour | ~$100 | Drop as the only route |
| B · Small live store + push | Flink → DynamoDB → DynamoDB Streams → Lambda → WebSocket; a new screen reads the snapshot from DynamoDB first | 1–3 s | Snapshot then changes; serverless; cheap; 48-hour charts from the same table | No ad-hoc slicing; key design matters | ~$150 | Pursue (recommended) |
| C · Fast analytic layer (ClickHouse) | Kinesis → ClickHouse (ClickPipes) or Flink → ClickHouse; the API polls every second and pushes only differences; Grafana can query it directly | 2–5 s | SQL, slicing and filters over days, many users at once | New vendor; does not push; a third copy with its own numbers to keep consistent with gold | Dev <$100; production $500–3k | Pursue only if live screens need slicing beyond 48 hours |
| D · Lakebase as the live store | Flink → Lakebase (Postgres); the app reads with SQL and polls | 2–5 s | SQL; Databricks-native; already our metadata store | No push; always-on compute; Flink to Postgres sink to own | ~$100–300 | Pursue as the Databricks-leaning alternative to B |
| E · Via gold | Gold → metric views → AI/BI dashboards (auto-refresh), Genie; or gold synced to Lakebase for an app | 15–75 min | The number of record; governed; no extra copy | Not live | part of the platform | Pursue for reports; Drop for live |
Rule: live screens say "live, not final". A number that goes into a report always comes from gold, whatever the route.
Databricks choices: DLT or not, Spark Streaming, Auto Loader, file arrival
Is Spark Structured Streaming worth it when we only run every 15 minutes?
Yes, for the platform lane. Even when it runs once and stops (trigger "available now"), a Spark streaming read gives us the bookkeeping for free: which Kinesis records or which files are already done (the checkpoint), exactly-once writes into Delta, and catch-up after a missed run. Doing the same with plain batch reads means writing our own "where did I stop" logic for every source.
| Use | Pros | Cons | Verdict |
|---|---|---|---|
| Structured Streaming, trigger available-now, every 15 min (platform lane) | Checkpoints, exactly-once to Delta, catch-up, same code later if we go faster; pay only for run minutes on serverless | Checkpoint is tied to the query; changing a query shape can need a new checkpoint (a replay) | Pursue |
| Structured Streaming always on (micro-batch, seconds) | One engine for both lanes | Always-on cluster; weaker event-time state than Flink | Drop for the live lane |
| Real-time mode (GA 2026, under 100 ms claimed) | Spark at Flink-like latency | Fixed classic cluster, always on; check Kinesis support | Drop now; revisit to retire Flink |
| Old Spark Streaming (DStreams) | — | Deprecated | Drop |
| Plain batch with our own high-water mark | Simple to read | We rebuild what checkpoints already do; easy to miss late files | Drop for streams and files |
DLT (Lakeflow Declarative Pipelines) or not?
DLT means you declare tables (streaming tables, materialized views) and their rules, and Databricks runs them: checkpoints, retries, ordering, data quality expectations, and AUTO CDC for SCD. The alternative ("non-DLT") is our own PySpark code using Structured Streaming inside Lakeflow Jobs. The reference ETL on this site is built that way: the 13 silver stages as pure PySpark functions, testable on a laptop.
| Option | Pros | Cons | Verdict |
|---|---|---|---|
Non-DLT: Structured Streaming in Lakeflow Jobs (our 13 stages in foreachBatch) | Full control of every step; runs locally and in CI; portable to Fabric, EMR, Glue, GCP (only adapters change); our run recorder sees every step | We own checkpoints, SCD2 MERGE with late changes, retries and DQ wiring | Pursue as the default engine |
| DLT for everything | Less code; managed checkpoints and retries; expectations; lineage; serverless; good event log | Databricks-only (breaks portability); tables owned by the pipeline; a full refresh can wipe a streaming table (must lock it); one update at a time per pipeline; harder to test outside Databricks; our 13 stages must be rewritten as declarations | Drop |
| DLT as an adapter for CDC tables only (AUTO CDC builds SCD1/SCD2 from DMS changes) | Out-of-order SCD2 handled by Databricks; removes our hardest MERGE code | Two engines to know; must still report into our run log | Pursue after a spike |
Auto Loader, in plain words
Auto Loader is a Spark streaming source for files. You point it at an S3 folder; it finds new files, remembers which ones it has read (in its checkpoint), handles schema changes, and puts unexpected fields in a rescue column. It finds new files in one of two ways:
- Listing the folder. Fine for small folders, but slow and costly for big ones.
- File events. S3 tells Databricks about each new file through queue notifications that Databricks sets up on the Unity Catalog external location. This is recommended.
Kinesis is for records; S3 events are for files. The sensor stream comes through Kinesis. DMS change files, LIMS results and partner files come through S3 and Auto Loader.
"A file lands in S3, call DLT for that file": four ways to react
| Option | How | Delay | Pros | Cons | Verdict |
|---|---|---|---|---|---|
| F1 · Auto Loader on a schedule | Job or pipeline every 15 min; Auto Loader (available-now) reads every new file since last run | ≤15 min + run | Simplest; files batched together; cheap | Runs even when nothing arrived (seconds of cost) | Pursue for regular feeds (DMS CDC) |
| F2 · Databricks file arrival trigger | Job trigger watches the external location or volume path; when new files appear it starts the job (which can run a DLT pipeline task); Auto Loader inside picks up all new files | ~1–2 min + run | Event-driven; no runs when nothing arrives; no AWS glue code | Checks about once a minute; a burst of files becomes one run (good) but runs queue; check trigger limits per workspace | Pursue for irregular drops (LIMS, partners) |
| F3 · S3 event → EventBridge → Lambda → Jobs API "run now" with the file path | One run per file, path passed as a parameter | ~1 min + startup | Per-file control | A pipeline start per file (slow, costly); one DLT update at a time, so files queue; passing paths bypasses Auto Loader's bookkeeping, so a missed event is a lost file; more code to own | Drop |
| F4 · Auto Loader continuous | Always-on stream on the folder | seconds to 1 min | Fastest for files | Always-on compute | Drop unless a T2 need appears |
Answer: do not hand a single file to DLT. Trigger the pipeline (F2), or run it on a schedule (F1), and let Auto Loader find every new file. That way a missed event never loses a file, and ten files arriving together cost one run, not ten.
Can S3 be used?
Yes, in four of the five roles below, but not as the conveyor belt itself. S3 is a warehouse, not a belt: it holds objects, it does not push them to readers, and every object written is a paid request.
| Use of S3 | How | Delay | Good for | Verdict |
|---|---|---|---|---|
| S3 as the event log between devices and processors | Devices write objects; processors list the bucket. | Seconds to minutes, unpredictable | Nothing in our design | Drop |
| S3 as micro-batch landing ("streaming over files") | Firehose buffers 60 s or 64 MB, writes one file; S3 event notification goes to SQS; Auto Loader picks it up. | 1–3 minutes | Sources that need minutes, not seconds; the AWS-only platform lane; CDC files from DMS | Pursue |
| S3 as Kappa's long memory | Bronze Delta (or Iceberg) tables on S3, kept forever. | n/a | Replay, history, ML training | Pursue |
| S3 for processor checkpoints and state | Flink and Spark save checkpoints to S3. | n/a | Recovery after restarts | Pursue (managed for us) |
| S3 Express One Zone as a fast log | Single-zone, low-latency bucket used as a queue. | Tens of milliseconds per request, still polled | Kafka-on-object-storage products use it internally; not something to build ourselves | Drop |
So: Kinesis (or Kafka) is the belt; S3 is where the belt empties into, and where history lives. For sources that only need minutes, you can skip the belt entirely and go straight to S3 files.
Streams that are not IoT
The same design takes any event source. What changes per source is the front door (R2) and whether it needs the belt (R3) at all. Rule: use the belt only when a reader needs it within minutes, or when several readers need the same events.
| Source kind | Tyre example | Event-native? | Front door | Needs the belt? | Recommended path | Verdict |
|---|---|---|---|---|---|---|
| Machine sensors | Curing presses, mixers | Yes | Plant gateway (SiteWise Edge on Greengrass) | Yes: two readers, seconds | Gateway → Kinesis → live lane + platform lane | Pursue |
| Connected products | Tyre sensors on fleet vehicles | Yes | IoT Core (MQTT, one certificate per vehicle), batched on the vehicle | Yes | IoT Core rule → Kinesis → both lanes | Pursue later, if in scope |
| Database changes (CDC) | SAP orders, MES production | No; CDC makes events | AWS DMS (full load + ongoing changes), or Debezium | Usually no | DMS → S3 Parquet files every 1–5 min → Auto Loader → bronze. Switch to DMS → Kinesis only if a live reader appears. | Pursue (files by default) |
| App events | Dealer portal clicks, warranty-claim app | Yes | API Gateway → Kinesis, or SDK from the backend | Only if someone acts on it live | API Gateway → Firehose → S3 (minutes); Kinesis if a live use appears | Pursue |
| SaaS webhooks | Salesforce, payment provider | Yes, but low volume and bursty | API Gateway → EventBridge (routes, archives) | No | EventBridge → Firehose → S3 → bronze | Pursue |
| Partner Kafka | Logistics partner's Kafka or Confluent topic | Yes | Their cluster | Theirs | Databricks reads their Kafka directly; or MSK Replicator if we need our own copy | Pursue when it appears |
| Files | LIMS results, partner CSVs | No | S3 bucket | No | S3 notifications → Auto Loader (already the batch design) | Pursue |
| Logs and metrics | App logs, CloudWatch metrics | Yes | CloudWatch subscription or metric stream | No | Firehose → S3 → ops tables | Pursue |
| Queues used as a log | SQS, SNS, EventBridge as the analytics source | — | — | — | A queue deletes a message once read: no replay, no second reader, no per-key order | Drop as a log |
Before you choose a path for a source, ask its owner the stream questions (group O) in the source system questionnaire: who acts and how fast, format and schema registry, key and order, event time and clocks, lateness and buffering, bursts, batching, identity, registry, heartbeat, other readers, history, commands back, personal data and test traffic.
Systems apart from IoT: yes, and they fit without a second architecture. Only machine sensors (and later connected tyres) need the belt and the live lane today. Everything else lands as files or through Firehose into the same bronze, under the same contracts and checks.
Binding options per part
The abstract parts (R1–R12) stay fixed; a binding picks the product for each part. Every table lists options from AWS, Databricks and open source with pros, cons and a verdict. In the Clean Data Engineering specs, the chosen rows become a binding file for that platform; another cloud is just another binding. One table per part. Prices are rough list prices for US regions (see Cost). "AWS-only" and "AWS + Databricks" share C1–C7; they differ from C8 onward.
C1 · Edge and plant gateway (R1)
| Option | Pros | Cons | Verdict |
|---|---|---|---|
| IoT SiteWise Edge on Greengrass, with stream manager exporting to Kinesis | OPC-UA and Modbus collectors built in; buffers to local disk when offline; batches; AWS manages updates | One more AWS runtime to learn; SiteWise cloud asset modelling is extra and priced per data point | Pursue (export straight to Kinesis; skip SiteWise cloud storage) |
| Third-party industrial software (Kepware, Ignition) publishing MQTT or to Kinesis | Plants often already run one | Licences; another vendor | Pursue if the plant already has it |
| Our own Python service on a small box | Full control | We own buffering, retries, updates, security patches | Drop |
C2 · Front door and identity (R2)
| Option | Pros | Cons | Cost note | Verdict |
|---|---|---|---|---|
| Gateway writes directly to Kinesis (IAM role via Greengrass token exchange) | Cheapest; fewest hops; batches up to 500 records per call | Identity is per gateway, not per machine | Kinesis charges only | Pursue for plants |
| IoT Core (MQTT, X.509 per device, rules engine → Kinesis) | Per-device identity, device shadows, commands back, millions of devices | Priced per message and per rule action | About $1 per million messages + $0.30 per million rule actions: ~$1,000 a month if every press sends alone | Pursue for connected tyres and commands; Drop for plant readings |
| API Gateway → Kinesis or Firehose | Apps and webhooks over HTTPS, with auth | Not for devices on MQTT | About $1 per million calls (HTTP API) | Pursue for apps |
| Databricks Zerobus Ingest (GA Feb 2026): send events straight into a Delta table | No belt to run; serverless | Only one reader (the table); no live lane from the same events; no Kinesis-style replay | Lakeflow Jobs serverless pricing | Drop for sensors; reconsider for a single-reader app stream |
C3 · Event log (R3)
| Option | Pros | Cons | Rough monthly, production scale | Verdict |
|---|---|---|---|---|
| Kinesis Data Streams, provisioned | Serverless to run; native to IoT Core, Firehose, Flink, Lambda and Databricks; 1 MB/s or 1,000 records/s in per shard | Shard sizing is ours; 7-day retention costs extra | ~$60–130 (2 shards, 7 days, 2 dedicated readers) | Pursue (recommended) |
| Kinesis on-demand | No sizing; good while volume is unknown | Higher per-GB price at steady load | ~$160 | Pursue for the first months, then switch |
| MSK provisioned (Kafka) | Kafka ecosystem: Debezium, Kafka Connect, partner topics, many readers, very long retention with tiered storage | Brokers to size, patch and tune; cross-zone traffic | ~$480 (3 × m7g.large + storage) | Pursue only if Kafka tools are needed |
| MSK Serverless | No brokers to manage | ~$550 a month floor before any data | ~$700 | Drop at this scale |
| Confluent Cloud | Best tooling; cloud-neutral | Highest price | $1k–3k | Drop unless multi-cloud is required |
| SQS, SNS, EventBridge | Messaging and routing | No replay, no second reader | — | Drop as the log |
C4 · Contract (R4)
| Option | Verdict |
|---|---|
Avro in Glue Schema Registry, backward-compatible changes only; fields reading_id, press_id (key), event_time, gateway_time, schema_version, tags; ODCS contract in the catalog | Pursue |
| Confluent Schema Registry | Pursue only with MSK and Confluent tools |
| JSON with no registry | Drop |
C5 · Live processor (R5)
| Option | Pros | Cons | Rough monthly | Verdict |
|---|---|---|---|---|
| Managed Service for Apache Flink | Event-time windows, watermarks, state, interval and temporal joins; exactly-once with checkpoints | Flink skills needed; always on | ~$250 (3 processing units) | Pursue |
| Lambda reading Kinesis | Stateless threshold alerts ("pressure above limit"); nearly free | No reliable windows or state | ~$10 | Pursue for simple limit alerts beside Flink |
| Databricks real-time mode (GA Mar 2026) | One engine (Spark) for both lanes; under 100 ms claimed | Classic compute only, fixed size, always on; check Kinesis source support | ~$700–1,000 for a small always-on cluster | Drop for now; revisit if the team wants one engine |
| Glue streaming (Spark) | Serverless Spark | Always on, weaker than Flink for the live lane, not needed for the platform lane | ~$650 | Drop |
| AWS IoT Events (detector models) | Was built for this | End of support 20 May 2026 | — | Drop (retired) |
C6 · Live store (R6)
| Option | Pros | Cons | Rough monthly | Verdict |
|---|---|---|---|---|
| DynamoDB (latest per press, plus 48-hour items with TTL) | Serverless; pay per write; DynamoDB Streams can push every change to the gateway | No ad-hoc slicing or SQL | ~$50 | Pursue (recommended) |
| Lakebase (Databricks Postgres) | SQL; already our metadata store; Databricks-native | Does not push changes, so the gateway must poll; always-on compute | ~$100–300 (check) | Pursue as the alternative |
| ClickHouse Cloud | Fast ad-hoc slicing over 48 hours | New vendor; does not push | Dev <$100; prod $500+ | Pursue only if live screens need slicing |
| ElastiCache (Valkey) | Fastest latest-value cache | No history; memory to size | ~$50–150 | Drop as primary |
| Timestream for LiveAnalytics | Time series | Closed to new customers since 20 Jun 2025 | — | Drop (closed) |
| Timestream for InfluxDB | Time series, AWS-managed | Instances to size; Influx skills | ~$100+ | Drop unless the team knows Influx |
C7 · Push and alerts (R7)
| Option | Verdict |
|---|---|
| API Gateway WebSocket: snapshot, then changes; at most 1 update a second per widget | Pursue |
| AppSync Events (managed pub/sub over WebSocket) | Pursue as a simpler alternative; test it in the physical step |
| SNS (SMS, email) and EventBridge (to ServiceNow, Teams) | Pursue for alerts |
C8 · Lake ingest: belt or files to bronze (R8)
| Option | Stack | Verdict |
|---|---|---|
Databricks Kinesis source in a streaming table, trigger every 15 minutes (our lander steps run in foreachBatch; dedupe on reading_id) | AWS + Databricks | Pursue (check that "available now" works with the Kinesis source on your runtime) |
| Firehose → S3 Parquet → Auto Loader | Both | Pursue as the fallback if the Kinesis source falls short |
| Firehose → S3 Tables (managed Iceberg) | AWS-only | Pursue for AWS-only |
| Zerobus | AWS + Databricks | Drop here (see C2) |
C9 · Lake tables and transforms (R9)
| Option | Stack | Verdict |
|---|---|---|
| Delta on S3 in Unity Catalog; Lakeflow streaming tables for bronze and silver; jobs for the stream-hour step and gold window recompute | AWS + Databricks | Pursue |
| S3 Tables (Iceberg); Glue ETL or EMR Serverless for silver and gold; Glue Data Catalog and Lake Formation | AWS-only | Pursue for AWS-only |
| Athena scheduled INSERTs for simple gold | AWS-only | Pursue for small, single-input aggregates |
C10 · Query and BI (R10)
| Option | Stack | Verdict |
|---|---|---|
| Databricks SQL, metric views, AI/BI dashboards, Genie | AWS + Databricks | Pursue |
| Athena + QuickSight | AWS-only | Pursue for AWS-only |
| Amazon Managed Grafana on the live store | Both | Pursue for engineers' time-series boards (optional) |
| Redshift Serverless | AWS-only | Drop at this scale unless already owned |
C11 · Commands back (R11) and C12 · Monitoring (R12)
| Option | Verdict |
|---|---|
| IoT Core jobs and device shadows for connected tyres; Greengrass deployments to change gateway settings | Pursue only when control is in scope |
| Monitoring: CloudWatch alarms on Kinesis reader lag (iterator age) and write throttling, Flink checkpoint time and back-pressure, DynamoDB throttles; a "silent stream" heartbeat per plant; a run recorder that logs every 15-minute run as it finishes (push, not poll) | Pursue |
AWS-only vs AWS + Databricks, side by side
| Role | AWS-only | AWS + Databricks |
|---|---|---|
| R1–R2 Plant to front door | Same: SiteWise Edge on Greengrass → Kinesis; IoT Core for connected tyres and commands; API Gateway for apps | |
| R3–R4 Belt and contract | Same: Kinesis provisioned, 7 days; Avro in Glue Schema Registry | |
| R5–R7 Live lane | Same: Managed Flink (+ Lambda for simple limits) → DynamoDB → API Gateway WebSocket; SNS and EventBridge for alerts | |
| R8 Lake ingest | Firehose → S3 Tables (Iceberg) | Lakeflow streaming table reads Kinesis every 15 min |
| R9 Tables and transforms | Iceberg; Glue ETL or EMR Serverless; Glue Catalog and Lake Formation | Delta; Lakeflow pipelines and jobs; Unity Catalog |
| R10 Use | Athena, QuickSight, SageMaker | Databricks SQL, AI/BI, Genie, Databricks ML |
| R12 Governance | Lake Formation, CloudTrail | Unity Catalog (lineage, tags, masks), system tables |
| Strength | Fewest vendors; lowest bill at small scale; everything pay-per-use | One governed home for batch and streams; same code for both; Genie and ML on the same tables; one governed platform for batch and streams |
| Weakness | Spark/Glue jobs and Iceberg upkeep are ours; weaker lineage and semantic layer; two engines (Flink plus Glue) to own | Two bills; Databricks cost on top for every 15-minute run |
| Verdict | Drop as the main design; keep as the "no Databricks" reference variant | Pursue (recommended when Databricks is the platform) |
The real split is by lane, not by vendor: the live lane is AWS-native in both stacks, because it is about seconds and pushing to screens. The platform lane goes wherever the rest of the platform lives. A Databricks-only design (Zerobus in, real-time mode for live, Lakebase as live store) is possible since 2026, but it gives up the shared belt and replay; I would not choose it today.
Performance and sizing
Three scales
| Demo world (DataTwin) | Production, assumed | Stretch: connected tyres | |
|---|---|---|---|
| What | The tyre world's sensor stream | 5 plants, 1,500 machines; one message every 5 s per machine carrying ~20 tags, ~1 KB | 200,000 vehicles; one 500-byte message a minute |
| Messages | 622,080 a day (~7 a second) | ~300 a second; ~778 million a month | ~3,300 a second; ~8.6 billion a month |
| Data in | tiny | ~0.3 MB/s; ~26 GB a day raw | ~1.7 MB/s; ~144 GB a day |
| Kinesis shards | 1 | 1 needed, 2 for headroom and parallel reads | 4 needed (record limit), 6 for headroom; or pack records to beat the 1,000/s limit |
| Flink | 1 unit | 2 units + 1 for orchestration | 4–8 units |
| Bronze growth | ~0.1 GB a day | ~4–6 GB a day compressed | ~25–35 GB a day compressed |
If your real plant numbers differ, only this table and the cost section change; the architecture does not.
Latency budget
| Hop | Live lane | Platform lane |
|---|---|---|
| Machine → gateway → Kinesis | 0.2–1 s (gateway batches every second) | same |
| Kinesis → reader | ~0.1 s with a dedicated reader (enhanced fan-out); ~1 s with shared polling | waits for the next 15-minute run |
| Processing | Limit alerts: immediate. 1-minute windows: window end + 5 s allowed lateness | 2–5 min per run |
| Store and push | ~0.1–0.3 s | gold hourly job at :15 |
| End to end | 1–3 s for alerts; up to ~65 s for window values | 15–20 min for silver; 15–75 min for gold |
Five performance rules
- Batch at the edge. Bundling readings per gateway per second cuts messages, requests and cost by 10–100 times.
- Key by press. Order is kept per press; the many presses spread across shards. Never key by plant: one hot shard.
- Give each lane its own reader (enhanced fan-out) so the 15-minute catch-up run cannot slow the live lane.
- Emit changes, not every reading, into the live store and to screens; at stretch scale this is the difference between $50 and $5,000 a month in DynamoDB writes.
- Bronze files sized well: 15-minute runs write fewer, larger files; liquid clustering and predictive optimization keep them tidy.
Cost (rough, per month, production scale)
| Item | AWS-only | AWS + Databricks | Note |
|---|---|---|---|
| Plant gateways (Greengrass fee) | ~$1 | ~$1 | Hardware not counted |
| Kinesis (2 shards, 7 days, 2 dedicated readers) | ~$120 | ~$120 | On-demand ~$160 |
| Managed Flink (3 units) | ~$250 | ~$250 | |
| Lambda limit alerts | ~$10 | ~$10 | |
| DynamoDB live store | ~$50 | ~$50 | Changes only, 1-minute windows |
| API Gateway WebSocket, SNS | ~$100 | ~$100 | ~50 live screens |
| Lake ingest | Firehose → S3 Tables ~$50 | Streaming table every 15 min ~$300–800 | Serverless DBUs; the biggest unknown; measure on your workload |
| Storage (S3, first year average) | ~$20 | ~$20 | Grows ~$2 a month each month |
| Silver, gold, BI | Glue/EMR + Athena + QuickSight ~$150–400 | Jobs + SQL warehouse ~$100–300 extra | Databricks warehouse is shared with batch |
| Total | ~$750–1,000 | ~$950–1,650 | Live lane alone is ~$530 in both |
| If every press goes through IoT Core instead of the gateway | +~$1,000 | +~$1,000 | Why C2 recommends gateway → Kinesis |
| Stretch scale (connected tyres) | IoT Core ~$9k unbatched, ~$1k with 10 readings per message; Kinesis ~$300; Flink ~$450–750; DynamoDB $50 or $5k depending on "changes only" | Per-message prices dominate: batch on the vehicle | |
What drives the bill
- Per-message prices (IoT Core, S3 PUTs, DynamoDB writes) dominate at scale: batch at the edge and emit only changes.
- Always-on compute (Flink): keep the live lane to what someone acts on within minutes.
- The 15-minute cadence sets the Databricks part: 30 minutes roughly halves it; 5 minutes roughly triples it.
- Retention on the belt: 7 days is cheap; history belongs in bronze.
Replays
A replay is reading old data again on purpose: after a fault, a bug fix or a logic change. Because Kinesis keeps 7 days and bronze keeps everything, every layer can be rebuilt. Rules: every replay is a logged run with a reason; replays use separate compute so live service levels are not hurt; replays never re-send alerts.
| Replay | When | How | Reads from | Why it is safe |
|---|---|---|---|---|
| Live lane restart | Flink crash or deploy | Restart from the last checkpoint (automatic) | Kinesis | Exactly-once state; upserts by press and minute |
| Live lane rebuild | New window or limit logic | Start Flink from a timestamp up to 7 days back | Kinesis | Writes are upserts; alerts marked "replay" and suppressed |
| Re-ingest bronze | Lander bug, lost run | New checkpoint starting at a Kinesis timestamp | Kinesis (≤7 days) | Silver deduplicates on reading_id |
| Rebuild silver | Rule change, SCD type change | Reset that table's checkpoint and re-read bronze for the range | Bronze (forever) | Bronze is never changed; restatement logged |
| Recompute gold windows | Late data, dimension fix, logic change | Window recompute for listed hours or days | Silver | Whole windows replaced; restatements visible in the SLA report |
| Gateway resend | Plant back online | Gateway sends its disk buffer | Gateway | Duplicates removed by reading_id; late data counted |
| History backfill | Years of historian data | Separate bulk job (ETL design) | Historian export on S3 | Never through the stream |
Insurance option: also copy the raw Kinesis stream to S3 through Firehose (~$25 a month). Then a reader that falls more than 7 days behind still loses nothing. Verdict: Pursue, because it is cheap and closes the only real data-loss gap.
Error handling
Principle: errors are rows, not log lines. Every rejected record lands in a table with run id, stage, reason and a pointer to the raw payload, so the SLA report can count it. One bad record never stops a pipeline.
| What goes wrong | Where | What happens | Shows up as |
|---|---|---|---|
| Plant network down | Gateway | Buffers to disk (hours), sends on reconnect | "Silent plant" alert after 5 min; late-data count when it returns |
| Writes throttled | Kinesis | SDK retries with back-off | Throttle alarm; add a shard |
| Bad message (cannot decode, unknown schema) | Live lane / platform lane | Live: side output to a dead-letter folder. Platform: kept raw in bronze, sent to quarantine in silver | Quarantine by reason |
| Contract or quality failure (missing field, impossible value, stuck sensor) | Silver | Row to quarantine with the rule that failed; the rest continue | DQ counts; alert above a threshold (e.g. 1%) |
| Duplicates | Silver | Removed on reading_id | Duplicate count per hop |
| Later than the watermark | Live lane / gold | Live: left out of live views. Gold: accepted, window recomputed | Late count by lateness bucket; restatements |
| Processor crash | Flink / Spark job | Restart from checkpoint; job retries 3 times | Run log; alert after retries |
| A 15-minute run fails or is skipped | Platform lane | Next run catches up (available-now reads everything new) | Alert after 2 missed runs; SLA breach marked |
| Reader falls behind | Any Kinesis reader | Keeps going; risk only near 7 days | Lag alarm at 1 h (live) and 24 h (platform); Firehose copy is the backstop |
| Sink failure (DynamoDB throttle, stale WebSocket) | Live lane | Retry; drop dead connections | Push error rate |
| Bad deploy | Any | Roll back, then replay the affected range | Replay run with reason |
The end-to-end SLA report
One report answers four questions every day: did we keep the promises, where did the time go, did anything get lost, and what went wrong.
Timestamps every reading carries
| Stamp | Set by | Meaning |
|---|---|---|
t0 event_time | Machine | When it happened |
t1 gateway_time | Plant gateway | When the plant sent it (also catches clock skew) |
t2 kinesis_arrival | Kinesis | When it reached AWS |
t3 live_emitted_at | Flink | When the live lane pushed it (alerts and windows) |
t4 bronze_at + run id | Lander | When it landed in bronze |
t5 silver_at + run id | Silver builder | When it became clean data |
t6 gold_complete_at | Gold builder | When its press-hour was marked complete |
How it is built
- Flink writes a one-minute summary per plant: counts, p50/p95/p99 of t3 − t0 and t3 − t2, late and bad counts. The summary goes through Firehose into an ops table.
- Each 15-minute run records counts and delay percentiles per hop through the run recorder.
- Two ops tables hold the result:
ops.stream_sla_minute: stream, plant, minute, hop, count, p50, p95, p99, late, duplicate and quarantined counts.ops.gold_hour_completeness: press-hour, complete at, inputs late, restated.
- Metric views sit on top, feeding an AI/BI dashboard. Ops agents read them through tools, never through raw queries.
The five pages of the report
1 · Scorecard
Each promise with % met for yesterday, 7 days and 30 days, plus breach count. Example: "Alerts ≤5 s: 99.6% (target 99%)".2 · Where the time goes
A waterfall of p95 delay per hop (machine → gateway → Kinesis → live, and → bronze → silver → gold) by plant.3 · Nothing lost
Counts per hop side by side: gateway sent, Kinesis received, bronze, silver (+ duplicates + quarantined), gold minutes covered. The difference must be zero or explained.4 · Errors and lateness
Quarantine by reason, late readings by bucket (<5 min, <1 h, <1 day, older), restated gold hours.5 · Incidents
Silent plants, missed runs, lag alarms and replays, each with start, end, cause and which promise it broke (event-time vs arrival-time).Made-up numbers to show the shape: most delay is waiting for the next run, which is a cadence choice, not a performance problem.
Non-functional concerns
| Concern | What it means here | Target | How the design meets it |
|---|---|---|---|
| Freshness | How far behind each screen or table may be | Live: alerts ≤5 s, windows ≤90 s. Silver ≤20 min. Gold ≤75 min. | Two lanes; freshness checked per table; a late run alerts |
| Throughput and growth | Handle 10× without redesign | 3,000 messages a second | Add shards and Flink units; same design |
| Ordering | Readings of one press in order | Per press | Partition key press_id; windows use event time anyway |
| Delivery and correctness | Nothing lost, nothing counted twice | Exactly-once results | At-least-once delivery + dedupe on reading_id + idempotent MERGE; Flink checkpoints |
| Late and out-of-order data | Gateway offline for a while | Live: 5 s lateness, later readings go to the platform lane only. Gold: 2-day lookback | Watermarks; window recompute re-checks 2 days |
| Replay and retention | Recover from a bad deploy or outage | 7 days on the belt; forever in bronze | Kinesis retention; bronze append-only, never deleted |
| Availability | Keep working when a part fails | Live lane 99.9%; no data loss while the gateway buffers | Gateway disk buffer; Kinesis, Flink and DynamoDB are multi-zone |
| Disaster recovery | A whole region down | Platform lane: hours; live lane: best effort | Gateways buffer for hours; bronze replicated to a DR region |
| Security | Only known senders and readers | Per-gateway identity; least privilege; encrypted everywhere | IAM role per gateway, X.509 per vehicle; KMS keys; VPC endpoints; no public brokers |
| Privacy | Personal data in streams | No PII in bronze outside restricted columns | Sensors carry none; dealer app events and vehicle location are PII: masked in silver, tagged in the catalog |
| Schema evolution | Firmware adds a tag | No reader breaks | Registry allows backward-compatible changes only; unknown fields go to a rescue column |
| Data quality | Impossible values, stuck sensors, clock skew | Bad rows visible, never silently dropped | Range and staleness checks in silver; quarantine table; gateway time vs press time check |
| Observability | Know when it is slow or silent | Alert within 5 minutes of a silent plant | Lag and throttle alarms; heartbeat per plant; run recorder; one alert per incident |
| Cost control | No surprise bills | Monthly budget with alerts | Edge batching; changes only; tags per data product for cost reports |
| Operability | Small team can run it | No self-managed clusters | All managed or serverless; Flink is the one skill to grow |
| Portability | Reuse on Azure, Fabric, GCP | Only adapters change | Stream port: Kinesis adapter today; Event Hubs and Pub/Sub adapters later; contracts and checks stay |
| Testability | Prove it before a client sees it | Planted incidents caught | DataTwin sensor stream with late, duplicate and silent-press incidents; replay tests; port contract tables on moto |
Decisions to make, with recommendations
| # | Question | Options | My recommendation |
|---|---|---|---|
| S1 | Which stack? | AWS-only · AWS + Databricks · Databricks-only (Zerobus + real-time mode) | AWS + Databricks, with an AWS-native live lane. Keep AWS-only as a documented variant. |
| S2 | How do plant readings enter? | Gateway → Kinesis directly · every machine through IoT Core | Gateway → Kinesis; IoT Core only for connected tyres and commands. Saves ~$1,000 a month. |
| S3 | Which belt? | Kinesis · MSK | Kinesis (on-demand first month, then provisioned). MSK only if Kafka tools or partner topics arrive. |
| S4 | Which live store? | DynamoDB · Lakebase · ClickHouse Cloud | DynamoDB: it pushes changes and costs ~$50. Lakebase if you want SQL and one fewer service. |
| S5 | How do SAP and MES changes arrive? | DMS → S3 files · DMS → Kinesis · Debezium on MSK | DMS → S3 files into the existing batch path; no belt until a live reader needs it. |
| S7 | DLT or not? | Non-DLT (our Structured Streaming jobs) · DLT for everything · DLT only for CDC tables | Non-DLT as default; DLT for CDC→SCD tables after a spike. |
| S8 | How do file drops start processing? | Schedule · Databricks file arrival trigger · S3 event → Lambda → run per file | Schedule for regular feeds, file arrival trigger for irregular ones; never one run per file. |
| S9 | Live UI route? | Direct push · live store + push · ClickHouse · Lakebase | Live store (DynamoDB) + push. ClickHouse only if screens need slicing. |
| S10 | Firehose raw copy of Kinesis to S3? | Yes · no | Yes (~$25 a month); removes the 7-day loss risk. |
| S6 | Are connected tyres in scope? | Now · later · never | Later. The design already fits them; it only adds IoT Core and more shards. |
Notes and sources
- AWS IoT Events end of support, 20 May 2026: AWS docs. AWS IoT Analytics was also discontinued (15 Dec 2025, per a third-party copy of the AWS notice; confirm on AWS).
- Timestream for LiveAnalytics closed to new customers from 20 Jun 2025: AWS docs.
- Zerobus Ingest generally available, 23 Feb 2026: Databricks blog, docs.
- Real-time mode for Structured Streaming generally available (March 2026): Databricks blog, docs.
- Related on this site: Real-time analytics, drawn simply, Medallion architecture, Data platform on AWS + Databricks, DataTwin.
FAQ
Questions people ask
Should a 1 to 5 second streaming lane call Databricks?
No. Keep it light and inside the streaming layer: validate, deduplicate, limit checks, one-minute windows, and small reference data copied in on a schedule. Databricks reads the raw stream separately and builds the numbers of record.
Where do SCD type 1 and type 2 happen in a streaming lakehouse?
In silver, by dataset kind: master data as SCD type 2, transactions as type 1, sensor events append only. Gold builds dimensions from silver's history and joins facts as of event time.
DLT or plain Structured Streaming?
Plain Structured Streaming in jobs as the default, because it is portable and fully testable; DLT's AUTO CDC is worth it for tables built from database changes, where out-of-order SCD2 is hardest to write by hand.
Can S3 be used as the stream?
Not as the belt itself: one object per event is slow and costs a request each. S3 works well as minute-level landing for Auto Loader and as the long-term history that makes Kappa replays possible.
Should I trigger a DLT pipeline for each file that lands in S3?
No. Use a file arrival trigger or a schedule and let Auto Loader pick up every new file, so a missed event never loses a file and ten files cost one run.
Related
Where it connects
Real-time analytics, drawn simply · Medallion architecture · Data platform on AWS + Databricks · All data engineering