Home › Data engineering › Streaming

Data engineering · free guide · one page

Streaming architecture on AWS and Databricks

IoT and ordinary sources, seconds and minutes, AWS alone or with Databricks: the architecture as roles first, then the options for every part with pros, cons and a verdict.

How live data gets from factory machines (and from databases, apps and files) to screens, alerts and reports. The guide starts with the architecture as roles with no product names, then gives options on AWS alone and on AWS with Databricks for every role, each with pros, cons and a verdict. A tyre maker is the worked example throughout.

All options in one table: streaming options at a glance. Try it: the streaming simulator animates this design for each binding and route, and three lanes together runs the fast, near-real-time and batch lanes over a simulated day. Building the fast lane inside Databricks instead: real-time mode.

The worked example

A tyre maker with five plants. Curing presses, mixers and extruders report temperature, pressure and cycle state every few seconds. Supervisors want alerts and live boards within seconds. Planners and quality teams want hourly numbers they can trust. SAP, MES, a lab system (LIMS) and a dealer portal also produce data that has to join with the sensor data.

Design rules used throughout

  • One event stream feeds two lanes: a live lane (seconds, for acting now) and a platform lane (minutes, the number of record).
  • The platform lane does not run continuously: streaming tables run once every 15 minutes and stop.
  • The live lane never writes gold. Gold is the number of record.
  • One contract per stream: Avro in a schema registry; key = press_id; every record carries reading_id and event time. The stream keeps 7 days.
  • Gold facts are built by recomputing changed time windows, with as-of joins to history dimensions, not by joining inside the stream.

What a stream is, in plain words

A batch is a box of data delivered at a time: "here are yesterday's orders". A stream is a conveyor belt: small messages keep arriving, one after another, and never "finish". Streaming is a way of delivering data, not a kind of data. The same orders can arrive as a nightly file or as one message per order.

Three things make streams harder than batches:

  1. Time has two clocks. Event time is when it happened at the press. Arrival time is when the message reached us. A gateway that was offline for 20 minutes delivers old readings late, so reports must group by event time.
  2. Nothing ever "ends". To say "the 10:00–11:00 average is final", you need a rule for how long to wait for stragglers. That rule is a watermark, for example "wait 10 minutes after the hour".
  3. Messages can arrive twice. Senders retry when unsure. Every reading carries a unique reading_id so duplicates can be removed and totals never double.

Where a tyre company gets streams from

Machines (IoT)

Curing presses, mixers and extruders report temperature, pressure and cycle state every few seconds.

Connected tyres (IoT, maybe later)

Sensors inside tyres on fleet vehicles report pressure and temperature. Millions of small senders.

Database changes

SAP orders and MES production records. These are not events by nature; change capture (CDC) turns every insert or update into an event.

App events

Dealer portal clicks, warranty-claim app submissions.

Webhooks

Salesforce or a payment provider calls our URL when something changes.

Files and logs

LIMS lab results, partner files, application logs. Files arrive in bursts but can be read "as a stream" of files.

The key question for every source is the same: who must act on this data, and how fast? If a supervisor must stop a press within a minute, it needs the live lane. If a planner reads it tomorrow, a 15-minute or daily load is cheaper and simpler.

Conceptual architecture (no products)

Twelve roles. Every physical design later is just a choice of product for each role.

R1 Machines, sensorsvia a plant gateway Connected productstyres on vehicles (later) Apps and webhooksdealer portal, SaaS Database changes (CDC)Files (LIMS, partners) R2 Front doorwho are you?one identity perdevice or appbatch at the edgeto cut cost R3 Event logordered per keykept for daysmany readersreplay after a fault R4 Contract:schema registry,key, event time,reading id LIVE LANE · seconds · for acting now (never writes gold) R5 Liveprocessorwindows, limits R6 Live storelatest values,last 48 hours R7 Push andalertssnapshot, changes Live boards,phones1–5 s behind PLATFORM LANE · minutes · the number of record R8 Lakeingestevery 15 min,log or files tobronze R9 Bronzekept forever:the long memoryreplay source R9 Silver,golddedupe, check,joins, windowrecompute R10 BI, AI,ML15–75 minbehind R12 Across everything: contracts and schema evolution · identity and encryption · monitoring (lag, freshness, a silent stream) · data quality and quarantine · catalog and lineage · cost · tests with the DataTwin world R11 commands back to devices (optional, dashed) default: as files, no stream needed

Solid arrows: the stream. Dashed arrows: sources that usually do not need a stream. Dotted: optional commands back to devices.

RoleJob, in one sentenceWhy it must exist
R1 Edge and gatewayCollects machine signals on the factory floor, buffers them when the network is down, and sends them in batches.Factory networks drop; sending each reading alone is slow and expensive.
R2 Front doorChecks who is sending, with one identity per device or app, and accepts messages.One stolen device can be switched off without touching the rest.
R3 Event logKeeps messages in order per key (press) for days, so many readers can read at their own pace and replay.Separates senders from readers; makes faults recoverable.
R4 ContractDefines the message shape and versions it in a registry.A firmware update must not silently break every reader.
R5 Live processorComputes 1-minute windows, limit and drift checks, and emits only changes.Someone must act within minutes.
R6 Live storeHolds latest values and the last 48 hours for live screens.A new screen needs a snapshot before changes start flowing.
R7 Push and alertsSends a snapshot, then changes, to screens; sends alerts to people and tickets.Screens should not keep asking "anything new?".
R8 Lake ingestCopies the log, or files, into bronze tables every 15 minutes.Reports and ML need history, not a belt.
R9 Lake tablesBronze keeps everything forever; silver cleans; gold joins and aggregates.One number of record, rebuildable from bronze.
R10 UseDashboards, Genie, ML features, APIs.Where value shows up.
R11 Commands backChanges settings on devices, such as reporting rate.Optional; only if the business needs control, not just watching.
R12 Cross-cuttingSecurity, monitoring, quality, catalog, cost, tests.Same concerns as batch, plus lag and "silent stream".

Seconds or minutes: two service levels, end to end

"Real time" means two different promises. Each needs its own path and has its own cost. End to end means from the moment something happens at the press to the moment a person can see it.

TierPromise (end to end)Who actsPathRough cost a month (production)Verdict
T0 · under a secondMachine stops itselfThe machinePlant control system (PLC, SCADA), not the cloud—Drop for the cloud
T1 · seconds99% of alerts on screen within 5 s; window values within 90 sOperator, supervisorLive lane: Kinesis → Flink (Lambda for simple limits) → DynamoDB → WebSocket push~$530Pursue
T2 · about a minuteData in a table within 1–3 minRarely anyoneAlways-on streaming table, or Auto Loader in continuous mode+$700–1,500 (always on)Drop unless a named user needs it
T3 · minutes99% of readings in silver within 20 min of arrival; gold hour complete within 75 min of hour endShift lead, planner, qualityPlatform lane: streaming tables trigger-once every 15 min → silver → hourly gold~$420–1,120Pursue
T4 · dailyReady by 06:00Management, financeBatchpart of batchPursue (existing)

End-to-end total for T1 + T3: about $950–1,650 a month at production scale. The hop-by-hop budget is in Performance and sizing. The report that proves the promises is described under SLA report.

Measure two ways. Report delay from event time (what the user feels) and from arrival time (what the platform controls). A gateway that was offline for an hour makes event-time delay look terrible, but that is a plant network problem, not a platform breach. The SLA report keeps both and says which one broke.

Three speeds, two lanes: where they meet

There are three speeds: seconds (1–5 s), near real time (15–75 min) and batch (daily). There are only two lanes. Near real time and batch are the same pipeline (same code, same tables) run at a different cadence. The seconds lane is a separate, light, disposable lane.

Kinesisraw events, oncesplit point (1) SECONDS LANE · AWS only · no Databricks call · 48 h, disposable light transformsvalidate, dedupe, limits 1-min windowsper press, changes only live store + pushtiles, alerts PLATFORM LANE · Databricks · near real time and batch = same pipeline bronzeevery 15 min silverSCD, quality gold (3): streams meetbatch dims + batch facts, window recompute batch sources: SAP, MES, LIMSCDC files daily or every few minutes Screen (4)live tile: "live,not final"beside gold numbers:"number of record"shown side by side,never added together (2) reference data flows UP:limits, press master, recipes alerts raised = new facts → platform

(1) One raw stream, read separately by each lane. (2) Reference data is copied from the platform lane into the seconds lane on a schedule. (3) Near real time and batch meet in gold. (4) The screen shows live and final numbers side by side, labelled.

Four questions people ask

QuestionAnswerVerdict
Does the 1–5 s lane only do light transforms, never call Databricks?Yes. Allowed: validate against the contract, short-window dedupe, unit conversion, limit checks, 1-minute windows per press, change detection, lookup of small reference data held inside the lane. Not allowed: calls to Databricks or any warehouse at runtime, multi-source joins, SCD2 as-of joins, history beyond 48 hours. A call to Databricks would add seconds and tie the lane's uptime to the platform's.Pursue
Should the fast lane send its data to the slow lane?Not its readings. The platform lane reads the raw stream from Kinesis itself, so a bug in the fast lane can never corrupt the number of record. The fast lane sends only what it creates: alerts raised and acknowledged (a new fact, "alert events"), and its one-minute SLA summaries. Both go via Firehose to bronze.Pursue (raw from Kinesis; alerts as their own dataset)
(the alternative) Platform lane reads the fast lane's outputCouples the two lanes; a Flink bug or a dropped late reading flows into gold.Drop
Do the speeds meet?In four places, and only these:
  1. At the source: the same raw event on Kinesis.
  2. Reference data upward: limits, the press master and recipes from silver or gold are published every 15 minutes or daily into the seconds lane (DynamoDB, or Flink broadcast state). The fast lane reads its own copy and never calls the platform.
  3. In gold: near-real-time facts join batch dimensions and batch facts.
  4. On the screen: live tiles beside gold numbers, labelled, never added together.
Pursue
Is near real time joined only with near real time, or with batch too?With batch too. Most joins in gold are near real time to batch: sensor hours joined to the SAP material dimension (daily) and to LIMS scrap (hours late). The rule: a gold row is only as fresh as its slowest input. Each hour carries a "complete" flag that turns true only when every input's watermark has passed it. Until then the row shows with unknown members (−1), and window recompute fixes it when the batch input lands. Each fact declares its freshness from its slowest input.Pursue
Can one dataset exist in all three speeds?Yes, as one dataset with one contract and one owner, served at different speeds. Press readings: seconds = latest values and 1-minute windows for 48 hours (disposable); near real time = silver readings and press-hour gold; batch = daily OEE, monthly reports, ML training from the same gold. Only one copy is the record (bronze, silver and gold); the live copy expires. Expect small differences between live and gold (late data, duplicates); the SLA report shows that drift and alerts above a set level (say 2%).Pursue (one dataset, many speeds, one record)

Kappa, Lambda, and where we sit

LambdaKappaOurs: Kappa through bronze sourcebatch codespeed codemerge views Two codebases compute the same number.They disagree; someone has to merge them. sourcelog, kept long(the truth)one stream job One codebase. New logic? Replay the log.Needs a log that keeps months or years. Kinesis, 7 daysbronze, forever(the long memory)silver → gold live laneacts, never writes goldscreens, alerts Same code reads stream or replay from bronze.Two lanes give two products, not two answers. replay

Lambda: two paths, one answer, merged. Kappa: one path, history rebuilt by replay. Ours: Kappa where bronze plays the long log; the live lane is a different product.

How Kappa fits, in four points

  1. Kappa's idea: everything is a stream, there is one code path, and to fix or change logic you replay the log from the start. No separate batch code.
  2. The catch: Kappa needs a log that keeps everything. Kinesis can keep up to 365 days and Kafka can keep forever with tiered storage, but paying a log to hold years of readings is expensive and awkward to query.
  3. Our answer: the log keeps 7 days (for outages); bronze keeps everything forever on S3, cheaply. Bronze is Kappa's long log. Spark reads bronze with the same code as a stream or as a full replay, so there is still one code path.
  4. Why our two lanes are not Lambda: Lambda computes the same number twice and merges. Our live lane computes "act now" signals that are never reported as the number of record; gold comes only from the platform lane. So the two lanes never disagree about a reported number.

Where Kappa does not fit, and what we do instead

CaseWhy pure Kappa strugglesWhat we doVerdict
Sensor readings to bronze and silverFits well: event-native, append-only, keyed by press.Stream (trigger-once, 15 min); replay from bronze when logic changes.Pursue Kappa
Gold facts that join several inputs (OEE, scrap per press-hour)Inputs arrive at different speeds (LIMS results hours later); a pure stream join would hold state for hours and still miss corrections.Window recompute: find changed hours, rebuild them fully.Pursue window recompute
SAP and MES tablesNot event-native. Making every table a stream adds CDC tooling and a log for data nobody needs in seconds.CDC lands as files every few minutes; the same incremental pipeline reads them.Drop Kappa here
Live lane after a logic changeFlink state must be rebuilt; Kinesis has only 7 days.Fine: the live lane only needs 48 hours. Replay 2 days from Kinesis.Pursue (7 days is enough)

One-line summary: Kappa for event-native sources, with bronze as the long log; incremental batch for databases and files; window recompute for multi-input gold. This is "one medallion, two speeds", with Kappa made explicit.

What happens in bronze, silver and gold

The same three layers as batch, run incrementally. Each layer has one job, so faults are easy to find.

LayerSensor stream (event)Database changes (CDC from SAP, MES)Never do here
Bronze
keep everything
Append every record as it came, plus envelope: shard, sequence number, Kinesis arrival time, ingest run id, schema version. Payload kept raw (or as VARIANT). Clustered on ingest date. Kept forever.Append every change row from DMS files (insert, update, delete, commit time, change sequence). Kept forever.Deduplicate, filter, fix values
Silver
one clean truth per source
The 13 standard stages: decode by schema version, types, event time in UTC plus plant local, quality checks (range, stuck sensor, clock skew), deduplicate on reading_id, conform press_id, append. Bad rows go to quarantine with a reason.Same stages, then history: master data (press, material, dealer) as SCD type 2; transactions (orders, claims) as SCD type 1 (type 2 where the business needs the past state). Changes applied in source order, not arrival order.Join across sources (silver is source-aligned)
Gold
business numbers
Stream-hour job: press-minute and press-hour aggregates; facts such as OEE and scrap per press-hour by window recompute; each hour carries a "complete" flag.Conformed dimensions with surrogate keys built from silver SCD2 (−1 unknown, −2 not applicable); facts join dimensions as of event time.Read the live store; inner-join to dimensions

Where SCD type 1 and type 2 happen

  • In silver. The dataset kind sets the history type. Master data is SCD2, transactions are SCD1, events are append-only, and snapshots are kept as snapshots. Only the listed tracked columns open a new version.
  • Sensor readings get no SCD. They are facts that never change. A "correction" is a new reading, and deduplication keeps the first.
  • Gold does not redo SCD. It builds dimensions from silver's SCD2 history, adds surrogate keys, and joins facts as of event time.
  • The streaming catch for SCD2: changes can arrive out of order, and a late change must be slotted into the middle of history. Two ways to do it:
    • DLT's AUTO CDC (formerly APPLY CHANGES) handles this with a sequence column.
    • Plain Spark needs our own MERGE in foreachBatch that re-cuts the valid-from and valid-to dates. This is one reason to consider DLT for CDC tables (see Databricks choices).
  • Changing a table from type 1 to type 2 is a replay from bronze, which still holds every change, so the past gets history too.

Joining data in a stream

In batch, a join sees both tables complete. In a stream, one or both sides are still arriving, so every join has to answer: how long do I remember the other side, and what if it is late? Memory held between messages is called state. State costs RAM, and lost state means a wrong answer.

Press cyclesBarcode scansevent time match window: scan within 10 min of cycle start watermark: "nothing older than this will come" state left of this line is dropped scan with no cycle: unmatched waits in state

Example: link each curing cycle to the tyre serial scanned at the press. The processor keeps unmatched cycles until the watermark passes, then gives up and routes them to an "unmatched" output.

Five kinds of join, with tyre examples

Join kindTyre exampleHow it worksGotchaEngine supportWhere we do itVerdict
Stream to static (lookup)Reading + press master (plant, line, mould)Each message looks up the current row of a small table.Uses the dimension as it is now. A late or changed dimension row does not fix messages already joined. Missing key: keep the row, mark it unknown.Spark: stream-static join, table re-read each micro-batch. Flink: lookup join (JDBC, DynamoDB) or broadcast state.Live lane only, to label screens.Pursue (live only)
Stream to stream, time-boundedPress cycle start + barcode scan at the press (picture above)Both sides wait in state until a match or the watermark passes.State size = message rate × window. No watermark means state grows forever. Unmatched rows must go somewhere visible.Spark: inner and outer joins with watermarks on both sides. Flink: interval join.Silver, when the link is needed within the hour; otherwise gold.Pursue, with a declared window
As-of (temporal) joinReading at 10:04 + the curing recipe valid at 10:04 (SCD2)Picks the dimension version valid at the event time, not today's.Needs the dimension's history. Late recipe changes mean re-joining.Flink: temporal table join, native. Spark streaming: not native; done in batch over Delta.Gold, by window recompute.Pursue in gold
Window first, then joinPer press-hour sensor averages + scrap from LIMS, which arrives hours laterAggregate each side to the same grain (press-hour), then join the aggregates.One side is hours late, so the result changes after it was first shown.Any engine; easiest in batch.Gold: the stream-hour job, then window recompute.Pursue in gold
Stream to changelog (CDC) tableDealer app events + SAP customer table kept up to date by CDCBoth sides are changing; the join must re-emit when either side changes.Unbounded state unless a time-to-live is set; results get retracted and re-emitted, which confuses most sinks.Flink: regular join on changelogs. Spark: not as a stream; use MERGE in batch.Gold, batch.Drop in stream
The join rule: join inside the stream only when the answer is needed in seconds and the other side is small or time-bounded. Otherwise land both sides in bronze and silver, and join in gold by window recompute. This keeps state small and makes corrections automatic.

Five join hazards and what handles each

  • Late dimension (a new press appears in readings before it appears in the master table): keep the reading with an unknown key (−1). Window recompute fixes it when the master row lands.
  • Fan-out (duplicate key on the "one" side multiplies rows): count duplicate keys on the one side separately; one net count hides mistakes that cancel out.
  • Clock skew (a press clock that is 3 minutes wrong): the gateway stamps its own time too; when the two differ by more than a limit, the reading is flagged.
  • State loss (a processor restart without its checkpoint): checkpoints on S3; the live lane can rebuild from 2 days of Kinesis.
  • Re-emitted results: sinks are idempotent (MERGE on key, or upsert by press and minute), so a retry never double counts.

Getting to the UI: direct, via a fast layer, or via gold

The question is not "how fast can data reach a screen" but "what does a screen need when it opens": the latest value, a chart of the last hours, or the ability to slice and filter.

RoutePathDelayProsConsRough costVerdict
A · Direct push, no storeFlink → Lambda → API Gateway WebSocket → screen~1 sFewest partsA screen that opens or reconnects sees nothing until the next change; no chart of the last hour~$100Drop as the only route
B · Small live store + pushFlink → DynamoDB → DynamoDB Streams → Lambda → WebSocket; a new screen reads the snapshot from DynamoDB first1–3 sSnapshot then changes; serverless; cheap; 48-hour charts from the same tableNo ad-hoc slicing; key design matters~$150Pursue (recommended)
C · Fast analytic layer (ClickHouse)Kinesis → ClickHouse (ClickPipes) or Flink → ClickHouse; the API polls every second and pushes only differences; Grafana can query it directly2–5 sSQL, slicing and filters over days, many users at onceNew vendor; does not push; a third copy with its own numbers to keep consistent with goldDev <$100; production $500–3kPursue only if live screens need slicing beyond 48 hours
D · Lakebase as the live storeFlink → Lakebase (Postgres); the app reads with SQL and polls2–5 sSQL; Databricks-native; already our metadata storeNo push; always-on compute; Flink to Postgres sink to own~$100–300Pursue as the Databricks-leaning alternative to B
E · Via goldGold → metric views → AI/BI dashboards (auto-refresh), Genie; or gold synced to Lakebase for an app15–75 minThe number of record; governed; no extra copyNot livepart of the platformPursue for reports; Drop for live

Rule: live screens say "live, not final". A number that goes into a report always comes from gold, whatever the route.

Databricks choices: DLT or not, Spark Streaming, Auto Loader, file arrival

Is Spark Structured Streaming worth it when we only run every 15 minutes?

Yes, for the platform lane. Even when it runs once and stops (trigger "available now"), a Spark streaming read gives us the bookkeeping for free: which Kinesis records or which files are already done (the checkpoint), exactly-once writes into Delta, and catch-up after a missed run. Doing the same with plain batch reads means writing our own "where did I stop" logic for every source.

UseProsConsVerdict
Structured Streaming, trigger available-now, every 15 min (platform lane)Checkpoints, exactly-once to Delta, catch-up, same code later if we go faster; pay only for run minutes on serverlessCheckpoint is tied to the query; changing a query shape can need a new checkpoint (a replay)Pursue
Structured Streaming always on (micro-batch, seconds)One engine for both lanesAlways-on cluster; weaker event-time state than FlinkDrop for the live lane
Real-time mode (GA 2026, under 100 ms claimed)Spark at Flink-like latencyFixed classic cluster, always on; check Kinesis supportDrop now; revisit to retire Flink
Old Spark Streaming (DStreams)—DeprecatedDrop
Plain batch with our own high-water markSimple to readWe rebuild what checkpoints already do; easy to miss late filesDrop for streams and files

DLT (Lakeflow Declarative Pipelines) or not?

DLT means you declare tables (streaming tables, materialized views) and their rules, and Databricks runs them: checkpoints, retries, ordering, data quality expectations, and AUTO CDC for SCD. The alternative ("non-DLT") is our own PySpark code using Structured Streaming inside Lakeflow Jobs. The reference ETL on this site is built that way: the 13 silver stages as pure PySpark functions, testable on a laptop.

OptionProsConsVerdict
Non-DLT: Structured Streaming in Lakeflow Jobs (our 13 stages in foreachBatch)Full control of every step; runs locally and in CI; portable to Fabric, EMR, Glue, GCP (only adapters change); our run recorder sees every stepWe own checkpoints, SCD2 MERGE with late changes, retries and DQ wiringPursue as the default engine
DLT for everythingLess code; managed checkpoints and retries; expectations; lineage; serverless; good event logDatabricks-only (breaks portability); tables owned by the pipeline; a full refresh can wipe a streaming table (must lock it); one update at a time per pipeline; harder to test outside Databricks; our 13 stages must be rewritten as declarationsDrop
DLT as an adapter for CDC tables only (AUTO CDC builds SCD1/SCD2 from DMS changes)Out-of-order SCD2 handled by Databricks; removes our hardest MERGE codeTwo engines to know; must still report into our run logPursue after a spike

Auto Loader, in plain words

Auto Loader is a Spark streaming source for files. You point it at an S3 folder; it finds new files, remembers which ones it has read (in its checkpoint), handles schema changes, and puts unexpected fields in a rescue column. It finds new files in one of two ways:

  • Listing the folder. Fine for small folders, but slow and costly for big ones.
  • File events. S3 tells Databricks about each new file through queue notifications that Databricks sets up on the Unity Catalog external location. This is recommended.

Kinesis is for records; S3 events are for files. The sensor stream comes through Kinesis. DMS change files, LIMS results and partner files come through S3 and Auto Loader.

"A file lands in S3, call DLT for that file": four ways to react

OptionHowDelayProsConsVerdict
F1 · Auto Loader on a scheduleJob or pipeline every 15 min; Auto Loader (available-now) reads every new file since last run≤15 min + runSimplest; files batched together; cheapRuns even when nothing arrived (seconds of cost)Pursue for regular feeds (DMS CDC)
F2 · Databricks file arrival triggerJob trigger watches the external location or volume path; when new files appear it starts the job (which can run a DLT pipeline task); Auto Loader inside picks up all new files~1–2 min + runEvent-driven; no runs when nothing arrives; no AWS glue codeChecks about once a minute; a burst of files becomes one run (good) but runs queue; check trigger limits per workspacePursue for irregular drops (LIMS, partners)
F3 · S3 event → EventBridge → Lambda → Jobs API "run now" with the file pathOne run per file, path passed as a parameter~1 min + startupPer-file controlA pipeline start per file (slow, costly); one DLT update at a time, so files queue; passing paths bypasses Auto Loader's bookkeeping, so a missed event is a lost file; more code to ownDrop
F4 · Auto Loader continuousAlways-on stream on the folderseconds to 1 minFastest for filesAlways-on computeDrop unless a T2 need appears

Answer: do not hand a single file to DLT. Trigger the pipeline (F2), or run it on a schedule (F1), and let Auto Loader find every new file. That way a missed event never loses a file, and ten files arriving together cost one run, not ten.

Can S3 be used?

Yes, in four of the five roles below, but not as the conveyor belt itself. S3 is a warehouse, not a belt: it holds objects, it does not push them to readers, and every object written is a paid request.

The number that settles it: at production scale (300 readings a second), one S3 object per reading is about 778 million PUT requests a month. At about $0.005 per 1,000 PUTs, that is roughly $3,900 a month in requests alone. Readers would also have to list or be notified about each object, and then the platform lane reads millions of tiny files. Buffering readings into one file a minute fixes all three, at the cost of a minute of delay.
Use of S3HowDelayGood forVerdict
S3 as the event log between devices and processorsDevices write objects; processors list the bucket.Seconds to minutes, unpredictableNothing in our designDrop
S3 as micro-batch landing ("streaming over files")Firehose buffers 60 s or 64 MB, writes one file; S3 event notification goes to SQS; Auto Loader picks it up.1–3 minutesSources that need minutes, not seconds; the AWS-only platform lane; CDC files from DMSPursue
S3 as Kappa's long memoryBronze Delta (or Iceberg) tables on S3, kept forever.n/aReplay, history, ML trainingPursue
S3 for processor checkpoints and stateFlink and Spark save checkpoints to S3.n/aRecovery after restartsPursue (managed for us)
S3 Express One Zone as a fast logSingle-zone, low-latency bucket used as a queue.Tens of milliseconds per request, still polledKafka-on-object-storage products use it internally; not something to build ourselvesDrop

So: Kinesis (or Kafka) is the belt; S3 is where the belt empties into, and where history lives. For sources that only need minutes, you can skip the belt entirely and go straight to S3 files.

Streams that are not IoT

The same design takes any event source. What changes per source is the front door (R2) and whether it needs the belt (R3) at all. Rule: use the belt only when a reader needs it within minutes, or when several readers need the same events.

Source kindTyre exampleEvent-native?Front doorNeeds the belt?Recommended pathVerdict
Machine sensorsCuring presses, mixersYesPlant gateway (SiteWise Edge on Greengrass)Yes: two readers, secondsGateway → Kinesis → live lane + platform lanePursue
Connected productsTyre sensors on fleet vehiclesYesIoT Core (MQTT, one certificate per vehicle), batched on the vehicleYesIoT Core rule → Kinesis → both lanesPursue later, if in scope
Database changes (CDC)SAP orders, MES productionNo; CDC makes eventsAWS DMS (full load + ongoing changes), or DebeziumUsually noDMS → S3 Parquet files every 1–5 min → Auto Loader → bronze. Switch to DMS → Kinesis only if a live reader appears.Pursue (files by default)
App eventsDealer portal clicks, warranty-claim appYesAPI Gateway → Kinesis, or SDK from the backendOnly if someone acts on it liveAPI Gateway → Firehose → S3 (minutes); Kinesis if a live use appearsPursue
SaaS webhooksSalesforce, payment providerYes, but low volume and burstyAPI Gateway → EventBridge (routes, archives)NoEventBridge → Firehose → S3 → bronzePursue
Partner KafkaLogistics partner's Kafka or Confluent topicYesTheir clusterTheirsDatabricks reads their Kafka directly; or MSK Replicator if we need our own copyPursue when it appears
FilesLIMS results, partner CSVsNoS3 bucketNoS3 notifications → Auto Loader (already the batch design)Pursue
Logs and metricsApp logs, CloudWatch metricsYesCloudWatch subscription or metric streamNoFirehose → S3 → ops tablesPursue
Queues used as a logSQS, SNS, EventBridge as the analytics source———A queue deletes a message once read: no replay, no second reader, no per-key orderDrop as a log

Before you choose a path for a source, ask its owner the stream questions (group O) in the source system questionnaire: who acts and how fast, format and schema registry, key and order, event time and clocks, lateness and buffering, bursts, batching, identity, registry, heartbeat, other readers, history, commands back, personal data and test traffic.

Systems apart from IoT: yes, and they fit without a second architecture. Only machine sensors (and later connected tyres) need the belt and the live lane today. Everything else lands as files or through Firehose into the same bronze, under the same contracts and checks.

Binding options per part

The abstract parts (R1–R12) stay fixed; a binding picks the product for each part. Every table lists options from AWS, Databricks and open source with pros, cons and a verdict. In the Clean Data Engineering specs, the chosen rows become a binding file for that platform; another cloud is just another binding. One table per part. Prices are rough list prices for US regions (see Cost). "AWS-only" and "AWS + Databricks" share C1–C7; they differ from C8 onward.

C1 · Edge and plant gateway (R1)

OptionProsConsVerdict
IoT SiteWise Edge on Greengrass, with stream manager exporting to KinesisOPC-UA and Modbus collectors built in; buffers to local disk when offline; batches; AWS manages updatesOne more AWS runtime to learn; SiteWise cloud asset modelling is extra and priced per data pointPursue (export straight to Kinesis; skip SiteWise cloud storage)
Third-party industrial software (Kepware, Ignition) publishing MQTT or to KinesisPlants often already run oneLicences; another vendorPursue if the plant already has it
Our own Python service on a small boxFull controlWe own buffering, retries, updates, security patchesDrop

C2 · Front door and identity (R2)

OptionProsConsCost noteVerdict
Gateway writes directly to Kinesis (IAM role via Greengrass token exchange)Cheapest; fewest hops; batches up to 500 records per callIdentity is per gateway, not per machineKinesis charges onlyPursue for plants
IoT Core (MQTT, X.509 per device, rules engine → Kinesis)Per-device identity, device shadows, commands back, millions of devicesPriced per message and per rule actionAbout $1 per million messages + $0.30 per million rule actions: ~$1,000 a month if every press sends alonePursue for connected tyres and commands; Drop for plant readings
API Gateway → Kinesis or FirehoseApps and webhooks over HTTPS, with authNot for devices on MQTTAbout $1 per million calls (HTTP API)Pursue for apps
Databricks Zerobus Ingest (GA Feb 2026): send events straight into a Delta tableNo belt to run; serverlessOnly one reader (the table); no live lane from the same events; no Kinesis-style replayLakeflow Jobs serverless pricingDrop for sensors; reconsider for a single-reader app stream

C3 · Event log (R3)

OptionProsConsRough monthly, production scaleVerdict
Kinesis Data Streams, provisionedServerless to run; native to IoT Core, Firehose, Flink, Lambda and Databricks; 1 MB/s or 1,000 records/s in per shardShard sizing is ours; 7-day retention costs extra~$60–130 (2 shards, 7 days, 2 dedicated readers)Pursue (recommended)
Kinesis on-demandNo sizing; good while volume is unknownHigher per-GB price at steady load~$160Pursue for the first months, then switch
MSK provisioned (Kafka)Kafka ecosystem: Debezium, Kafka Connect, partner topics, many readers, very long retention with tiered storageBrokers to size, patch and tune; cross-zone traffic~$480 (3 × m7g.large + storage)Pursue only if Kafka tools are needed
MSK ServerlessNo brokers to manage~$550 a month floor before any data~$700Drop at this scale
Confluent CloudBest tooling; cloud-neutralHighest price$1k–3kDrop unless multi-cloud is required
SQS, SNS, EventBridgeMessaging and routingNo replay, no second reader —Drop as the log

C4 · Contract (R4)

OptionVerdict
Avro in Glue Schema Registry, backward-compatible changes only; fields reading_id, press_id (key), event_time, gateway_time, schema_version, tags; ODCS contract in the catalogPursue
Confluent Schema RegistryPursue only with MSK and Confluent tools
JSON with no registryDrop

C5 · Live processor (R5)

OptionProsConsRough monthlyVerdict
Managed Service for Apache FlinkEvent-time windows, watermarks, state, interval and temporal joins; exactly-once with checkpointsFlink skills needed; always on~$250 (3 processing units)Pursue
Lambda reading KinesisStateless threshold alerts ("pressure above limit"); nearly freeNo reliable windows or state~$10Pursue for simple limit alerts beside Flink
Databricks real-time mode (GA Mar 2026)One engine (Spark) for both lanes; under 100 ms claimedClassic compute only, fixed size, always on; check Kinesis source support~$700–1,000 for a small always-on clusterDrop for now; revisit if the team wants one engine
Glue streaming (Spark)Serverless SparkAlways on, weaker than Flink for the live lane, not needed for the platform lane~$650Drop
AWS IoT Events (detector models)Was built for thisEnd of support 20 May 2026—Drop (retired)

C6 · Live store (R6)

OptionProsConsRough monthlyVerdict
DynamoDB (latest per press, plus 48-hour items with TTL)Serverless; pay per write; DynamoDB Streams can push every change to the gatewayNo ad-hoc slicing or SQL~$50Pursue (recommended)
Lakebase (Databricks Postgres)SQL; already our metadata store; Databricks-nativeDoes not push changes, so the gateway must poll; always-on compute~$100–300 (check)Pursue as the alternative
ClickHouse CloudFast ad-hoc slicing over 48 hoursNew vendor; does not pushDev <$100; prod $500+Pursue only if live screens need slicing
ElastiCache (Valkey)Fastest latest-value cacheNo history; memory to size~$50–150Drop as primary
Timestream for LiveAnalyticsTime seriesClosed to new customers since 20 Jun 2025—Drop (closed)
Timestream for InfluxDBTime series, AWS-managedInstances to size; Influx skills~$100+Drop unless the team knows Influx

C7 · Push and alerts (R7)

OptionVerdict
API Gateway WebSocket: snapshot, then changes; at most 1 update a second per widgetPursue
AppSync Events (managed pub/sub over WebSocket)Pursue as a simpler alternative; test it in the physical step
SNS (SMS, email) and EventBridge (to ServiceNow, Teams)Pursue for alerts

C8 · Lake ingest: belt or files to bronze (R8)

OptionStackVerdict
Databricks Kinesis source in a streaming table, trigger every 15 minutes (our lander steps run in foreachBatch; dedupe on reading_id)AWS + DatabricksPursue (check that "available now" works with the Kinesis source on your runtime)
Firehose → S3 Parquet → Auto LoaderBothPursue as the fallback if the Kinesis source falls short
Firehose → S3 Tables (managed Iceberg)AWS-onlyPursue for AWS-only
ZerobusAWS + DatabricksDrop here (see C2)

C9 · Lake tables and transforms (R9)

OptionStackVerdict
Delta on S3 in Unity Catalog; Lakeflow streaming tables for bronze and silver; jobs for the stream-hour step and gold window recomputeAWS + DatabricksPursue
S3 Tables (Iceberg); Glue ETL or EMR Serverless for silver and gold; Glue Data Catalog and Lake FormationAWS-onlyPursue for AWS-only
Athena scheduled INSERTs for simple goldAWS-onlyPursue for small, single-input aggregates

C10 · Query and BI (R10)

OptionStackVerdict
Databricks SQL, metric views, AI/BI dashboards, GenieAWS + DatabricksPursue
Athena + QuickSightAWS-onlyPursue for AWS-only
Amazon Managed Grafana on the live storeBothPursue for engineers' time-series boards (optional)
Redshift ServerlessAWS-onlyDrop at this scale unless already owned

C11 · Commands back (R11) and C12 · Monitoring (R12)

OptionVerdict
IoT Core jobs and device shadows for connected tyres; Greengrass deployments to change gateway settingsPursue only when control is in scope
Monitoring: CloudWatch alarms on Kinesis reader lag (iterator age) and write throttling, Flink checkpoint time and back-pressure, DynamoDB throttles; a "silent stream" heartbeat per plant; a run recorder that logs every 15-minute run as it finishes (push, not poll)Pursue

AWS-only vs AWS + Databricks, side by side

RoleAWS-onlyAWS + Databricks
R1–R2 Plant to front doorSame: SiteWise Edge on Greengrass → Kinesis; IoT Core for connected tyres and commands; API Gateway for apps
R3–R4 Belt and contractSame: Kinesis provisioned, 7 days; Avro in Glue Schema Registry
R5–R7 Live laneSame: Managed Flink (+ Lambda for simple limits) → DynamoDB → API Gateway WebSocket; SNS and EventBridge for alerts
R8 Lake ingestFirehose → S3 Tables (Iceberg)Lakeflow streaming table reads Kinesis every 15 min
R9 Tables and transformsIceberg; Glue ETL or EMR Serverless; Glue Catalog and Lake FormationDelta; Lakeflow pipelines and jobs; Unity Catalog
R10 UseAthena, QuickSight, SageMakerDatabricks SQL, AI/BI, Genie, Databricks ML
R12 GovernanceLake Formation, CloudTrailUnity Catalog (lineage, tags, masks), system tables
StrengthFewest vendors; lowest bill at small scale; everything pay-per-useOne governed home for batch and streams; same code for both; Genie and ML on the same tables; one governed platform for batch and streams
WeaknessSpark/Glue jobs and Iceberg upkeep are ours; weaker lineage and semantic layer; two engines (Flink plus Glue) to ownTwo bills; Databricks cost on top for every 15-minute run
VerdictDrop as the main design; keep as the "no Databricks" reference variantPursue (recommended when Databricks is the platform)

The real split is by lane, not by vendor: the live lane is AWS-native in both stacks, because it is about seconds and pushing to screens. The platform lane goes wherever the rest of the platform lives. A Databricks-only design (Zerobus in, real-time mode for live, Lakebase as live store) is possible since 2026, but it gives up the shared belt and replay; I would not choose it today.

Performance and sizing

Three scales

Demo world (DataTwin)Production, assumedStretch: connected tyres
WhatThe tyre world's sensor stream5 plants, 1,500 machines; one message every 5 s per machine carrying ~20 tags, ~1 KB200,000 vehicles; one 500-byte message a minute
Messages622,080 a day (~7 a second)~300 a second; ~778 million a month~3,300 a second; ~8.6 billion a month
Data intiny~0.3 MB/s; ~26 GB a day raw~1.7 MB/s; ~144 GB a day
Kinesis shards11 needed, 2 for headroom and parallel reads4 needed (record limit), 6 for headroom; or pack records to beat the 1,000/s limit
Flink1 unit2 units + 1 for orchestration4–8 units
Bronze growth~0.1 GB a day~4–6 GB a day compressed~25–35 GB a day compressed

If your real plant numbers differ, only this table and the cost section change; the architecture does not.

Latency budget

HopLive lanePlatform lane
Machine → gateway → Kinesis0.2–1 s (gateway batches every second)same
Kinesis → reader~0.1 s with a dedicated reader (enhanced fan-out); ~1 s with shared pollingwaits for the next 15-minute run
ProcessingLimit alerts: immediate. 1-minute windows: window end + 5 s allowed lateness2–5 min per run
Store and push~0.1–0.3 sgold hourly job at :15
End to end1–3 s for alerts; up to ~65 s for window values15–20 min for silver; 15–75 min for gold

Five performance rules

  1. Batch at the edge. Bundling readings per gateway per second cuts messages, requests and cost by 10–100 times.
  2. Key by press. Order is kept per press; the many presses spread across shards. Never key by plant: one hot shard.
  3. Give each lane its own reader (enhanced fan-out) so the 15-minute catch-up run cannot slow the live lane.
  4. Emit changes, not every reading, into the live store and to screens; at stretch scale this is the difference between $50 and $5,000 a month in DynamoDB writes.
  5. Bronze files sized well: 15-minute runs write fewer, larger files; liquid clustering and predictive optimization keep them tidy.

Cost (rough, per month, production scale)

Rough US-region list prices as I remember them for 2025–2026, not quotes. Check with the AWS Pricing Calculator and Databricks pricing before showing a client. People and support costs are not included.
ItemAWS-onlyAWS + DatabricksNote
Plant gateways (Greengrass fee)~$1~$1Hardware not counted
Kinesis (2 shards, 7 days, 2 dedicated readers)~$120~$120On-demand ~$160
Managed Flink (3 units)~$250~$250
Lambda limit alerts~$10~$10
DynamoDB live store~$50~$50Changes only, 1-minute windows
API Gateway WebSocket, SNS~$100~$100~50 live screens
Lake ingestFirehose → S3 Tables ~$50Streaming table every 15 min ~$300–800Serverless DBUs; the biggest unknown; measure on your workload
Storage (S3, first year average)~$20~$20Grows ~$2 a month each month
Silver, gold, BIGlue/EMR + Athena + QuickSight ~$150–400Jobs + SQL warehouse ~$100–300 extraDatabricks warehouse is shared with batch
Total~$750–1,000~$950–1,650Live lane alone is ~$530 in both
If every press goes through IoT Core instead of the gateway+~$1,000+~$1,000Why C2 recommends gateway → Kinesis
Stretch scale (connected tyres)IoT Core ~$9k unbatched, ~$1k with 10 readings per message; Kinesis ~$300; Flink ~$450–750; DynamoDB $50 or $5k depending on "changes only"Per-message prices dominate: batch on the vehicle

What drives the bill

  1. Per-message prices (IoT Core, S3 PUTs, DynamoDB writes) dominate at scale: batch at the edge and emit only changes.
  2. Always-on compute (Flink): keep the live lane to what someone acts on within minutes.
  3. The 15-minute cadence sets the Databricks part: 30 minutes roughly halves it; 5 minutes roughly triples it.
  4. Retention on the belt: 7 days is cheap; history belongs in bronze.

Replays

A replay is reading old data again on purpose: after a fault, a bug fix or a logic change. Because Kinesis keeps 7 days and bronze keeps everything, every layer can be rebuilt. Rules: every replay is a logged run with a reason; replays use separate compute so live service levels are not hurt; replays never re-send alerts.

ReplayWhenHowReads fromWhy it is safe
Live lane restartFlink crash or deployRestart from the last checkpoint (automatic)KinesisExactly-once state; upserts by press and minute
Live lane rebuildNew window or limit logicStart Flink from a timestamp up to 7 days backKinesisWrites are upserts; alerts marked "replay" and suppressed
Re-ingest bronzeLander bug, lost runNew checkpoint starting at a Kinesis timestampKinesis (≤7 days)Silver deduplicates on reading_id
Rebuild silverRule change, SCD type changeReset that table's checkpoint and re-read bronze for the rangeBronze (forever)Bronze is never changed; restatement logged
Recompute gold windowsLate data, dimension fix, logic changeWindow recompute for listed hours or daysSilverWhole windows replaced; restatements visible in the SLA report
Gateway resendPlant back onlineGateway sends its disk bufferGatewayDuplicates removed by reading_id; late data counted
History backfillYears of historian dataSeparate bulk job (ETL design)Historian export on S3Never through the stream

Insurance option: also copy the raw Kinesis stream to S3 through Firehose (~$25 a month). Then a reader that falls more than 7 days behind still loses nothing. Verdict: Pursue, because it is cheap and closes the only real data-loss gap.

Error handling

Principle: errors are rows, not log lines. Every rejected record lands in a table with run id, stage, reason and a pointer to the raw payload, so the SLA report can count it. One bad record never stops a pipeline.

What goes wrongWhereWhat happensShows up as
Plant network downGatewayBuffers to disk (hours), sends on reconnect"Silent plant" alert after 5 min; late-data count when it returns
Writes throttledKinesisSDK retries with back-offThrottle alarm; add a shard
Bad message (cannot decode, unknown schema)Live lane / platform laneLive: side output to a dead-letter folder. Platform: kept raw in bronze, sent to quarantine in silverQuarantine by reason
Contract or quality failure (missing field, impossible value, stuck sensor)SilverRow to quarantine with the rule that failed; the rest continueDQ counts; alert above a threshold (e.g. 1%)
DuplicatesSilverRemoved on reading_idDuplicate count per hop
Later than the watermarkLive lane / goldLive: left out of live views. Gold: accepted, window recomputedLate count by lateness bucket; restatements
Processor crashFlink / Spark jobRestart from checkpoint; job retries 3 timesRun log; alert after retries
A 15-minute run fails or is skippedPlatform laneNext run catches up (available-now reads everything new)Alert after 2 missed runs; SLA breach marked
Reader falls behindAny Kinesis readerKeeps going; risk only near 7 daysLag alarm at 1 h (live) and 24 h (platform); Firehose copy is the backstop
Sink failure (DynamoDB throttle, stale WebSocket)Live laneRetry; drop dead connectionsPush error rate
Bad deployAnyRoll back, then replay the affected rangeReplay run with reason

The end-to-end SLA report

One report answers four questions every day: did we keep the promises, where did the time go, did anything get lost, and what went wrong.

Timestamps every reading carries

StampSet byMeaning
t0 event_timeMachineWhen it happened
t1 gateway_timePlant gatewayWhen the plant sent it (also catches clock skew)
t2 kinesis_arrivalKinesisWhen it reached AWS
t3 live_emitted_atFlinkWhen the live lane pushed it (alerts and windows)
t4 bronze_at + run idLanderWhen it landed in bronze
t5 silver_at + run idSilver builderWhen it became clean data
t6 gold_complete_atGold builderWhen its press-hour was marked complete

How it is built

  • Flink writes a one-minute summary per plant: counts, p50/p95/p99 of t3 − t0 and t3 − t2, late and bad counts. The summary goes through Firehose into an ops table.
  • Each 15-minute run records counts and delay percentiles per hop through the run recorder.
  • Two ops tables hold the result:
    • ops.stream_sla_minute: stream, plant, minute, hop, count, p50, p95, p99, late, duplicate and quarantined counts.
    • ops.gold_hour_completeness: press-hour, complete at, inputs late, restated.
  • Metric views sit on top, feeding an AI/BI dashboard. Ops agents read them through tools, never through raw queries.

The five pages of the report

1 · Scorecard

Each promise with % met for yesterday, 7 days and 30 days, plus breach count. Example: "Alerts ≤5 s: 99.6% (target 99%)".

2 · Where the time goes

A waterfall of p95 delay per hop (machine → gateway → Kinesis → live, and → bronze → silver → gold) by plant.

3 · Nothing lost

Counts per hop side by side: gateway sent, Kinesis received, bronze, silver (+ duplicates + quarantined), gold minutes covered. The difference must be zero or explained.

4 · Errors and lateness

Quarantine by reason, late readings by bucket (<5 min, <1 h, <1 day, older), restated gold hours.

5 · Incidents

Silent plants, missed runs, lag alarms and replays, each with start, end, cause and which promise it broke (event-time vs arrival-time).
Example, platform lane p95 per hop (minutes) machine → Kinesiswait for 15-min runbronze → silversilver → gold hour 0.2153.56 → 25 total

Made-up numbers to show the shape: most delay is waiting for the next run, which is a cadence choice, not a performance problem.

Non-functional concerns

ConcernWhat it means hereTargetHow the design meets it
FreshnessHow far behind each screen or table may beLive: alerts ≤5 s, windows ≤90 s. Silver ≤20 min. Gold ≤75 min.Two lanes; freshness checked per table; a late run alerts
Throughput and growthHandle 10× without redesign3,000 messages a secondAdd shards and Flink units; same design
OrderingReadings of one press in orderPer pressPartition key press_id; windows use event time anyway
Delivery and correctnessNothing lost, nothing counted twiceExactly-once resultsAt-least-once delivery + dedupe on reading_id + idempotent MERGE; Flink checkpoints
Late and out-of-order dataGateway offline for a whileLive: 5 s lateness, later readings go to the platform lane only. Gold: 2-day lookbackWatermarks; window recompute re-checks 2 days
Replay and retentionRecover from a bad deploy or outage7 days on the belt; forever in bronzeKinesis retention; bronze append-only, never deleted
AvailabilityKeep working when a part failsLive lane 99.9%; no data loss while the gateway buffersGateway disk buffer; Kinesis, Flink and DynamoDB are multi-zone
Disaster recoveryA whole region downPlatform lane: hours; live lane: best effortGateways buffer for hours; bronze replicated to a DR region
SecurityOnly known senders and readersPer-gateway identity; least privilege; encrypted everywhereIAM role per gateway, X.509 per vehicle; KMS keys; VPC endpoints; no public brokers
PrivacyPersonal data in streamsNo PII in bronze outside restricted columnsSensors carry none; dealer app events and vehicle location are PII: masked in silver, tagged in the catalog
Schema evolutionFirmware adds a tagNo reader breaksRegistry allows backward-compatible changes only; unknown fields go to a rescue column
Data qualityImpossible values, stuck sensors, clock skewBad rows visible, never silently droppedRange and staleness checks in silver; quarantine table; gateway time vs press time check
ObservabilityKnow when it is slow or silentAlert within 5 minutes of a silent plantLag and throttle alarms; heartbeat per plant; run recorder; one alert per incident
Cost controlNo surprise billsMonthly budget with alertsEdge batching; changes only; tags per data product for cost reports
OperabilitySmall team can run itNo self-managed clustersAll managed or serverless; Flink is the one skill to grow
PortabilityReuse on Azure, Fabric, GCPOnly adapters changeStream port: Kinesis adapter today; Event Hubs and Pub/Sub adapters later; contracts and checks stay
TestabilityProve it before a client sees itPlanted incidents caughtDataTwin sensor stream with late, duplicate and silent-press incidents; replay tests; port contract tables on moto

Decisions to make, with recommendations

#QuestionOptionsMy recommendation
S1Which stack?AWS-only · AWS + Databricks · Databricks-only (Zerobus + real-time mode)AWS + Databricks, with an AWS-native live lane. Keep AWS-only as a documented variant.
S2How do plant readings enter?Gateway → Kinesis directly · every machine through IoT CoreGateway → Kinesis; IoT Core only for connected tyres and commands. Saves ~$1,000 a month.
S3Which belt?Kinesis · MSKKinesis (on-demand first month, then provisioned). MSK only if Kafka tools or partner topics arrive.
S4Which live store?DynamoDB · Lakebase · ClickHouse CloudDynamoDB: it pushes changes and costs ~$50. Lakebase if you want SQL and one fewer service.
S5How do SAP and MES changes arrive?DMS → S3 files · DMS → Kinesis · Debezium on MSKDMS → S3 files into the existing batch path; no belt until a live reader needs it.
S7DLT or not?Non-DLT (our Structured Streaming jobs) · DLT for everything · DLT only for CDC tablesNon-DLT as default; DLT for CDC→SCD tables after a spike.
S8How do file drops start processing?Schedule · Databricks file arrival trigger · S3 event → Lambda → run per fileSchedule for regular feeds, file arrival trigger for irregular ones; never one run per file.
S9Live UI route?Direct push · live store + push · ClickHouse · LakebaseLive store (DynamoDB) + push. ClickHouse only if screens need slicing.
S10Firehose raw copy of Kinesis to S3?Yes · noYes (~$25 a month); removes the 7-day loss risk.
S6Are connected tyres in scope?Now · later · neverLater. The design already fits them; it only adds IoT Core and more shards.

Notes and sources

FAQ

Questions people ask

Should a 1 to 5 second streaming lane call Databricks?

No. Keep it light and inside the streaming layer: validate, deduplicate, limit checks, one-minute windows, and small reference data copied in on a schedule. Databricks reads the raw stream separately and builds the numbers of record.

Where do SCD type 1 and type 2 happen in a streaming lakehouse?

In silver, by dataset kind: master data as SCD type 2, transactions as type 1, sensor events append only. Gold builds dimensions from silver's history and joins facts as of event time.

DLT or plain Structured Streaming?

Plain Structured Streaming in jobs as the default, because it is portable and fully testable; DLT's AUTO CDC is worth it for tables built from database changes, where out-of-order SCD2 is hardest to write by hand.

Can S3 be used as the stream?

Not as the belt itself: one object per event is slow and costs a request each. S3 works well as minute-level landing for Auto Loader and as the long-term history that makes Kappa replays possible.

Should I trigger a DLT pipeline for each file that lands in S3?

No. Use a file arrival trigger or a schedule and let Auto Loader pick up every new file, so a missed event never loses a file and ten files cost one run.

Related

Where it connects

Real-time analytics, drawn simply · Medallion architecture · Data platform on AWS + Databricks · All data engineering