Data PlatformZoom level 1
Ingestion: from a source system to bronze
| Part | Starting point | Called by (found by reading the code) |
|---|---|---|
| One ingestion run (landing to bronze, pull to bronze) | ingest ingest/runner.py:112 | run_ingest · scripts/run_hop.py:60run_days · ingest/daybyday.py:129 |
| The day loop that plays the world and judges bronze | run_days ingest/daybyday.py:102 | main · scripts/run_platform.py:615 |
Ingestion brings data from the systems that produce it into bronze, the first layer of the platform. Bronze holds every batch exactly as the source sent it, with a few columns added that say where each row came from.
Sources differ a great deal. One sends files, one must be asked, one never stops talking. If each had its own loading code, each would have its own idea of a duplicate, of a broken file and of what to record. Here the sources differ only in how a batch is found. Once a batch is in hand, the same checks run for all of them.
Three words used on every page
| Word | Meaning |
|---|---|
| Landing zone | A storage folder that holds delivered files exactly as they arrived, each with a small manifest file beside it. A delivery is written once and never changed. |
| Lander | Our code that takes a batch, checks it and writes it to bronze. It is the same code whether the batch came from landing, from a database or from a stream. |
| Bronze | One table per dataset that keeps every accepted batch as delivered. Nothing is cleaned or merged here. |
Push, pull and stream
| Way in | Who starts it | Where a batch comes from | Why it exists |
|---|---|---|---|
| Push | The source side. A mover, which is a service outside our code, copies files into landing. | A delivery in landing. | Change capture services and file transfers already exist as products. The platform only asks that they write each delivery once with a manifest. |
| Pull | Our run. It asks the source. | Rows read from a database copy or from pages of an interface. There is no landing hop. | Some sources only answer questions. |
| Stream | Our run. It reads each part of the stream from where it stopped to the end, then stops. | Records from a stream. The stream itself plays the part of landing. | Sensor readings arrive all day. Reading them in scheduled runs keeps one loading model for everything. |
The nine tyre datasets and the way each comes in
Each dataset manifest of the tyre company states its way in with two fields, ingest.mode and ingest.read. The table lists them as the manifests have them.
| Dataset | Mode | Read | What the source is |
|---|---|---|---|
erp.sales_order_line | push | change_feed | Order lines from the database of the ERP, the company's system for orders and stock, as a feed of changes. |
erp.delivery | push | change_feed | Deliveries from the ERP database. |
mes.production | push | change_feed | Production records from the factory system. |
lims.inspection_result | push | files | CSV files from the laboratory system. |
partner.dealer_sellout | push | files | Monthly CSV files from dealers. |
erp.material | pull | full_copy | The material master, as a nightly full copy. |
mes.press | pull | full_copy | The press master, as a nightly full copy. |
dealer_api.dealer | pull | pages | Dealers from an interface, read page by page. |
press_stream.press_reading | stream | events | Press sensor readings. |
The pages under this one
| Page | What it answers |
|---|---|
| Run loop | What one run does, step by step, and every call it makes to the metadata store. |
| Five pipelines | How each of the five pipelines finds what is new and what it remembers for next time. |
| Entry steps | The checks every batch passes before it may enter bronze. |
| Landing to bronze | What landing looks like, what a bronze row looks like, and why writing a batch twice changes nothing. |
| Sources and movers | The source ports and the adapters behind them, and which of them are real connectors. |
The design decisions
| Decision | What it means | Why |
|---|---|---|
| One set of entry steps for every way in | A pushed file, a pulled copy, a page and a stream chunk all go through the same function. | A rule such as the duplicate check is written and tested once. |
| A batch is identified by its content | The batch key is a hash of the batch's bytes. The file name and the arrival time are not part of it. | A file that is sent again under a new name is still recognised. |
| A held batch is never a success | A batch that fails a check is not written. Its dataset run is recorded as failed, with the reason. | A broken delivery must be seen by a person, not skipped quietly. |
| Positions live in the metadata store | How far a dataset has been read is kept in the store and moves only when the dataset run succeeds. | A failed run is taken again from the same place. |
| Bronze is written once per dataset per run | The new batches of a run are written in one commit, and only then recorded as loaded. | The record never says loaded before the rows are there. |
src/dataplatform/ingest/__init__.py:1The package's own summary of its four modules."""Ingestion (W1b; ADR-002, D11): sources -> bronze, driven by the metadata store, every run logged in it.
ports.py the ports ingestion talks to: landing (push), the bronze writer, the tokenizer
steps.py the lander steps, one library for every way in (ING-01): parse, contract, derive, protect,
volume, mass-delete guard, stamp, write, log
pipelines.py one pipeline per source type, chosen by the dataset's ingest mode and read:
push change_feed, files landing -> lander
pull full_copy, pages the lander steps in-process, straight to bronze
stream events the stream bus is the landing; trigger-once, from checkpoints
runner.py one run: start_run -> per dataset start_dataset_run ... finish_dataset_run -> finish_run,
for run types normal, reload and backfill
Names here say what a thing is, never which product holds it (R1).
"""How we know it works
tests/dataplatform/test_ingest_world.py runs the simulated tyre company for 14 days at a small scale through ingestion with local adapters. For every day, the number of rows that reached bronze per dataset must equal the simulated company's own count. The company also plants eleven incidents, one on each of days 3 to 13, and each must get the verdict ingestion owes it.
tests/dataplatform/test_ingest_world.py:86The planted incidents: name, day, the rule it tests, and the verdict expected from ingestion.INCIDENTS = [
("readings-out-of-order", 3, "LATE-03", "flagged"),
("order-lines-deleted", 4, "DEL-01", "flagged"),
("sellout-file-late", 5, "LATE-01", "deferred"),
("sellout-file-resent", 6, "DUP-01", "handled"),
("lab-column-renamed", 7, "SCH-02", "handled"),
("promised-date-out-of-order", 8, "SCD-02", "flagged"),
("material-copy-short", 9, "DEL-04", "handled"),
("promised-date-corrected", 10, "HIST-05", "deferred"),
("new-dealer-late", 11, "LATE-02", "deferred"),
("change-feed-drops-rows", 12, "REC-05", "deferred"),
("stream-replayed", 13, "DUP-04", "flagged"),
]A verdict of handled means ingestion did what the rule asks, such as skipping a file sent twice. Flagged means ingestion counted the problem and left the fix to silver. Deferred means the problem is not ingestion's to catch. Tests and proof explains the simulated company.
- The connectors to real source systems are thin. Most adapters behind the source ports are local stand-ins built for the simulated company. Sources and movers says which is which.
- Datasets are run one after another inside a run. There is no parallel loading.
- Streams are read in scheduled runs that stop when they reach the end. There is no continuously running reader.
- A source discovery helper exists and is tested, but no runner calls it.