cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Ingestion

Data PlatformZoom level 1

Ingestion: from a source system to bronze

Where this starts in the code
PartStarting pointCalled by (found by reading the code)
One ingestion run (landing to bronze, pull to bronze)ingest ingest/runner.py:112run_ingest · scripts/run_hop.py:60
run_days · ingest/daybyday.py:129
The day loop that plays the world and judges bronzerun_days ingest/daybyday.py:102main · scripts/run_platform.py:615

Ingestion brings data from the systems that produce it into bronze, the first layer of the platform. Bronze holds every batch exactly as the source sent it, with a few columns added that say where each row came from.

Sources differ a great deal. One sends files, one must be asked, one never stops talking. If each had its own loading code, each would have its own idea of a duplicate, of a broken file and of what to record. Here the sources differ only in how a batch is found. Once a batch is in hand, the same checks run for all of them.

Read it left to right. The upper path is push, the lower path is pull and stream. The highlighted box is where every batch is checked. Click a box to zoom in.

Three words used on every page

WordMeaning
Landing zoneA storage folder that holds delivered files exactly as they arrived, each with a small manifest file beside it. A delivery is written once and never changed.
LanderOur code that takes a batch, checks it and writes it to bronze. It is the same code whether the batch came from landing, from a database or from a stream.
BronzeOne table per dataset that keeps every accepted batch as delivered. Nothing is cleaned or merged here.

Push, pull and stream

Way inWho starts itWhere a batch comes fromWhy it exists
PushThe source side. A mover, which is a service outside our code, copies files into landing.A delivery in landing.Change capture services and file transfers already exist as products. The platform only asks that they write each delivery once with a manifest.
PullOur run. It asks the source.Rows read from a database copy or from pages of an interface. There is no landing hop.Some sources only answer questions.
StreamOur run. It reads each part of the stream from where it stopped to the end, then stops.Records from a stream. The stream itself plays the part of landing.Sensor readings arrive all day. Reading them in scheduled runs keeps one loading model for everything.

The nine tyre datasets and the way each comes in

Each dataset manifest of the tyre company states its way in with two fields, ingest.mode and ingest.read. The table lists them as the manifests have them.

DatasetModeReadWhat the source is
erp.sales_order_linepushchange_feedOrder lines from the database of the ERP, the company's system for orders and stock, as a feed of changes.
erp.deliverypushchange_feedDeliveries from the ERP database.
mes.productionpushchange_feedProduction records from the factory system.
lims.inspection_resultpushfilesCSV files from the laboratory system.
partner.dealer_selloutpushfilesMonthly CSV files from dealers.
erp.materialpullfull_copyThe material master, as a nightly full copy.
mes.presspullfull_copyThe press master, as a nightly full copy.
dealer_api.dealerpullpagesDealers from an interface, read page by page.
press_stream.press_readingstreameventsPress sensor readings.

The pages under this one

PageWhat it answers
Run loopWhat one run does, step by step, and every call it makes to the metadata store.
Five pipelinesHow each of the five pipelines finds what is new and what it remembers for next time.
Entry stepsThe checks every batch passes before it may enter bronze.
Landing to bronzeWhat landing looks like, what a bronze row looks like, and why writing a batch twice changes nothing.
Sources and moversThe source ports and the adapters behind them, and which of them are real connectors.

The design decisions

DecisionWhat it meansWhy
One set of entry steps for every way inA pushed file, a pulled copy, a page and a stream chunk all go through the same function.A rule such as the duplicate check is written and tested once.
A batch is identified by its contentThe batch key is a hash of the batch's bytes. The file name and the arrival time are not part of it.A file that is sent again under a new name is still recognised.
A held batch is never a successA batch that fails a check is not written. Its dataset run is recorded as failed, with the reason.A broken delivery must be seen by a person, not skipped quietly.
Positions live in the metadata storeHow far a dataset has been read is kept in the store and moves only when the dataset run succeeds.A failed run is taken again from the same place.
Bronze is written once per dataset per runThe new batches of a run are written in one commit, and only then recorded as loaded.The record never says loaded before the rows are there.
src/dataplatform/ingest/__init__.py:1The package's own summary of its four modules.
"""Ingestion (W1b; ADR-002, D11): sources -> bronze, driven by the metadata store, every run logged in it.

    ports.py      the ports ingestion talks to: landing (push), the bronze writer, the tokenizer
    steps.py      the lander steps, one library for every way in (ING-01): parse, contract, derive, protect,
                  volume, mass-delete guard, stamp, write, log
    pipelines.py  one pipeline per source type, chosen by the dataset's ingest mode and read:
                      push   change_feed, files      landing -> lander
                      pull   full_copy, pages        the lander steps in-process, straight to bronze
                      stream events                  the stream bus is the landing; trigger-once, from checkpoints
    runner.py     one run: start_run -> per dataset start_dataset_run ... finish_dataset_run -> finish_run,
                  for run types normal, reload and backfill

Names here say what a thing is, never which product holds it (R1).
"""

How we know it works

tests/dataplatform/test_ingest_world.py runs the simulated tyre company for 14 days at a small scale through ingestion with local adapters. For every day, the number of rows that reached bronze per dataset must equal the simulated company's own count. The company also plants eleven incidents, one on each of days 3 to 13, and each must get the verdict ingestion owes it.

tests/dataplatform/test_ingest_world.py:86The planted incidents: name, day, the rule it tests, and the verdict expected from ingestion.
INCIDENTS = [
    ("readings-out-of-order", 3, "LATE-03", "flagged"),
    ("order-lines-deleted", 4, "DEL-01", "flagged"),
    ("sellout-file-late", 5, "LATE-01", "deferred"),
    ("sellout-file-resent", 6, "DUP-01", "handled"),
    ("lab-column-renamed", 7, "SCH-02", "handled"),
    ("promised-date-out-of-order", 8, "SCD-02", "flagged"),
    ("material-copy-short", 9, "DEL-04", "handled"),
    ("promised-date-corrected", 10, "HIST-05", "deferred"),
    ("new-dealer-late", 11, "LATE-02", "deferred"),
    ("change-feed-drops-rows", 12, "REC-05", "deferred"),
    ("stream-replayed", 13, "DUP-04", "flagged"),
]

A verdict of handled means ingestion did what the rule asks, such as skipping a file sent twice. Flagged means ingestion counted the problem and left the fix to silver. Deferred means the problem is not ingestion's to catch. Tests and proof explains the simulated company.

Not built yet, or known gaps
  • The connectors to real source systems are thin. Most adapters behind the source ports are local stand-ins built for the simulated company. Sources and movers says which is which.
  • Datasets are run one after another inside a run. There is no parallel loading.
  • Streams are read in scheduled runs that stop when they reach the end. There is no continuously running reader.
  • A source discovery helper exists and is tested, but no runner calls it.

Look inside