Home › Learn › Hard problems by layer › Ingestion

Hard problems by layer · 2 of 10

Ingestion: hard problems and anti-patterns

The hard challenges and common anti-patterns of ingestion in a data platform, with how each is solved and tested. This layer must bring every delivery in exactly once, with a recorded outcome, through one set of entry steps.

↑ All layersSource systemsIngestionBronzeBronze to silverSilverSilver to goldGoldGold to semanticSemantic layerSemantic to UIAll challenges

Where we are

Ingestion

SourcesystemsIngestionBronzeBronze tosilverSilverSilver togoldGoldGold tosemanticSemanticlayerSemantic toUI
The ten layers and hops from source to screen. The dark box is this page.

What this layer must achieve. Bring every delivery in exactly once, with a recorded outcome, through one set of entry steps.

Handled by: lander, puller, metadata store, observability.

The hard part

7 challenges, and how they are solved

6 solved1 assistedStatus: 7 built
Solved means the engine does it once the decision is declared, and a test proves it. Assisted means the tool suggests and checks, and a person confirms. People decide means a business decision the tool records and enforces.

Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.

ChallengeWhat goes wrongTyre-world exampleHow we solve itVerdictStatus
Push and pull behave differently
ING-01
Files pushed by a source and data pulled by the platform go through different code, so the same problem is caught on one path and missed on the other.Partner files are pushed; the order database is pulled.Pushed files land and go through the lander; pulled data runs the same lander steps in the same process, straight to bronze.

assurance rules ING-01; decisions ADR-002 Push lands files; pull goes straight to bronze through the lander steps.

solvedbuilt
Reruns that double the data
ING-02
A retried load appends the same rows again.A failed run is restarted halfway through a day.Every batch has a key; bronze writes replace by batch key, and a run restarts after the last batch the file log shows as loaded.

assurance rules DUP-02, IDEMP-01, ORCH-03.

Shift-left checks that guard it: SL-PURE-03.

solvedbuilt
Duplicate deliveries
ING-03
The same content arrives twice and is counted twice.Day 6, the partner file resent under a new name. (tested on day 6)The content checksum is the batch key; the landing area refuses a second write of the same delivery.

assurance rules DUP-01.

Shift-left checks that guard it: SL-SEM-04, SL-TEST-07, SL-TEST-09.

solvedbuilt
Partial or empty deliveries
ING-04
A half-written file or an empty batch is loaded as if complete.A lab file arrives with three rows on a day that normally has hundreds.A declared volume band per dataset; empty and out-of-band batches are held with the reason in the file log; every file ends with a recorded outcome.

assurance rules DQ-04, FILE-01.

solvedbuilt
Large history loads and backfills
ING-05
Loading years of history through the daily path overwhelms it, or the handover to daily loads loses or repeats a day.Backfilling a month of order lines.Reload and backfill are declared run types with a plan in the metadata store; the handover point is recorded.

assurance rules HIST-03, HIST-04.

solvedbuilt
Credentials in code
ING-06
Passwords in notebooks or repositories leak and cannot be rotated.Database and interface credentials for four sources.Secrets resolved at run time from the secret store, per environment; never in the repository.

assurance rules REL-05, REL-04.

Shift-left checks that guard it: SL-CLEAN-07, SL-SEC-06, SL-SECRET-01, SL-SECRET-02, SL-SECRET-03, SL-SETUP-04.

solvedbuilt
Schema drift at the door
ING-07
A changed column loads into the wrong place or as nulls.Day 7, the renamed lab column. (tested on day 7)The contract check stops the batch and names the column; new columns are kept aside or stop the batch, as the dataset declares.

assurance rules SCH-01, SCH-02.

Shift-left checks that guard it: SL-CLEAN-29, SL-COST-CODE-12.

assistedbuilt

What not to do

Common anti-patterns

An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.

Anti-patternWhy it hurtsDo this instead
Overwriting a file of the same name in landingIf a resend overwrites a file, we cannot tell what was loaded, and a replay gives a different answer.Give every delivery a unique name, list it in the manifest, and send corrections as new files.
Reading landing files from anything but the landerA job that reads raw files skips tokenising, contract checks and the file log, so its output has no lineage.Read bronze, which holds every record as received with full provenance.
Hard-coded settings in notebooksPaths, thresholds and keys buried in code differ between copies, and nobody can see or audit them.Put the value in YAML at the right level (platform, source, dataset or column) and read it from the metadata store.
A copy-pasted pipeline per tableA fix must be made fifty times, and the copies drift apart.Add the table as metadata to the common pipeline or harness.

FAQ

Questions people ask

What are the hardest problems in the ingestion layer?

Push and pull behave differently; Reruns that double the data; Duplicate deliveries; Partial or empty deliveries; Large history loads and backfills.

What is the most common mistake at ingestion?

Overwriting a file of the same name in landing. Give every delivery a unique name, list it in the manifest, and send corrections as new files.

Next

Where to go next

Next: Bronze

↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture