Home › Learn › Hard problems by layer › Ingestion
Hard problems by layer · 2 of 10
Ingestion: hard problems and anti-patterns
The hard challenges and common anti-patterns of ingestion in a data platform, with how each is solved and tested. This layer must bring every delivery in exactly once, with a recorded outcome, through one set of entry steps.
Where we are
Ingestion
What this layer must achieve. Bring every delivery in exactly once, with a recorded outcome, through one set of entry steps.
Handled by: lander, puller, metadata store, observability.
The hard part
7 challenges, and how they are solved
Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.
| Challenge | What goes wrong | Tyre-world example | How we solve it | Verdict | Status |
|---|---|---|---|---|---|
| Push and pull behave differently ING-01 | Files pushed by a source and data pulled by the platform go through different code, so the same problem is caught on one path and missed on the other. | Partner files are pushed; the order database is pulled. | Pushed files land and go through the lander; pulled data runs the same lander steps in the same process, straight to bronze. assurance rules ING-01; decisions ADR-002 Push lands files; pull goes straight to bronze through the lander steps. | solved | built |
| Reruns that double the data ING-02 | A retried load appends the same rows again. | A failed run is restarted halfway through a day. | Every batch has a key; bronze writes replace by batch key, and a run restarts after the last batch the file log shows as loaded. assurance rules DUP-02, IDEMP-01, ORCH-03. Shift-left checks that guard it: SL-PURE-03. | solved | built |
| Duplicate deliveries ING-03 | The same content arrives twice and is counted twice. | Day 6, the partner file resent under a new name. (tested on day 6) | The content checksum is the batch key; the landing area refuses a second write of the same delivery. assurance rules DUP-01. Shift-left checks that guard it: SL-SEM-04, SL-TEST-07, SL-TEST-09. | solved | built |
| Partial or empty deliveries ING-04 | A half-written file or an empty batch is loaded as if complete. | A lab file arrives with three rows on a day that normally has hundreds. | A declared volume band per dataset; empty and out-of-band batches are held with the reason in the file log; every file ends with a recorded outcome. assurance rules DQ-04, FILE-01. | solved | built |
| Large history loads and backfills ING-05 | Loading years of history through the daily path overwhelms it, or the handover to daily loads loses or repeats a day. | Backfilling a month of order lines. | Reload and backfill are declared run types with a plan in the metadata store; the handover point is recorded. assurance rules HIST-03, HIST-04. | solved | built |
| Credentials in code ING-06 | Passwords in notebooks or repositories leak and cannot be rotated. | Database and interface credentials for four sources. | Secrets resolved at run time from the secret store, per environment; never in the repository. assurance rules REL-05, REL-04. Shift-left checks that guard it: SL-CLEAN-07, SL-SEC-06, SL-SECRET-01, SL-SECRET-02, SL-SECRET-03, SL-SETUP-04. | solved | built |
| Schema drift at the door ING-07 | A changed column loads into the wrong place or as nulls. | Day 7, the renamed lab column. (tested on day 7) | The contract check stops the batch and names the column; new columns are kept aside or stop the batch, as the dataset declares. assurance rules SCH-01, SCH-02. Shift-left checks that guard it: SL-CLEAN-29, SL-COST-CODE-12. | assisted | built |
What not to do
Common anti-patterns
An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.
| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Overwriting a file of the same name in landing | If a resend overwrites a file, we cannot tell what was loaded, and a replay gives a different answer. | Give every delivery a unique name, list it in the manifest, and send corrections as new files. |
| Reading landing files from anything but the lander | A job that reads raw files skips tokenising, contract checks and the file log, so its output has no lineage. | Read bronze, which holds every record as received with full provenance. |
| Hard-coded settings in notebooks | Paths, thresholds and keys buried in code differ between copies, and nobody can see or audit them. | Put the value in YAML at the right level (platform, source, dataset or column) and read it from the metadata store. |
| A copy-pasted pipeline per table | A fix must be made fifty times, and the copies drift apart. | Add the table as metadata to the common pipeline or harness. |
FAQ
Questions people ask
What are the hardest problems in the ingestion layer?
Push and pull behave differently; Reruns that double the data; Duplicate deliveries; Partial or empty deliveries; Large history loads and backfills.
What is the most common mistake at ingestion?
Overwriting a file of the same name in landing. Give every delivery a unique name, list it in the manifest, and send corrections as new files.
Next
Where to go next
Next: Bronze
↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture