Home › Learn › Hard problems by layer › Bronze to silver
Hard problems by layer · 4 of 10
Bronze to silver: hard problems and anti-patterns
The hard challenges and common anti-patterns of bronze to silver in a data platform, with how each is solved and tested. This hop must make one source faithful (typed, de-duplicated, history kept, deletes marked) with the same steps for every source.
Where we are
Bronze to silver
What this hop must achieve. Make one source faithful (typed, de-duplicated, history kept, deletes marked) with the same steps for every source.
This is a hop: the step that moves data from one layer to the next. Handled by: silver engine, data expectation services, metadata store.
The hard part
8 challenges, and how they are solved
Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.
| Challenge | What goes wrong | Tyre-world example | How we solve it | Verdict | Status |
|---|---|---|---|---|---|
| A different pipeline per table B2S-01 | Every table gets its own hand-written code, so each fixes the same problems differently. | Nine tyre datasets, one engine. | One engine with thirteen standard stages driven by settings; stages are pure functions and reads and writes happen only at the edges. assurance rules ENG-01, PURE-01; decisions ADR-007 Thirteen standard stages from bronze to silver; ADR-013 IO only at the edges; stages are pure functions. Shift-left checks that guard it: SL-PURE-06. | solved | built |
| Changes arriving out of order B2S-02 | An older change applied after a newer one overwrites the current value. | Day 8, a promised-date change from earlier arrives after a newer one. (tested on day 8) | History is ordered by the source's own sequence, never by arrival; the late change is inserted in its place. assurance rules SCD-02. Shift-left checks that guard it: SL-TEST-09. | solved | built |
| Deletes B2S-03 | Deleted rows are either lost from history or still counted. | Day 4, five order lines deleted. (tested on day 4) | Deletes are soft (marked, kept joinable) and flow through to derived tables. assurance rules DEL-01, DEL-03; decisions ADR-016 History kind follows dataset kind; deletes are soft. | solved | built |
| Duplicates inside the source and replayed events B2S-04 | The same reading or row counted twice. | Day 13, 19,440 replayed readings. (tested on day 13) | De-duplicate by the declared key (for example the reading id) with a declared rule for which copy wins. assurance rules DUP-03, DUP-04. Shift-left checks that guard it: SL-CLEAN-32. | solved | built |
| A bad full copy B2S-05 | A short copy marks hundreds of records deleted. | Day 9, 720 of 1,800 materials missing. (tested on day 9) | The mass-delete guard compares the copy with silver's current keys and holds the batch above the declared share. assurance rules DEL-04. | solved | built |
| Bad values B2S-06 | Values that fail a rule are silently nulled or dropped, or one bad row stops the whole load. | An impossible cure temperature in a press reading. | Rules in configuration with a severity that sets the action (warn, quarantine, block); quarantined rows are reviewed and released through the same checks. assurance rules DQ-02, DQ-03, SEV-01, QUAR-01; decisions ADR-009 Quality severity sets the action. | solved | designed |
| A bad load that got through B2S-07 | Found days later, after good loads built on it; restoring old versions throws the good loads away. | The day-9 short copy slipping past the guard. | Void the batch, rebuild the silver table from bronze, rerun the gold windows that use it; recon and known answers prove it. assurance rules HIST-04, TEST-05; decisions ADR-025 Recover a bad load by voiding it and rerunning from bronze. Shift-left checks that guard it: SL-COST-CODE-21, SL-PURE-03. | solved | designed |
| Columns lost on the way B2S-08 | A hand-picked column list silently drops a column a consumer needs. | A new contract column never reaches silver. | Every contract column reaches silver unless a selection is declared. assurance rules PROJ-01; decisions ADR-008 Silver takes all columns by default. | solved | built |
What not to do
Common anti-patterns
An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this hop.
| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Free-form SQL in bronze to silver | Hand-written SQL per table brings back the copy-paste pipelines the engine replaced, and it cannot be unit-tested. | Declare the need as a setting; if no setting fits, add a tested stage option to the engine for everyone. |
| Silently dropping or nulling bad values | Rows that vanish quietly make gold totals wrong with no trace of why. | Quarantine the row with its reason and let a steward release it once fixed. |
| Joining sources or adding business logic in silver | Silver is source-aligned; mixing sources there blurs ownership and breaks replays per source. | Combine sources in gold, inside a fact's pure build() function. |
| A copy-pasted pipeline per table | A fix must be made fifty times, and the copies drift apart. | Add the table as metadata to the common pipeline or harness. |
FAQ
Questions people ask
What are the hardest problems in the bronze to silver step?
A different pipeline per table; Changes arriving out of order; Deletes; Duplicates inside the source and replayed events; A bad full copy.
What is the most common mistake at bronze to silver?
Free-form SQL in bronze to silver. Declare the need as a setting; if no setting fits, add a tested stage option to the engine for everyone.
Next
Where to go next
Next: Silver
↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture