Home › Learn › Hard problems by layer › Bronze to silver

Hard problems by layer · 4 of 10

Bronze to silver: hard problems and anti-patterns

The hard challenges and common anti-patterns of bronze to silver in a data platform, with how each is solved and tested. This hop must make one source faithful (typed, de-duplicated, history kept, deletes marked) with the same steps for every source.

↑ All layersSource systemsIngestionBronzeBronze to silverSilverSilver to goldGoldGold to semanticSemantic layerSemantic to UIAll challenges

Where we are

Bronze to silver

SourcesystemsIngestionBronzeBronze tosilverSilverSilver togoldGoldGold tosemanticSemanticlayerSemantic toUI
The ten layers and hops from source to screen. The dark box is this page.

What this hop must achieve. Make one source faithful (typed, de-duplicated, history kept, deletes marked) with the same steps for every source.

This is a hop: the step that moves data from one layer to the next. Handled by: silver engine, data expectation services, metadata store.

The hard part

8 challenges, and how they are solved

8 solvedStatus: 6 built, 2 designed
Solved means the engine does it once the decision is declared, and a test proves it. Assisted means the tool suggests and checks, and a person confirms. People decide means a business decision the tool records and enforces.

Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.

ChallengeWhat goes wrongTyre-world exampleHow we solve itVerdictStatus
A different pipeline per table
B2S-01
Every table gets its own hand-written code, so each fixes the same problems differently.Nine tyre datasets, one engine.One engine with thirteen standard stages driven by settings; stages are pure functions and reads and writes happen only at the edges.

assurance rules ENG-01, PURE-01; decisions ADR-007 Thirteen standard stages from bronze to silver; ADR-013 IO only at the edges; stages are pure functions.

Shift-left checks that guard it: SL-PURE-06.

solvedbuilt
Changes arriving out of order
B2S-02
An older change applied after a newer one overwrites the current value.Day 8, a promised-date change from earlier arrives after a newer one. (tested on day 8)History is ordered by the source's own sequence, never by arrival; the late change is inserted in its place.

assurance rules SCD-02.

Shift-left checks that guard it: SL-TEST-09.

solvedbuilt
Deletes
B2S-03
Deleted rows are either lost from history or still counted.Day 4, five order lines deleted. (tested on day 4)Deletes are soft (marked, kept joinable) and flow through to derived tables.

assurance rules DEL-01, DEL-03; decisions ADR-016 History kind follows dataset kind; deletes are soft.

solvedbuilt
Duplicates inside the source and replayed events
B2S-04
The same reading or row counted twice.Day 13, 19,440 replayed readings. (tested on day 13)De-duplicate by the declared key (for example the reading id) with a declared rule for which copy wins.

assurance rules DUP-03, DUP-04.

Shift-left checks that guard it: SL-CLEAN-32.

solvedbuilt
A bad full copy
B2S-05
A short copy marks hundreds of records deleted.Day 9, 720 of 1,800 materials missing. (tested on day 9)The mass-delete guard compares the copy with silver's current keys and holds the batch above the declared share.

assurance rules DEL-04.

solvedbuilt
Bad values
B2S-06
Values that fail a rule are silently nulled or dropped, or one bad row stops the whole load.An impossible cure temperature in a press reading.Rules in configuration with a severity that sets the action (warn, quarantine, block); quarantined rows are reviewed and released through the same checks.

assurance rules DQ-02, DQ-03, SEV-01, QUAR-01; decisions ADR-009 Quality severity sets the action.

solveddesigned
A bad load that got through
B2S-07
Found days later, after good loads built on it; restoring old versions throws the good loads away.The day-9 short copy slipping past the guard.Void the batch, rebuild the silver table from bronze, rerun the gold windows that use it; recon and known answers prove it.

assurance rules HIST-04, TEST-05; decisions ADR-025 Recover a bad load by voiding it and rerunning from bronze.

Shift-left checks that guard it: SL-COST-CODE-21, SL-PURE-03.

solveddesigned
Columns lost on the way
B2S-08
A hand-picked column list silently drops a column a consumer needs.A new contract column never reaches silver.Every contract column reaches silver unless a selection is declared.

assurance rules PROJ-01; decisions ADR-008 Silver takes all columns by default.

solvedbuilt

What not to do

Common anti-patterns

An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this hop.

Anti-patternWhy it hurtsDo this instead
Free-form SQL in bronze to silverHand-written SQL per table brings back the copy-paste pipelines the engine replaced, and it cannot be unit-tested.Declare the need as a setting; if no setting fits, add a tested stage option to the engine for everyone.
Silently dropping or nulling bad valuesRows that vanish quietly make gold totals wrong with no trace of why.Quarantine the row with its reason and let a steward release it once fixed.
Joining sources or adding business logic in silverSilver is source-aligned; mixing sources there blurs ownership and breaks replays per source.Combine sources in gold, inside a fact's pure build() function.
A copy-pasted pipeline per tableA fix must be made fifty times, and the copies drift apart.Add the table as metadata to the common pipeline or harness.

FAQ

Questions people ask

What are the hardest problems in the bronze to silver step?

A different pipeline per table; Changes arriving out of order; Deletes; Duplicates inside the source and replayed events; A bad full copy.

What is the most common mistake at bronze to silver?

Free-form SQL in bronze to silver. Declare the need as a setting; if no setting fits, add a tested stage option to the engine for everyone.

Next

Where to go next

Next: Silver

↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture