Home › Learn › Hard problems by layer › Silver

Hard problems by layer · 5 of 10

Silver: hard problems and anti-patterns

The hard challenges and common anti-patterns of silver in a data platform, with how each is solved and tested. This layer must hold each source's truth, cleaned and with history, without mixing sources yet.

↑ All layersSource systemsIngestionBronzeBronze to silverSilverSilver to goldGoldGold to semanticSemantic layerSemantic to UIAll challenges

Where we are

Silver

SourcesystemsIngestionBronzeBronze tosilverSilverSilver togoldGoldGold tosemanticSemanticlayerSemantic toUI
The ten layers and hops from source to screen. The dark box is this page.

What this layer must achieve. Hold each source's truth, cleaned and with history, without mixing sources yet.

Handled by: silver engine, data expectation services, observability.

The hard part

5 challenges, and how they are solved

4 solved1 assistedStatus: 4 built, 1 designed
Solved means the engine does it once the decision is declared, and a test proves it. Assisted means the tool suggests and checks, and a person confirms. People decide means a business decision the tool records and enforces.

Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.

ChallengeWhat goes wrongTyre-world exampleHow we solve itVerdictStatus
Conforming too early
SLV-01
Joining sources in silver ties them together, so one late source blocks another and changes ripple everywhere.Orders and dealers stay separate in silver.Silver is source-aligned; conforming and joining happen in gold. Data flows one way through the layers.

assurance rules FLOW-01; decisions ADR-001 Silver is source-aligned; conforming happens in gold.

solvedbuilt
The wrong kind of history
SLV-02
History kept where it is not needed, or lost where it is.Dealer region keeps history; a phone number is simply overwritten.History kind per dataset (and per attribute where needed) is declared; the engine applies it.

assurance rules SCD-02; decisions ADR-016 History kind follows dataset kind; deletes are soft.

Shift-left checks that guard it: SL-TEST-09.

assistedbuilt
Late stream events
SLV-03
Late readings land in the wrong hour or are dropped.Day 3, 188 late readings. (tested on day 3)Event time with a declared late allowance; readings counted in their event hour.

assurance rules LATE-03.

solvedbuilt
Drift nobody sees
SLV-04
A column's values shift slowly and nothing notices until a report is wrong.Scrap reason codes changing mix.Every dataset is profiled on load and drift is visible and alertable.

assurance rules PROF-01.

solveddesigned
Writes that are not repeatable
SLV-05
A rerun adds rows or versions.Rerunning day 8.Merge on a row key computed from the data; the bookmark records which bronze batches each run consumed.

assurance rules IDEMP-01, DUP-02.

Shift-left checks that guard it: SL-PURE-03.

solvedbuilt

What not to do

Common anti-patterns

An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.

Anti-patternWhy it hurtsDo this instead
Joining sources or adding business logic in silverSilver is source-aligned; mixing sources there blurs ownership and breaks replays per source.Combine sources in gold, inside a fact's pure build() function.
Chaining layers by clock timeStarting silver at 03:00 because bronze usually ends by 02:30 breaks the first night bronze runs late, and nobody notices.Trigger on data readiness: the batch-ready marker for silver, input bookmarks for gold.
Partitioning and hand-written OPTIMIZE jobsPartitions picked by hand often create many small files, and maintenance jobs are extra code to run.Use automatic clustering and leave upkeep to the platform's automatic optimization.

FAQ

Questions people ask

What are the hardest problems in the silver layer?

Conforming too early; The wrong kind of history; Late stream events; Drift nobody sees; Writes that are not repeatable.

What is the most common mistake at silver?

Joining sources or adding business logic in silver. Combine sources in gold, inside a fact's pure build() function.

Next

Where to go next

Next: Silver to gold

↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture