Home › Learn › Hard problems by layer › Silver
Hard problems by layer · 5 of 10
Silver: hard problems and anti-patterns
The hard challenges and common anti-patterns of silver in a data platform, with how each is solved and tested. This layer must hold each source's truth, cleaned and with history, without mixing sources yet.
Where we are
Silver
What this layer must achieve. Hold each source's truth, cleaned and with history, without mixing sources yet.
Handled by: silver engine, data expectation services, observability.
The hard part
5 challenges, and how they are solved
Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.
| Challenge | What goes wrong | Tyre-world example | How we solve it | Verdict | Status |
|---|---|---|---|---|---|
| Conforming too early SLV-01 | Joining sources in silver ties them together, so one late source blocks another and changes ripple everywhere. | Orders and dealers stay separate in silver. | Silver is source-aligned; conforming and joining happen in gold. Data flows one way through the layers. assurance rules FLOW-01; decisions ADR-001 Silver is source-aligned; conforming happens in gold. | solved | built |
| The wrong kind of history SLV-02 | History kept where it is not needed, or lost where it is. | Dealer region keeps history; a phone number is simply overwritten. | History kind per dataset (and per attribute where needed) is declared; the engine applies it. assurance rules SCD-02; decisions ADR-016 History kind follows dataset kind; deletes are soft. Shift-left checks that guard it: SL-TEST-09. | assisted | built |
| Late stream events SLV-03 | Late readings land in the wrong hour or are dropped. | Day 3, 188 late readings. (tested on day 3) | Event time with a declared late allowance; readings counted in their event hour. assurance rules LATE-03. | solved | built |
| Drift nobody sees SLV-04 | A column's values shift slowly and nothing notices until a report is wrong. | Scrap reason codes changing mix. | Every dataset is profiled on load and drift is visible and alertable. assurance rules PROF-01. | solved | designed |
| Writes that are not repeatable SLV-05 | A rerun adds rows or versions. | Rerunning day 8. | Merge on a row key computed from the data; the bookmark records which bronze batches each run consumed. assurance rules IDEMP-01, DUP-02. Shift-left checks that guard it: SL-PURE-03. | solved | built |
What not to do
Common anti-patterns
An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.
| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Joining sources or adding business logic in silver | Silver is source-aligned; mixing sources there blurs ownership and breaks replays per source. | Combine sources in gold, inside a fact's pure build() function. |
| Chaining layers by clock time | Starting silver at 03:00 because bronze usually ends by 02:30 breaks the first night bronze runs late, and nobody notices. | Trigger on data readiness: the batch-ready marker for silver, input bookmarks for gold. |
| Partitioning and hand-written OPTIMIZE jobs | Partitions picked by hand often create many small files, and maintenance jobs are extra code to run. | Use automatic clustering and leave upkeep to the platform's automatic optimization. |
FAQ
Questions people ask
What are the hardest problems in the silver layer?
Conforming too early; The wrong kind of history; Late stream events; Drift nobody sees; Writes that are not repeatable.
What is the most common mistake at silver?
Joining sources or adding business logic in silver. Combine sources in gold, inside a fact's pure build() function.
Next
Where to go next
Next: Silver to gold
↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture