Home › Learn › Hard problems by layer › Bronze
Hard problems by layer · 3 of 10
Bronze: hard problems and anti-patterns
The hard challenges and common anti-patterns of bronze in a data platform, with how each is solved and tested. This layer must keep every record exactly as delivered, traceable to its delivery, so anything later can be rebuilt.
Where we are
Bronze
What this layer must achieve. Keep every record exactly as delivered, traceable to its delivery, so anything later can be rebuilt.
Handled by: lander, metadata store.
The hard part
5 challenges, and how they are solved
Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.
| Challenge | What goes wrong | Tyre-world example | How we solve it | Verdict | Status |
|---|---|---|---|---|---|
| Changing data on the way in BRZ-01 | Cleaning or removing duplicates in bronze destroys the evidence needed to rebuild or audit. | Stream replays and late readings are kept and flagged, not dropped. | Bronze is append-only and raw; cleaning starts in silver. assurance rules RAW-01. | solved | built |
| Rows that cannot be traced BRZ-02 | Nobody can say which file or run a row came from. | Every bronze row carries its batch key, run id, source file and window. | Load metadata on every row, and every run recorded with its configuration. assurance rules OPS-03, CFG-03. | solved | built |
| Personal data stored in clear text BRZ-03 | The rawest layer becomes the biggest privacy risk. | Dealer contact columns. | Restricted columns are tokenised before the first write; the token key lives outside code. assurance rules PII-01. | solved | built |
| Replays and late events BRZ-04 | Dropping them hides problems; keeping them unflagged doubles counts downstream. | Day 3, 188 readings up to 90 seconds late; day 13, a 45-minute stream replay. (tested on day 13) | Kept as delivered with arrival time and position; silver applies event time and de-duplication. assurance rules DUP-04, LATE-03. | solved | built |
| Table layout decay BRZ-05 | Thousands of small files make every read slow. | Many small lab files a day. | Automatic clustering and maintenance declared once in the binding, not hand-written jobs. assurance rules PERF-05; decisions ADR-011 Table layout: automatic clustering and automatic maintenance. Shift-left checks that guard it: SL-COST-CODE-09, SL-COST-CODE-15, SL-COST-CODE-16, SL-COST-PLAN-04. | solved | built |
What not to do
Common anti-patterns
An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.
| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Cleaning or removing duplicates in bronze | A cleaned bronze no longer shows what the source sent, so a cleaning bug can never be undone by replay. | Keep bronze raw and do every cleaning step in the bronze-to-silver engine. |
| Personal data in clear text in bronze | Bronze is kept for a long time as the source for replays, so anything personal stored there in clear text stays exposed. | Tokenise in the lander and keep the clear value only in the token vault. |
| Partitioning and hand-written OPTIMIZE jobs | Partitions picked by hand often create many small files, and maintenance jobs are extra code to run. | Use automatic clustering and leave upkeep to the platform's automatic optimization. |
FAQ
Questions people ask
What are the hardest problems in the bronze layer?
Changing data on the way in; Rows that cannot be traced; Personal data stored in clear text; Replays and late events; Table layout decay.
What is the most common mistake at bronze?
Cleaning or removing duplicates in bronze. Keep bronze raw and do every cleaning step in the bronze-to-silver engine.
Next
Where to go next
Next: Bronze to silver
↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture