Home › Learn › Hard problems by layer › Bronze

Hard problems by layer · 3 of 10

Bronze: hard problems and anti-patterns

The hard challenges and common anti-patterns of bronze in a data platform, with how each is solved and tested. This layer must keep every record exactly as delivered, traceable to its delivery, so anything later can be rebuilt.

↑ All layersSource systemsIngestionBronzeBronze to silverSilverSilver to goldGoldGold to semanticSemantic layerSemantic to UIAll challenges

Where we are

Bronze

SourcesystemsIngestionBronzeBronze tosilverSilverSilver togoldGoldGold tosemanticSemanticlayerSemantic toUI
The ten layers and hops from source to screen. The dark box is this page.

What this layer must achieve. Keep every record exactly as delivered, traceable to its delivery, so anything later can be rebuilt.

Handled by: lander, metadata store.

The hard part

5 challenges, and how they are solved

5 solvedStatus: 5 built
Solved means the engine does it once the decision is declared, and a test proves it. Assisted means the tool suggests and checks, and a person confirms. People decide means a business decision the tool records and enforces.

Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.

ChallengeWhat goes wrongTyre-world exampleHow we solve itVerdictStatus
Changing data on the way in
BRZ-01
Cleaning or removing duplicates in bronze destroys the evidence needed to rebuild or audit.Stream replays and late readings are kept and flagged, not dropped.Bronze is append-only and raw; cleaning starts in silver.

assurance rules RAW-01.

solvedbuilt
Rows that cannot be traced
BRZ-02
Nobody can say which file or run a row came from.Every bronze row carries its batch key, run id, source file and window.Load metadata on every row, and every run recorded with its configuration.

assurance rules OPS-03, CFG-03.

solvedbuilt
Personal data stored in clear text
BRZ-03
The rawest layer becomes the biggest privacy risk.Dealer contact columns.Restricted columns are tokenised before the first write; the token key lives outside code.

assurance rules PII-01.

solvedbuilt
Replays and late events
BRZ-04
Dropping them hides problems; keeping them unflagged doubles counts downstream.Day 3, 188 readings up to 90 seconds late; day 13, a 45-minute stream replay. (tested on day 13)Kept as delivered with arrival time and position; silver applies event time and de-duplication.

assurance rules DUP-04, LATE-03.

solvedbuilt
Table layout decay
BRZ-05
Thousands of small files make every read slow.Many small lab files a day.Automatic clustering and maintenance declared once in the binding, not hand-written jobs.

assurance rules PERF-05; decisions ADR-011 Table layout: automatic clustering and automatic maintenance.

Shift-left checks that guard it: SL-COST-CODE-09, SL-COST-CODE-15, SL-COST-CODE-16, SL-COST-PLAN-04.

solvedbuilt

What not to do

Common anti-patterns

An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.

Anti-patternWhy it hurtsDo this instead
Cleaning or removing duplicates in bronzeA cleaned bronze no longer shows what the source sent, so a cleaning bug can never be undone by replay.Keep bronze raw and do every cleaning step in the bronze-to-silver engine.
Personal data in clear text in bronzeBronze is kept for a long time as the source for replays, so anything personal stored there in clear text stays exposed.Tokenise in the lander and keep the clear value only in the token vault.
Partitioning and hand-written OPTIMIZE jobsPartitions picked by hand often create many small files, and maintenance jobs are extra code to run.Use automatic clustering and leave upkeep to the platform's automatic optimization.

FAQ

Questions people ask

What are the hardest problems in the bronze layer?

Changing data on the way in; Rows that cannot be traced; Personal data stored in clear text; Replays and late events; Table layout decay.

What is the most common mistake at bronze?

Cleaning or removing duplicates in bronze. Keep bronze raw and do every cleaning step in the bronze-to-silver engine.

Next

Where to go next

Next: Bronze to silver

↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture