Home › Learn › Hard problems by layer › Source systems

Hard problems by layer · 1 of 10

Source systems: hard problems and anti-patterns

The hard challenges and common anti-patterns of source systems in a data platform, with how each is solved and tested. This layer must know exactly what each source promises and how it changes, before reading a single row.

↑ All layersSource systemsIngestionBronzeBronze to silverSilverSilver to goldGoldGold to semanticSemantic layerSemantic to UIAll challenges

Where we are

Source systems

SourcesystemsIngestionBronzeBronze tosilverSilverSilver togoldGoldGold tosemanticSemanticlayerSemantic toUI
The ten layers and hops from source to screen. The dark box is this page.

What this layer must achieve. Know exactly what each source promises and how it changes, before reading a single row.

Handled by: data contracts, metadata store, puller, lander.

The hard part

7 challenges, and how they are solved

4 solved3 assistedStatus: 6 built, 1 gap
Solved means the engine does it once the decision is declared, and a test proves it. Assisted means the tool suggests and checks, and a person confirms. People decide means a business decision the tool records and enforces.

Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.

ChallengeWhat goes wrongTyre-world exampleHow we solve itVerdictStatus
No contract with the source
SRC-01
The source team renames or drops a column without telling anyone; loads break or quietly load the wrong thing.Day 7, one plant's lab system renames defect_code to defect. (tested on day 7)One data contract per source dataset (columns, types, keys, personal-data class, freshness), checked at every load; a change outside the contract stops the batch and names the column.

assurance rules SCH-01, SCH-02, GATE-02.

Shift-left checks that guard it: SL-CLEAN-29, SL-COST-CODE-12.

assistedbuilt
Deletes are not signalled
SRC-02
Rows vanish at the source and nothing downstream knows, so deleted orders keep counting.Day 4, five order lines deleted in the order system; day 9, a nightly full copy arrives 40% short. (tested on day 4)Change feeds carry a delete marker; full copies are compared with what silver holds now, and a copy that would delete more than the declared share is held, not loaded.

assurance rules DEL-01, DEL-04; decisions ADR-016 History kind follows dataset kind; deletes are soft.

solvedbuilt
Full copies versus change feeds
SRC-03
Reading a full copy every run is slow and hides deletes; reading a change feed without a starting copy misses history.Materials arrive as a nightly full copy; order lines as a change feed.Each dataset declares how it is extracted (full copy, change feed, stream, paged interface) and how history is handed over to incremental loads.

assurance rules ING-01, HIST-03.

assistedbuilt
Time zones and business dates
SRC-04
Timestamps without a zone, or a day cut at midnight UTC, put production on the wrong business day.Plants in India and Hungary record press readings in local time.Each source declares its time zone; the business date per plant is computed once in gold from a shared calendar.

assurance rules LATE-04.

Shift-left checks that guard it: SL-CLEAN-16.

solvedgap
Paged interfaces and rate limits
SRC-05
A page fails halfway or the interface throttles, and the load either stops or skips records.The dealer interface serves 200 records a page; a new dealer appears late (days 11 and 12). (tested on day 11)One batch per page with a cursor bookmark; retries only for passing failures; a reload of a paged interface is logged as a no-op because bronze is the replay point.

assurance rules ORCH-02, IDEMP-01; decisions ADR-002 Push lands files; pull goes straight to bronze through the lander steps.

Shift-left checks that guard it: SL-PURE-03.

solvedbuilt
Files resent or late
SRC-06
A partner sends the same file twice under a new name, or sends last month's file weeks late.Day 5, dealer D-0042's October file arrives late; day 6, it is resent. (tested on day 6)The batch key is the file's content checksum, so a resend is refused; each file drop declares how many deliveries a period expects, so lateness is visible from the due day.

assurance rules DUP-01, LATE-01.

Shift-left checks that guard it: SL-CLEAN-16, SL-SEM-04, SL-TEST-07, SL-TEST-09.

solvedbuilt
Personal data at the source
SRC-07
Contact details flow into every layer in clear text.Dealer contact email and phone, lab inspector ids.The contract marks each column's personal-data class; restricted columns are tokenised before they are first stored.

assurance rules PII-01, SEC-01.

Shift-left checks that guard it: SL-SEC-07, SL-SEC-09.

assistedbuilt

What not to do

Common anti-patterns

An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.

Anti-patternWhy it hurtsDo this instead
Trusting the source to announce its changesChanges arrive unannounced; the first sign is a wrong number in a report.Check every load against a written contract and stop on a change.
Copying the whole source every run when a change feed existsSlow and costly, and deletes stay invisible unless every copy is compared.Declare the extraction kind per dataset; use change feeds where offered and guard full copies.

FAQ

Questions people ask

What are the hardest problems in the source systems layer?

No contract with the source; Deletes are not signalled; Full copies versus change feeds; Time zones and business dates; Paged interfaces and rate limits.

What is the most common mistake at source systems?

Trusting the source to announce its changes. Check every load against a written contract and stop on a change.

Next

Where to go next

Next: Ingestion

↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture