Home › Learn › Hard problems by layer › Source systems
Hard problems by layer · 1 of 10
Source systems: hard problems and anti-patterns
The hard challenges and common anti-patterns of source systems in a data platform, with how each is solved and tested. This layer must know exactly what each source promises and how it changes, before reading a single row.
Where we are
Source systems
What this layer must achieve. Know exactly what each source promises and how it changes, before reading a single row.
Handled by: data contracts, metadata store, puller, lander.
The hard part
7 challenges, and how they are solved
Every example comes from the tyre world, a synthetic tyre company whose fourteen simulated days carry planted problems with known right answers, so each fix is tested, not just described.
| Challenge | What goes wrong | Tyre-world example | How we solve it | Verdict | Status |
|---|---|---|---|---|---|
| No contract with the source SRC-01 | The source team renames or drops a column without telling anyone; loads break or quietly load the wrong thing. | Day 7, one plant's lab system renames defect_code to defect. (tested on day 7) | One data contract per source dataset (columns, types, keys, personal-data class, freshness), checked at every load; a change outside the contract stops the batch and names the column. assurance rules SCH-01, SCH-02, GATE-02. Shift-left checks that guard it: SL-CLEAN-29, SL-COST-CODE-12. | assisted | built |
| Deletes are not signalled SRC-02 | Rows vanish at the source and nothing downstream knows, so deleted orders keep counting. | Day 4, five order lines deleted in the order system; day 9, a nightly full copy arrives 40% short. (tested on day 4) | Change feeds carry a delete marker; full copies are compared with what silver holds now, and a copy that would delete more than the declared share is held, not loaded. assurance rules DEL-01, DEL-04; decisions ADR-016 History kind follows dataset kind; deletes are soft. | solved | built |
| Full copies versus change feeds SRC-03 | Reading a full copy every run is slow and hides deletes; reading a change feed without a starting copy misses history. | Materials arrive as a nightly full copy; order lines as a change feed. | Each dataset declares how it is extracted (full copy, change feed, stream, paged interface) and how history is handed over to incremental loads. assurance rules ING-01, HIST-03. | assisted | built |
| Time zones and business dates SRC-04 | Timestamps without a zone, or a day cut at midnight UTC, put production on the wrong business day. | Plants in India and Hungary record press readings in local time. | Each source declares its time zone; the business date per plant is computed once in gold from a shared calendar. assurance rules LATE-04. Shift-left checks that guard it: SL-CLEAN-16. | solved | gap |
| Paged interfaces and rate limits SRC-05 | A page fails halfway or the interface throttles, and the load either stops or skips records. | The dealer interface serves 200 records a page; a new dealer appears late (days 11 and 12). (tested on day 11) | One batch per page with a cursor bookmark; retries only for passing failures; a reload of a paged interface is logged as a no-op because bronze is the replay point. assurance rules ORCH-02, IDEMP-01; decisions ADR-002 Push lands files; pull goes straight to bronze through the lander steps. Shift-left checks that guard it: SL-PURE-03. | solved | built |
| Files resent or late SRC-06 | A partner sends the same file twice under a new name, or sends last month's file weeks late. | Day 5, dealer D-0042's October file arrives late; day 6, it is resent. (tested on day 6) | The batch key is the file's content checksum, so a resend is refused; each file drop declares how many deliveries a period expects, so lateness is visible from the due day. assurance rules DUP-01, LATE-01. Shift-left checks that guard it: SL-CLEAN-16, SL-SEM-04, SL-TEST-07, SL-TEST-09. | solved | built |
| Personal data at the source SRC-07 | Contact details flow into every layer in clear text. | Dealer contact email and phone, lab inspector ids. | The contract marks each column's personal-data class; restricted columns are tokenised before they are first stored. assurance rules PII-01, SEC-01. | assisted | built |
What not to do
Common anti-patterns
An anti-pattern is a habit that looks reasonable and keeps causing the same damage. These are the ones seen most often at this layer.
| Anti-pattern | Why it hurts | Do this instead |
|---|---|---|
| Trusting the source to announce its changes | Changes arrive unannounced; the first sign is a wrong number in a report. | Check every load against a written contract and stop on a change. |
| Copying the whole source every run when a change feed exists | Slow and costly, and deletes stay invisible unless every copy is compared. | Declare the extraction kind per dataset; use change feeds where offered and guard full copies. |
FAQ
Questions people ask
What are the hardest problems in the source systems layer?
No contract with the source; Deletes are not signalled; Full copies versus change feeds; Time zones and business dates; Paged interfaces and rate limits.
What is the most common mistake at source systems?
Trusting the source to announce its changes. Check every load against a written contract and stop on a change.
Next
Where to go next
Next: Ingestion
↑ All layers · All 69 challenges, filterable · Shift left: fail fast on data code · Medallion architecture