Data PlatformZoom level 2
Planted incidents: eleven faults the platform must catch
A planted incident is a fault put into the simulated company's data on purpose, on a known day. A dealer's file arrives four days late. A file is sent twice. The lab renames a column. Each one is something real sources do.
Clean data proves that the platform works when nothing goes wrong. Incidents prove the part that matters more: that it notices when something does, handles it at the right layer, and stays quiet on a clean day.
How an incident is declared
Every incident is one entry in the world file's calendar. The entry names the day, the dataset, the exact values, the detector expected to fire, and the oracle: one sentence that says what a correct platform must hold afterwards. A detector here is the id of a rule in the specification catalog, such as DEL-01 for deletes at the source.
data-platform-catalog/slices/02-tyre/world.yaml:117One calendar entry in full. - id: sellout-file-late
day: 5
story: late_data
kind: late_delivery
dataset: partner.dealer_sellout
does: Dealer D-0042's October file, due on day 1, arrives on day 5.
params: {dealer: D-0042, month: "2026-10", due_day: 1}
expect:
detector: LATE-01
handled_by: [DPC-0080, DPC-0059, DPC-0079]
oracle: October for D-0042 is published flagged incomplete until day 5, then rebuilt, marked complete and logged as a restatementThe generator reads the values from the params block and keeps no table of its own. When the plain sentence and the block disagree, the block wins.
The four verdicts
| Verdict | Meaning |
|---|---|
| handled | The layer did what the rule asks. |
| flagged | The layer noticed the condition and recorded it. A later layer applies it. |
| deferred | Not this layer's rule. The layer only loads the rows. |
| missed | The layer should have caught it and did not. This fails the test. |
Every incident
The table has one row per incident. Day is the day in the world's own 14-day calendar. Compressed is the day in the compressed 3-day calendar. Detector is the rule id the world file names. The last two columns say what each layer does with it.
| Incident | Day | Compressed | What happens | Detector | Ingestion | Where it is handled |
|---|---|---|---|---|---|---|
readings-out-of-order | 3 | 2 | For two hours, 4% of readings from 6 presses arrive up to 90 seconds late. | LATE-03 | flagged: late readings counted | Silver places each reading in the hour of its event time. |
order-lines-deleted | 4 | 3 | 5 cancelled order lines from day 1 are hard-deleted at the source. | DEL-01 | flagged: delete rows counted | Silver keeps the 5 keys and marks them deleted. Gold no longer counts them. |
sellout-file-late | 5 | 2 | Dealer D-0042's October file, due on day 1, arrives on day 5. | LATE-01 | deferred | Gold publishes October flagged incomplete, then rebuilds it and logs a restatement. Freshness shows the dealer's bookmark lagging. |
sellout-file-resent | 6 | 3 | The same file is sent again, byte for byte, under a new name. | DUP-01 | handled: the batch is skipped as a duplicate | Ingestion. Gold confirms nothing changed. |
lab-column-renamed | 7 | 2 | The lab renames defect_code to defect in one plant's files from 10:00. | SCH-02 | handled: those files are stopped, naming the column | Ingestion. The other plant's files keep loading. |
promised-date-out-of-order | 8 | 3 | A promised-date change arrives with a change sequence older than one already loaded. | SCD-02 | flagged: an older change counted | Silver puts the late change into history and keeps the newer one current. |
material-copy-short | 9 | 2 | The daily full copy of materials comes back with 40% of rows missing. | DEL-04 | handled: the copy is held as a mass delete | Ingestion. Nothing reaches bronze, and silver keeps every material. |
promised-date-corrected | 10 | 2 | 12 corrections to promised dates are recorded in the metadata store, to take effect a day later. | HIST-05 | deferred | Gold applies the correction plan on its date and rebuilds only the plan's windows. Silver is unchanged. |
new-dealer-late | 11 | 2 | Orders for a new dealer start on day 11. The dealer's record appears in the interface on day 13. | LATE-02 | deferred | Gold keeps the lines on the unknown member, then gives them the right key the day the dealer arrives. |
change-feed-drops-rows | 12 | 3 | The change feed silently skips 7 delivery rows. | REC-05 | deferred | The live source compared with silver after the silver run must name the 7 keys. |
stream-replayed | 13 | 3 | After a restart, the stream replays the last 45 minutes of readings. | DUP-04 | flagged: replayed ids counted | Silver drops the replayed readings, so each is held once. |
data-platform-catalog/slices/02-tyre/world.yaml:226The compressed calendar: the moves, and the one world setting it changes.calendars: # the same incidents on other days; chosen by name at run time, never by default compressed: purpose: >- The 3-day development loop (user, 13 Nov): every incident above lands on day 2 or 3, so a short run on the platform still meets all eleven. Day 1 stays clean, because several incidents reach back to it. The loader refuses a move that breaks the order (a source day after the incident, an effect before it). days: 3 world: # existing world settings this calendar changes (a path that is not there is an error) systems.partner_drop.file_window_days: 1 # every on-time October file on day 1, so October can complete in 3 days moves: readings-out-of-order: {day: 2} order-lines-deleted: {day: 3, from_day: 1} # lines are cancelled the day after from_day, deleted after that sellout-file-late: {day: 2, due_day: 1} sellout-file-resent: {day: 3} lab-column-renamed: {day: 2, until_day: 2} promised-date-out-of-order: {day: 3, from_day: 1} material-copy-short: {day: 2} promised-date-corrected: {day: 2, from_day: 1, activate_day: 3} new-dealer-late: {day: 2, appears_day: 3} change-feed-drops-rows: {day: 3} stream-replayed: {day: 3}
The results in the recorded runs
The verdicts below are copied from the two design notes. The silver rows are from the 14-day full-volume run of 4 November 2026 in docs/W1C-SILVER.md. The gold rows are from the 14-day run at scale 0.1 of 5 November 2026 in docs/W1D-GOLD.md. The one ingestion row is from the first of the two notes. Both runs used the world's own calendar, so the day column is the 14-day day.
| Day | Incident | Detector | Layer | Verdict | Evidence recorded |
|---|---|---|---|---|---|
| 3 | readings-out-of-order | LATE-03 | Silver | handled | 188 late readings. Silver per event hour equals the oracle in all 24 hours. |
| 4 | order-lines-deleted | DEL-01 | Silver | handled | 5 of 5 deleted keys kept in silver and marked deleted. |
| 4 | order-lines-deleted | DEL-01 | Gold | handled | 0 of 5 deleted lines in the on-time-in-full fact. |
| 5 | sellout-file-late | LATE-01 | Gold | handled | October flagged incomplete on days 1 to 4, complete and restated on day 5. Rows went from 7,392 to 9,404. |
| 6 | sellout-file-resent | DUP-01 | Gold | handled | Sell-out per month unchanged, no restatement. |
| 7 | lab-column-renamed | SCH-02 | Gold | equal to the oracle | The affected plant's 12 hours flagged incomplete. The other plant's rows complete. |
| 8 | promised-date-out-of-order | SCD-02 | Silver | handled | 3 versions. The late change sits in history, not current. |
| 9 | material-copy-short | DEL-04 | Ingestion | handled | 720 of 1,800 keys missing, which is 40%. The copy was held. |
| 11 | promised-date-corrected | HIST-05 | Gold | handled | The plan applied on its date. Restated windows 13 to 18 November, none outside the plan. |
| 11, 12 | new-dealer-late | LATE-02 | Gold | flagged | 6 then 10 of the dealer's lines on the unknown member, equal to the oracle. No row dropped. |
| 13 | new-dealer-late | LATE-02 | Gold | handled | The dealer arrived. 0 lines on the unknown member. |
| 13 | stream-replayed | DUP-04 | Silver | handled | 19,440 replayed rows dropped. Silver readings equal the oracle. |
First, neither design note records a verdict for change-feed-drops-rows. The code that judges it exists in the silver day-by-day module, and the comparison it relies on has its own unit test, but I found no recorded run that shows the seven keys being named.
Second, neither note records a run on the compressed calendar. The compressed days in the table above come from the world file. A test proves that the compressed calendar loads, moves every incident into three days and keeps day 1 clean. It does not run the platform on it.
How a verdict is reached
src/dataplatform/ingest/daybyday.py:70Ingestion's judge: one verdict per incident, from the dataset's run report.def judge(day: int, finding: dict[str, Any], d: DatasetReport | None) -> Verdict:
det, ds = finding["detector"], finding["dataset"]
c, noticed = (d.counts, d.noticed) if d else ({}, {})
def v(verdict: str, evidence: str) -> Verdict:
return Verdict(day, finding["incident"], det, ds, verdict, evidence)
if det == "DUP-01":
ok = c.get("skipped_duplicate", 0) > 0
return v("handled" if ok else "missed", f"{c.get('skipped_duplicate', 0)} batch(es) skipped as duplicates")
if det == "SCH-02":
col = str(finding["detail"].get("missing_column", ""))
named = d is not None and any(col in r for r in d.reasons)
return v(
"handled" if c.get("rejected_contract", 0) and named else "missed",
f"{c.get('rejected_contract', 0)} file(s) stopped naming {col}; others loaded ({c.get('loaded', 0)})",
)
if det == "DEL-04":
return v(
"handled" if c.get("held_mass_delete", 0) else "missed", "; ".join(d.reasons) if d else "dataset not run"
)
if det == "DEL-01" and finding["detail"].get("via") == "full_copy":
return v("deferred", "the next full copy lacks the key; silver's key comparison marks it deleted (W1c)")
if det == "SCD-02" and "field" in finding["detail"]:
return v("deferred", "a master record changed value; silver closes the old version and opens the new (W1c)")
if det in NOTICED:
return v("flagged" if noticed.get(det, 0) else "missed", f"{noticed.get(det, 0)} {NOTICED[det]}")
if det in DEFERRED:
return v("deferred", DEFERRED[det])
return v("missed", "no ingestion rule for this detector")Walk through it. For a duplicate, the verdict is handled when at least one batch was skipped as a duplicate. For a renamed column, it is handled when a file was rejected and the reason names the missing column. For a short copy, it is handled when a batch was held as a mass delete. For the four detectors ingestion only notices, the verdict is flagged when the count it noticed is above zero. For the four that belong to later layers, the verdict is deferred, with a sentence saying who owns it. Anything else is missed.
src/dataplatform/silver/daybyday.py:104Silver's judge reads what the silver table holds. The delete case is shown.def judge(
n: int, finding: dict[str, Any], tables: SilverTables, ps: dict[str, SilverPlan], day: SilverDay,
tokenize: Tokenize | None = None,
) -> Verdict | None:
det, ds = finding["detector"], finding["dataset"]
if det not in SILVER_DETECTORS or ds not in ps:
return None
plan, detail = ps[ds], finding["detail"]
def v(ok: bool, evidence: str) -> Verdict:
return Verdict(n, finding["incident"], det, ds, "handled" if ok else "missed", evidence)
if det == "DEL-01":
rows = [tables.versions(plan, _stored_key(plan, k, tokenize)) for k in detail["keys"]]
marked = sum(1 for r in rows if r and any(x["_is_current"] and x["_is_deleted"] for x in r))
return v(marked == len(rows), f"{marked} of {len(rows)} deleted keys kept in silver and marked deleted")Silver's judge does not read counts from a log. For the deleted lines it asks silver for the versions of each deleted key and checks that a current version is marked deleted. The gold judge works the same way, with queries on published gold.
No alarm on a clean day
Catching every incident is half the promise. The other half is that nothing fires when nothing happened. One test walks every day's findings and fails if an incident shows up on a day the calendar did not give it. It also checks that days 1, 2 and 14 are clean.
tests/dataplatform/test_world_tyre.py:232Each incident on its own day and no other.def test_detectors_fire_only_on_their_days(run):
"""SIM-01: each incident shows up on its calendar day (a late dimension also on the day after) and no other."""
allowed = {e["id"]: {e["day"]} for e in run["dwd"].world["calendar"]}
allowed["new-dealer-late"] = {11, 12}
for day, r in run["reports"].items():
for f in r["findings"]:
assert day in allowed[f["incident"]], f"{f['incident']} fired on day {day}"
assert [d for d, r in run["reports"].items() if r["clean_day"]] == [1, 2, 14]How the incidents are tested
| Test file | What it proves about incidents | Needs |
|---|---|---|
test_world_tyre.py (12) | Each of the eleven is really in the sources on its day, with the detector the world file names. | Python only |
test_ingest_world.py (6) | Ingestion gives each of the eleven the verdict it owes, over 14 days at scale 0.05. | The Delta package. Marked slow. |
test_silver_world.py (3) | Silver handles DEL-01, SCD-02, LATE-03 and DUP-04, and the short copy is still held. | Spark. Marked slow. |
test_twin_engine.py (2) | The scenario checks for ingestion, silver and gold pass on the engine at scale 0.02. | Spark for silver and gold. Marked slow. |
test_twin_scenarios.py (8) | Every calendar incident has a scenario file, and on a stand-in run log each raises one alert and no clean day raises any. | Python only |
tests/dataplatform/test_ingest_world.py:86The verdict ingestion owes each incident, as a table.INCIDENTS = [
("readings-out-of-order", 3, "LATE-03", "flagged"),
("order-lines-deleted", 4, "DEL-01", "flagged"),
("sellout-file-late", 5, "LATE-01", "deferred"),
("sellout-file-resent", 6, "DUP-01", "handled"),
("lab-column-renamed", 7, "SCH-02", "handled"),
("promised-date-out-of-order", 8, "SCD-02", "flagged"),
("material-copy-short", 9, "DEL-04", "handled"),
("promised-date-corrected", 10, "HIST-05", "deferred"),
("new-dealer-late", 11, "LATE-02", "deferred"),
("change-feed-drops-rows", 12, "REC-05", "deferred"),
("stream-replayed", 13, "DUP-04", "flagged"),
]- The silent loss of seven delivery rows has no recorded verdict, and neither slow test passes the live source reader that the check needs. Only the run script does.
- No recorded run uses the compressed calendar.
- Silver does not write detector rows to the run log for the incidents it handles, so operations cannot alert on them from the engine's run log yet.
- The late dealer file is listed by the world file as a finding only on day 5, when the file arrives. The lateness is visible from day 1. The operations notes record this as a gap in the specification.
- Checks for the data expectation services layer are pending in every scenario.