cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Tests and proof › Planted incidents

Data PlatformZoom level 2

Planted incidents: eleven faults the platform must catch

A planted incident is a fault put into the simulated company's data on purpose, on a known day. A dealer's file arrives four days late. A file is sent twice. The lab renames a column. Each one is something real sources do.

Clean data proves that the platform works when nothing goes wrong. Incidents prove the part that matters more: that it notices when something does, handles it at the right layer, and stays quiet on a clean day.

The world's own calendar: one incident a day from day 3 to day 131clean2clean3late readings4deletes5late file6file resent7column renamed8late change9short copy10correction11new dealer12feed skips13replay14cleanThe compressed calendar: the same eleven on days 2 and 3 of a 3-day run1clean2six incidents3five incidents
Days 1, 2 and 14 are clean. The top row is the 14-day calendar. The bottom row is the same incidents packed into 3 days.

How an incident is declared

Every incident is one entry in the world file's calendar. The entry names the day, the dataset, the exact values, the detector expected to fire, and the oracle: one sentence that says what a correct platform must hold afterwards. A detector here is the id of a rule in the specification catalog, such as DEL-01 for deletes at the source.

  - id: sellout-file-late
    day: 5
    story: late_data
    kind: late_delivery
    dataset: partner.dealer_sellout
    does: Dealer D-0042's October file, due on day 1, arrives on day 5.
    params: {dealer: D-0042, month: "2026-10", due_day: 1}
    expect:
      detector: LATE-01
      handled_by: [DPC-0080, DPC-0059, DPC-0079]
      oracle: October for D-0042 is published flagged incomplete until day 5, then rebuilt, marked complete and logged as a restatement

The generator reads the values from the params block and keeps no table of its own. When the plain sentence and the block disagree, the block wins.

The four verdicts

VerdictMeaning
handledThe layer did what the rule asks.
flaggedThe layer noticed the condition and recorded it. A later layer applies it.
deferredNot this layer's rule. The layer only loads the rows.
missedThe layer should have caught it and did not. This fails the test.

Every incident

The table has one row per incident. Day is the day in the world's own 14-day calendar. Compressed is the day in the compressed 3-day calendar. Detector is the rule id the world file names. The last two columns say what each layer does with it.

IncidentDayCompressedWhat happensDetectorIngestionWhere it is handled
readings-out-of-order32For two hours, 4% of readings from 6 presses arrive up to 90 seconds late.LATE-03flagged: late readings countedSilver places each reading in the hour of its event time.
order-lines-deleted435 cancelled order lines from day 1 are hard-deleted at the source.DEL-01flagged: delete rows countedSilver keeps the 5 keys and marks them deleted. Gold no longer counts them.
sellout-file-late52Dealer D-0042's October file, due on day 1, arrives on day 5.LATE-01deferredGold publishes October flagged incomplete, then rebuilds it and logs a restatement. Freshness shows the dealer's bookmark lagging.
sellout-file-resent63The same file is sent again, byte for byte, under a new name.DUP-01handled: the batch is skipped as a duplicateIngestion. Gold confirms nothing changed.
lab-column-renamed72The lab renames defect_code to defect in one plant's files from 10:00.SCH-02handled: those files are stopped, naming the columnIngestion. The other plant's files keep loading.
promised-date-out-of-order83A promised-date change arrives with a change sequence older than one already loaded.SCD-02flagged: an older change countedSilver puts the late change into history and keeps the newer one current.
material-copy-short92The daily full copy of materials comes back with 40% of rows missing.DEL-04handled: the copy is held as a mass deleteIngestion. Nothing reaches bronze, and silver keeps every material.
promised-date-corrected10212 corrections to promised dates are recorded in the metadata store, to take effect a day later.HIST-05deferredGold applies the correction plan on its date and rebuilds only the plan's windows. Silver is unchanged.
new-dealer-late112Orders for a new dealer start on day 11. The dealer's record appears in the interface on day 13.LATE-02deferredGold keeps the lines on the unknown member, then gives them the right key the day the dealer arrives.
change-feed-drops-rows123The change feed silently skips 7 delivery rows.REC-05deferredThe live source compared with silver after the silver run must name the 7 keys.
stream-replayed133After a restart, the stream replays the last 45 minutes of readings.DUP-04flagged: replayed ids countedSilver drops the replayed readings, so each is held once.
data-platform-catalog/slices/02-tyre/world.yaml:226The compressed calendar: the moves, and the one world setting it changes.
calendars:                            # the same incidents on other days; chosen by name at run time, never by default
  compressed:
    purpose: >-
      The 3-day development loop (user, 13 Nov): every incident above lands on day 2 or 3, so a short run on the
      platform still meets all eleven. Day 1 stays clean, because several incidents reach back to it. The loader
      refuses a move that breaks the order (a source day after the incident, an effect before it).
    days: 3
    world:                            # existing world settings this calendar changes (a path that is not there is an error)
      systems.partner_drop.file_window_days: 1   # every on-time October file on day 1, so October can complete in 3 days
    moves:
      readings-out-of-order: {day: 2}
      order-lines-deleted: {day: 3, from_day: 1}   # lines are cancelled the day after from_day, deleted after that
      sellout-file-late: {day: 2, due_day: 1}
      sellout-file-resent: {day: 3}
      lab-column-renamed: {day: 2, until_day: 2}
      promised-date-out-of-order: {day: 3, from_day: 1}
      material-copy-short: {day: 2}
      promised-date-corrected: {day: 2, from_day: 1, activate_day: 3}
      new-dealer-late: {day: 2, appears_day: 3}
      change-feed-drops-rows: {day: 3}
      stream-replayed: {day: 3}

The results in the recorded runs

The verdicts below are copied from the two design notes. The silver rows are from the 14-day full-volume run of 4 November 2026 in docs/W1C-SILVER.md. The gold rows are from the 14-day run at scale 0.1 of 5 November 2026 in docs/W1D-GOLD.md. The one ingestion row is from the first of the two notes. Both runs used the world's own calendar, so the day column is the 14-day day.

DayIncidentDetectorLayerVerdictEvidence recorded
3readings-out-of-orderLATE-03Silverhandled188 late readings. Silver per event hour equals the oracle in all 24 hours.
4order-lines-deletedDEL-01Silverhandled5 of 5 deleted keys kept in silver and marked deleted.
4order-lines-deletedDEL-01Goldhandled0 of 5 deleted lines in the on-time-in-full fact.
5sellout-file-lateLATE-01GoldhandledOctober flagged incomplete on days 1 to 4, complete and restated on day 5. Rows went from 7,392 to 9,404.
6sellout-file-resentDUP-01GoldhandledSell-out per month unchanged, no restatement.
7lab-column-renamedSCH-02Goldequal to the oracleThe affected plant's 12 hours flagged incomplete. The other plant's rows complete.
8promised-date-out-of-orderSCD-02Silverhandled3 versions. The late change sits in history, not current.
9material-copy-shortDEL-04Ingestionhandled720 of 1,800 keys missing, which is 40%. The copy was held.
11promised-date-correctedHIST-05GoldhandledThe plan applied on its date. Restated windows 13 to 18 November, none outside the plan.
11, 12new-dealer-lateLATE-02Goldflagged6 then 10 of the dealer's lines on the unknown member, equal to the oracle. No row dropped.
13new-dealer-lateLATE-02GoldhandledThe dealer arrived. 0 lines on the unknown member.
13stream-replayedDUP-04Silverhandled19,440 replayed rows dropped. Silver readings equal the oracle.
Two things the recorded runs do not show

First, neither design note records a verdict for change-feed-drops-rows. The code that judges it exists in the silver day-by-day module, and the comparison it relies on has its own unit test, but I found no recorded run that shows the seven keys being named.

Second, neither note records a run on the compressed calendar. The compressed days in the table above come from the world file. A test proves that the compressed calendar loads, moves every incident into three days and keeps day 1 clean. It does not run the platform on it.

How a verdict is reached

src/dataplatform/ingest/daybyday.py:70Ingestion's judge: one verdict per incident, from the dataset's run report.
def judge(day: int, finding: dict[str, Any], d: DatasetReport | None) -> Verdict:
    det, ds = finding["detector"], finding["dataset"]
    c, noticed = (d.counts, d.noticed) if d else ({}, {})

    def v(verdict: str, evidence: str) -> Verdict:
        return Verdict(day, finding["incident"], det, ds, verdict, evidence)

    if det == "DUP-01":
        ok = c.get("skipped_duplicate", 0) > 0
        return v("handled" if ok else "missed", f"{c.get('skipped_duplicate', 0)} batch(es) skipped as duplicates")
    if det == "SCH-02":
        col = str(finding["detail"].get("missing_column", ""))
        named = d is not None and any(col in r for r in d.reasons)
        return v(
            "handled" if c.get("rejected_contract", 0) and named else "missed",
            f"{c.get('rejected_contract', 0)} file(s) stopped naming {col}; others loaded ({c.get('loaded', 0)})",
        )
    if det == "DEL-04":
        return v(
            "handled" if c.get("held_mass_delete", 0) else "missed", "; ".join(d.reasons) if d else "dataset not run"
        )
    if det == "DEL-01" and finding["detail"].get("via") == "full_copy":
        return v("deferred", "the next full copy lacks the key; silver's key comparison marks it deleted (W1c)")
    if det == "SCD-02" and "field" in finding["detail"]:
        return v("deferred", "a master record changed value; silver closes the old version and opens the new (W1c)")
    if det in NOTICED:
        return v("flagged" if noticed.get(det, 0) else "missed", f"{noticed.get(det, 0)} {NOTICED[det]}")
    if det in DEFERRED:
        return v("deferred", DEFERRED[det])
    return v("missed", "no ingestion rule for this detector")

Walk through it. For a duplicate, the verdict is handled when at least one batch was skipped as a duplicate. For a renamed column, it is handled when a file was rejected and the reason names the missing column. For a short copy, it is handled when a batch was held as a mass delete. For the four detectors ingestion only notices, the verdict is flagged when the count it noticed is above zero. For the four that belong to later layers, the verdict is deferred, with a sentence saying who owns it. Anything else is missed.

src/dataplatform/silver/daybyday.py:104Silver's judge reads what the silver table holds. The delete case is shown.
def judge(
    n: int, finding: dict[str, Any], tables: SilverTables, ps: dict[str, SilverPlan], day: SilverDay,
    tokenize: Tokenize | None = None,
) -> Verdict | None:
    det, ds = finding["detector"], finding["dataset"]
    if det not in SILVER_DETECTORS or ds not in ps:
        return None
    plan, detail = ps[ds], finding["detail"]

    def v(ok: bool, evidence: str) -> Verdict:
        return Verdict(n, finding["incident"], det, ds, "handled" if ok else "missed", evidence)

    if det == "DEL-01":
        rows = [tables.versions(plan, _stored_key(plan, k, tokenize)) for k in detail["keys"]]
        marked = sum(1 for r in rows if r and any(x["_is_current"] and x["_is_deleted"] for x in r))
        return v(marked == len(rows), f"{marked} of {len(rows)} deleted keys kept in silver and marked deleted")

Silver's judge does not read counts from a log. For the deleted lines it asks silver for the versions of each deleted key and checks that a current version is marked deleted. The gold judge works the same way, with queries on published gold.

No alarm on a clean day

Catching every incident is half the promise. The other half is that nothing fires when nothing happened. One test walks every day's findings and fails if an incident shows up on a day the calendar did not give it. It also checks that days 1, 2 and 14 are clean.

tests/dataplatform/test_world_tyre.py:232Each incident on its own day and no other.
def test_detectors_fire_only_on_their_days(run):
    """SIM-01: each incident shows up on its calendar day (a late dimension also on the day after) and no other."""
    allowed = {e["id"]: {e["day"]} for e in run["dwd"].world["calendar"]}
    allowed["new-dealer-late"] = {11, 12}
    for day, r in run["reports"].items():
        for f in r["findings"]:
            assert day in allowed[f["incident"]], f"{f['incident']} fired on day {day}"
    assert [d for d, r in run["reports"].items() if r["clean_day"]] == [1, 2, 14]

How the incidents are tested

Test fileWhat it proves about incidentsNeeds
test_world_tyre.py (12)Each of the eleven is really in the sources on its day, with the detector the world file names.Python only
test_ingest_world.py (6)Ingestion gives each of the eleven the verdict it owes, over 14 days at scale 0.05.The Delta package. Marked slow.
test_silver_world.py (3)Silver handles DEL-01, SCD-02, LATE-03 and DUP-04, and the short copy is still held.Spark. Marked slow.
test_twin_engine.py (2)The scenario checks for ingestion, silver and gold pass on the engine at scale 0.02.Spark for silver and gold. Marked slow.
test_twin_scenarios.py (8)Every calendar incident has a scenario file, and on a stand-in run log each raises one alert and no clean day raises any.Python only
tests/dataplatform/test_ingest_world.py:86The verdict ingestion owes each incident, as a table.
INCIDENTS = [
    ("readings-out-of-order", 3, "LATE-03", "flagged"),
    ("order-lines-deleted", 4, "DEL-01", "flagged"),
    ("sellout-file-late", 5, "LATE-01", "deferred"),
    ("sellout-file-resent", 6, "DUP-01", "handled"),
    ("lab-column-renamed", 7, "SCH-02", "handled"),
    ("promised-date-out-of-order", 8, "SCD-02", "flagged"),
    ("material-copy-short", 9, "DEL-04", "handled"),
    ("promised-date-corrected", 10, "HIST-05", "deferred"),
    ("new-dealer-late", 11, "LATE-02", "deferred"),
    ("change-feed-drops-rows", 12, "REC-05", "deferred"),
    ("stream-replayed", 13, "DUP-04", "flagged"),
]
Not built yet, or known gaps
  • The silent loss of seven delivery rows has no recorded verdict, and neither slow test passes the live source reader that the check needs. Only the run script does.
  • No recorded run uses the compressed calendar.
  • Silver does not write detector rows to the run log for the incidents it handles, so operations cannot alert on them from the engine's run log yet.
  • The late dealer file is listed by the world file as a finding only on day 5, when the file arrives. The lateness is visible from day 1. The operations notes record this as a gap in the specification.
  • Checks for the data expectation services layer are pending in every scenario.