cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Tests and proof › The oracle

Data PlatformZoom level 2

The oracle: the true answer for every day and every layer

The oracle is the simulated company's own account of what a correct platform must show at the end of each day. It gives numbers for bronze, for silver, for gold and for the certified metrics.

Without it, a test can only check that the platform did not crash, or that it agrees with itself. With it, every layer is checked against an independent truth, every simulated day.

Open: World day NWorld day Nthe sources' outputzoom in ›Oracle, day Nthe true answerBronzeSilverGoldrowsmeasures, versions, hourswindows, completenessThe oracle is computed from the world's own record. It never reads the platform.
The top row is the platform. The oracle box below is the truth for the same day. Each arrow from it to a layer is one comparison.

What the oracle records for one day

The oracle is one file per simulated day. The table lists its parts. The first column is the key in the file, and the second is what it holds.

KeyWhat it says
sourcesRows each source produced that day.
bronzeRows a correct platform appends to bronze that day, per dataset. A file sent twice adds 0. A short full copy adds 0, because it is held. Files with a renamed column are stopped.
silverPer dataset at the end of the day: current rows, the dataset's own reconciliation measures, and for datasets with history the number of versions and of rows marked deleted. For readings, rows per event hour.
goldThe on-time-in-full fact per promised date, the scrap fact per day and plant, the sales fact per month and region, each with a completeness flag.
metricsThe certified metrics as the metric views define them.
known_answersEvery known answer in the metric files, beside what the world produced and whether they match.
findingsThe incidents planted that day and the detector each must fire.
clean_dayTrue when no incident was planted that day.
src/dataplatform/world/domains/tyre_oracle.py:22The oracle for one day, assembled from the model's own record.
def day_oracle(m: "TyreCompany", d: int) -> dict[str, Any]:
    today = m.clock.date(d)
    otif = _otif(m, d, today)
    scrap = _scrap(m, d)
    sales = _sales(m, d)
    return {
        "day": d,
        "date": today.isoformat(),
        "sources": m.ledger[d]["source"],
        "bronze": m.ledger[d]["bronze"],
        "silver": _silver(m, d),
        "gold": {"fact_otif": otif["fact"], "fact_scrap": scrap["fact"], "fact_sales": sales["fact"]},
        "metrics": {**otif["metrics"], **scrap["metrics"], **sales["metrics"]},
        "known_answers": known_answers(m, {**otif["metrics"], **scrap["metrics"], **sales["metrics"]}),
        "findings": m.findings[d],
        "clean_day": not m.findings[d],
    }

Expected and truth: two numbers where they can differ

Usually what a correct platform shows is what really happened. Two incidents break that. When the change feed skips seven delivery rows, a correct platform cannot show them, because they never reached it. When lab files arrive with a renamed column, a correct platform rightly stops them. For these the oracle gives two numbers.

NumberMeaningUsed for
expectedWhat a correct platform shows, given what reached it.Comparing the layer's tables.
truthWhat really happened at the source.The gap between the two is what reconciliation must find.
src/dataplatform/world/domains/tyre_oracle.py:52The silver part of the oracle. Deliveries and inspections carry a truth beside the expected numbers.
def _silver(m: "TyreCompany", d: int) -> dict[str, Any]:
    lines = list(m.lines.values())
    current = [ln.row for ln in lines if not ln.deleted]
    dl_all = list(m.deliveries.values())
    dl_seen = [x for x in dl_all if not x["lost"]]
    insp_sent = [x for x in m.inspections.values() if x["filed_day"] is not None]
    insp_ok = [x for x in insp_sent if not x["rejected"]]
    reads = [m.readings[k] for k in sorted(m.readings) if k <= d]
    return {
        SOL: {
            **measures(m, SOL, current),
            "qty_total": sum(r["ordered_qty"] for r in current),
            "deleted_marked": sum(ln.deleted for ln in lines),
            "versions": sum(ln.versions for ln in lines),
        },
        DLV: {
            **measures(m, DLV, dl_seen),
            "truth": measures(m, DLV, dl_all),
            "missing_keys": sorted(x["delivery_id"] for x in dl_all if x["lost"]),
        },
        MAT: {"row_count": len(m._last_copy[MAT]), "versions": m.material_versions},
        PRESS: {"row_count": len(m._last_copy[PRESS]), "versions": m.press_versions},
        PROD: measures(m, PROD, m.production),
        INSP: {
            **measures(m, INSP, insp_ok),
            "truth": measures(m, INSP, insp_sent),
            "held_back_rows": len(insp_sent) - len(insp_ok),
        },
        SELL: {"row_count": len(m.sellout), "units_total": float(sum(m.sellout.values()))},
        DEALER: {"row_count": len(m.dealer_seen), "versions": m.dealer_versions},
        READ: {
            "row_count": sum(r["unique"] for r in reads),
            "arrived_total": sum(r["arrived"] for r in reads),
            "today_by_event_hour": m.readings[d]["by_hour"],
        },
    }

Notice the first helper it calls, measures. It reads the measure list from the dataset's own manifest. The oracle and the platform therefore count exactly the same things, and a measure added to a manifest is checked without touching the oracle.

src/dataplatform/world/domains/tyre_oracle.py:42The oracle counts what the manifest says to count.
def measures(m: "TyreCompany", ds: str, rows: Iterable[dict[str, Any]]) -> dict[str, float]:
    """The dataset manifest's own reconciliation measures, so the oracle and the platform count the same things."""
    rows = list(rows)
    out: dict[str, float] = {}
    for meas in m.dwd.datasets[ds].manifest["reconciliation"]["measures"]:
        col = meas.get("column")
        out[meas["name"]] = len(rows) if not col else round(sum(float(r[col] or 0) for r in rows), 2)
    return out

A worked example

These are numbers from the recorded full-volume run of 4 November 2026, for the sales order lines. The first column is the day, and the next three are what the oracle expected and silver held. Five lines were deleted at the source on day 4.

DayCurrent order linesVersionsMarked deletedSilver equals the oracle
323,87924,7110yes
432,71034,2745yes
541,27343,4975yes

How each layer is compared

LayerCodeWhat is comparedTolerance
Bronzesrc/dataplatform/ingest/daybyday.pyRows the day's run appended per dataset against the oracle's bronze count.Exact
Silversrc/dataplatform/silver/daybyday.pyEvery number the oracle gives for the dataset that silver can count: current rows, measures, versions, deletes marked. For readings, the number of event hours whose count differs.0.01, because sums are rounded to cents
Goldsrc/dataplatform/gold/daybyday.py and src/dataplatform/world/domains/tyre_gold.pyTotals per fact, and every window: lines, quantity, on-time-in-full lines, lines on the unknown member and the completeness flag.0.01
Semanticsrc/dataplatform/semantic/answers.pyThe five known answers, asked of gold the way each metric view defines them.The tolerance each known answer states
src/dataplatform/ingest/daybyday.py:65Bronze equals the oracle when every dataset's count is equal.
    @property
    def bronze_ok(self) -> bool:
        return all(got == want for got, want in self.bronze.values())
src/dataplatform/silver/daybyday.py:78Silver against the oracle: one pair of numbers per measure.
def compare(oracle_silver: dict[str, Any], tables: SilverTables, ps: dict[str, SilverPlan]) -> dict[str, Any]:
    """Every number the oracle gives for silver that silver can count, per dataset, as (silver, oracle)."""
    out: dict[str, dict[str, tuple[float, float]]] = {}
    for name, want in sorted(oracle_silver.items()):
        plan = ps.get(name)
        if plan is None:
            continue
        got = tables.measures(plan, plan.measures)
        row = {k: (got[k], float(v)) for k, v in want.items() if k in got and isinstance(v, int | float)}
        if BY_HOUR in want and plan.event_time:
            per_hour = hourly(tables, plan, want[BY_HOUR])
            row["event_hours_differing"] = (float(sum(g != w for g, w in per_hour.values())), 0.0)
        out[name] = row
    return out

Walk through it. For each dataset in the oracle's silver part, silver's measures are read through the tables port. Every numeric key the oracle gives that silver also produced becomes a pair of silver's number and the oracle's. For a dataset with event times, silver's rows per hour are compared with the oracle's, and the number of hours that differ becomes one more pair whose expected value is zero. A day is equal when no pair differs by more than the tolerance and the run succeeded.

src/dataplatform/world/domains/tyre_gold.py:79Gold against the oracle for the on-time-in-full fact, window by window.
    def _otif(self, want: dict[str, Any], reader: GoldReader, notes: dict[str, Any]) -> dict[str, tuple[float, float]]:
        got = {r["w"]: r for r in _rows(reader, Q_OTIF)}
        exp = want.get("by_promised_date") or {}
        bad = []
        for w in sorted(set(got) | set(exp)):
            g, e = got.get(w) or {}, exp.get(w) or {}
            pairs = [(_f(g.get("lines")), _f(e.get("lines"))), (_f(g.get("qty")), _f(e.get("ordered_qty_total"))),
                     (_f(g.get("otif")), _f(e.get("otif_lines"))), (_f(g.get("orphans")), _f(e.get("orphan_lines"))),
                     (_f(g.get("complete")), float(bool(e.get("complete"))))]
            if any(a != b for a, b in pairs):
                bad.append(w)
        notes["otif_differing"] = bad
        notes["otif_windows"] = {w: [_f(r["lines"]), _f(r["qty"]), _f(r["otif"])] for w, r in got.items()}
        orphans = sum(_f(r["orphans"]) for r in got.values())
        notes["otif_orphans"] = orphans
        return {"rows": (sum(_f(r["lines"]) for r in got.values()), _f(want.get("rows"))),
                "ordered_qty_total": (sum(_f(r["qty"]) for r in got.values()), _f(want.get("ordered_qty_total"))),
                "orphan_lines": (orphans, _f(want.get("orphan_lines"))),
                "windows_differing": (float(len(bad)), 0.0)}

The gold comparison is the domain's own, because only the domain knows what a promised date is. It reads published gold with a query through a reader port, never the platform's logs.

The known answers at the semantic layer

A known answer is one question with its correct result, written beside a certified metric in its file. It says, for example, that the on-time-in-full share for the south region in November is 0.8961. The values are the oracle's numbers for seed 11 at full volume after fourteen days.

    known_answer: {where: "plant = 'P-CHN' AND day = 2026-11-12", value: 0.0180, tolerance: 0.0001}

The table is from the gold design note. The second column is the value in the metric file. The third is what gold gave in the recorded run of 5 November 2026 at one tenth of full volume, which equalled that run's oracle. Three answers depend on volume, so at one tenth they differ from the file and agree with the oracle.

Known answerValue in the fileGold at scale 0.1Depends on scale
On-time-in-full share, south, November0.89610.8851yes
Order lines, November107,18810,216yes
Sell-in units, north, November1,279,642115,637yes
Sell-out units, north, October150,803150,803no, dealer files do not scale
Scrap rate, plant P-CHN, 12 November0.01800.017981no, presses do not scale
src/dataplatform/world/domains/tyre_oracle.py:255The oracle reads each known answer from the metric file and puts its own number beside it.
def known_answers(m: "TyreCompany", metrics: dict[str, Any]) -> dict[str, Any]:
    """Every known answer the metric views carry (ADR-022), beside what the world produced for the same question.
    The questions and values are read from the metric view files, so the oracle never restates them."""
    out: dict[str, Any] = {}
    for view, spec in sorted(m.dwd.metrics.items()):
        for measure in spec.get("measures") or []:
            ka = measure.get("known_answer")
            if not ka or measure["name"] not in metrics:
                continue
            where = _where(ka["where"])
            t = next(where[k] for k in _TIME if k in where)
            g = next((where[k] for k in _GROUP if k in where), None)
            got = metrics[measure["name"]]
            value = next((got[k] for k in ([f"{t}/{g}"] if g else [t, f"{t}/all"]) if k in got), None)
            if isinstance(value, dict):
                value = value.get("expected")
            tol = float(ka.get("tolerance", 0))
            out[f"{view} {measure['name']} {ka['where']}"] = {
                "spec": ka["value"],
                "world": value,
                "tolerance": tol,
                "match": value is not None and abs(float(value) - float(ka["value"])) <= tol,
            }
    return out

The recorded runs

Three runs are written down in the design notes. Every number on this site that describes silver or gold against the oracle comes from one of them.

RunWhere recordedResult
14 days at full volume, local Spark, 4 November 2026docs/W1C-SILVER.md294 numbers compared between silver and the oracle over 14 days, all equal. Bronze equal every day. Run time 2,539 seconds, of which silver 1,711.
14 days at scale 0.1, a fresh Spark session each day, 5 November 2026docs/W1D-GOLD.mdBronze, silver and gold equal the oracle on every one of the 14 days. 1,493 seconds in all.
Full volume through gold, 4 November 2026docs/W1D-GOLD.mdGold equalled the oracle on days 1 to 7. On days 8 to 11 one completeness flag differed, and that was fixed afterwards. On day 12 the run ran out of memory and wrote no report.
14 days at scale 0.1 on DuckDB and on Spark, 5 November 2026docs/DUCKDB-SPIKE.mdBronze, silver and gold equal the oracle every day on both engines, with identical known answers.

How the oracle is tested

test_world_tyre.py (12) checks the oracle against the mock sources directly: the oracle's row counts and totals for silver must equal what the source tables and the stream actually hold. test_semantic_metrics.py (14) checks that every known answer names an address the oracle fills. A further test in that file reproduces the five known answers from a full-volume world. It runs only when an environment variable is set, because it takes about three minutes, and its note says it passed on 4 November 2026.

tests/dataplatform/test_world_tyre.py:243The oracle's silver numbers against what the sources hold.
@pytest.mark.parametrize(
    "ds,measure,source",
    [
        (SOL, "row_count", lambda run, s, t: len(run["s"].db.rows(s, t))),
        (SOL, "net_value_total", lambda run, s, t: round(sum(r["net_value"] for r in run["s"].db.rows(s, t)), 2)),
        (DLV, "row_count", lambda run, s, t: len(run["s"].db.feed(s, t))),  # what can reach silver
        (PROD, "row_count", lambda run, s, t: len(run["s"].db.rows(s, t))),
        (PROD, "tyres_cured_total", lambda run, s, t: float(sum(r["tyres_cured"] for r in run["s"].db.rows(s, t)))),
    ],
)
Not built yet, or known gaps
  • Bronze is compared by row count only. Sums and keys are first compared at silver.
  • The full-volume run through gold for all fourteen days is owed. The design note says so.
  • The oracle covers the three facts and the certified metrics. It gives no expected value for the dimension tables' surrogate keys, which are proven by table-driven tests instead.
  • Press readings feed no gold fact today, so the stream is proven up to silver only.
  • The known answers hold at full volume. At any other scale the test compares gold with the oracle of the same run, and three of the five differ from the file by design.