cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Tests and proof › The simulated company

Data PlatformZoom level 2

The simulated tyre company

The simulated company is a program that pretends to be a tyre maker. It runs for simulated days. Each day it takes orders, cures tyres, inspects them, receives dealer files and emits sensor readings, and writes all of it into mock source systems.

It exists because real data cannot be used to prove a platform. With real data nobody knows the right answer. With generated data the generator knows exactly what it produced, so every number the platform shows can be checked.

World filesources, volumes, incidentsManifests, contractscolumns, keys, how readDomain modelthe only code thatknows tyres2 databaseswith change feeds2 file dropslab and dealer files1 interfacedealer records1 streampress readingsOpen: Oracle fileOracle fileone per dayzoom in ›Open: The platformThe platformingests themzoom in ›
The domain model in the middle is the only code that knows the business. The four dashed boxes are the sources the platform reads.

The world file

The company is described in one file, data-platform-catalog/slices/02-tyre/world.yaml. The file says what the sources are, how much they produce, when the clock starts, and which incidents happen on which day. It does not declare columns or keys. Those come from the dataset manifests and contracts that it cites, so the simulated company and the platform read the same specification.

data-platform-catalog/slices/02-tyre/world.yaml:34The clock: fourteen days from 9 November 2026, in the business time zone, with two plants.
clock:
  start: 2026-11-09T00:00:00+05:30    # jobs read time only through the clock port (DPC-0032)
  # Partner files are due in the first 5 days of a month, but the clock starts on 9 November. Days 1 to 5 of the
  # clock (9 to 13 November) therefore stand in for October's file window: every dealer's October file is due on
  # day 1, and a file that arrives later is late (see sellout-file-late). November's files fall after the clock ends.
  days: 14
  business_timezone: Asia/Kolkata
  plants:
    P-CHN: {timezone: Asia/Kolkata, presses: 48, shifts_per_day: 3}
    P-HUN: {timezone: Europe/Budapest, presses: 24, shifts_per_day: 2}
data-platform-catalog/slices/02-tyre/world.yaml:80The volumes and shares the generator aims for.
population:
  dealers: 600
  materials: 1800
  orders_per_day: {min: 1500, max: 3000}
  lines_per_order: {min: 1, max: 6}
  on_time_share: 0.93
  in_full_share: 0.96
  cancel_share: 0.03                  # lines set to line_status cancelled; OTIF leaves them out (fact_otif)
  tyres_cured_per_press_hour: {min: 20, max: 34}
  scrap_share: {mean: 0.018, sd: 0.006}

The six source systems

The table lists the six systems in the world file. The second column is the kind of source, the third the datasets it holds, and the fourth how it delivers.

SystemKindDatasetsHow it delivers
erp_dbDatabase with a change feedSales order lines, deliveries, materialsOrder lines and deliveries every 15 minutes through the change feed. Materials as a full copy daily at 01:00.
mes_dbDatabase with a change feedProduction batches, pressesProduction every 15 minutes. Presses as a full copy daily at 01:00.
lims_dropFile dropInspection resultsOne CSV file per plant per hour.
partner_dropFile dropDealer sell-outOne CSV file per dealer per month, in the first 5 days of the month.
dealer_apiRequest and response interfaceDealersPages of 200 records since a cursor, read every 6 hours.
press_streamEvent streamPress readingsOne reading per press every 10 seconds.

A change feed is a list of every insert, update and delete in the order the database committed them. A full copy is the whole table sent again. Between them the six systems cover every way the platform can read a source.

How one simulated day runs

  1. The domain model produces the day. It works out everything the sources did that day and returns it as one object.
  2. The engine writes it through ports. Each output goes to the port that matches its system's kind. A write to a system the world file does not declare is refused.
  3. The oracle is saved. The model computes what a correct platform should show at the end of the day, and the engine writes it as one file for that day.
  4. The platform runs. The caller runs ingestion, then silver, then gold, and compares each with the oracle file.
src/dataplatform/world/engine.py:90One world day: write the sources, then save the oracle.
    def step(self, day: int) -> dict[str, Any]:
        """One world day: the sources' output written through the ports, then the oracle's totals saved. Days run in
        order from 1 after prepare(), so a caller can run the platform between two days."""
        (self.out / "oracle").mkdir(parents=True, exist_ok=True)
        emission = self.model.day(day)
        written = self.write(emission)
        oracle = self.model.oracle(day)
        oracle["written"] = written
        (self.out / "oracle" / f"day-{day:02d}.json").write_text(json.dumps(oracle, indent=1, sort_keys=True))
        for name, note in emission.notes.items():
            (self.out / "world").mkdir(parents=True, exist_ok=True)
            (self.out / "world" / name).write_text(json.dumps(note, indent=1, sort_keys=True))
        return oracle
src/dataplatform/world/domains/tyre.py:180What the tyre model does in one day, system by system.
    def day(self, d: int) -> Emission:
        e = Emission()
        self.ledger[d] = {"source": {}, "bronze": {}}
        self.findings[d] = []
        self._masters(d, e)
        self._erp(d, e)
        self._plant(d, e)
        self._partner(d, e)
        self._dealer_api(d, e)
        e.events[(self.sys[READ], READ)] = self._readings(d)
        e.corrections = self.pending_corrections
        self.pending_corrections = []
        return e

The engine knows nothing about tyres. It only checks that the model writes to declared systems through the right port, and that the model can plant every kind of incident on the calendar. If the calendar names a kind the model has no handler for, the engine refuses to start and names it.

src/dataplatform/world/engine.py:72The refusal at start-up.
class World:
    def __init__(self, dwd: Dwd, model: DomainModel, sinks: Sinks, out_dir: str | Path):
        self.dwd, self.model, self.sinks, self.out = dwd, model, sinks, Path(out_dir)
        unhandled = sorted({e["kind"] for e in dwd.world.get("calendar", [])} - set(model.handles))
        if unhandled:
            raise WorldError(f"the domain model cannot plant incident kind(s) {unhandled}; add them or fix the DWD")

A small worked example

One order line through three days of the simulated company. The values are an example typed for this page, in the shape the generator writes to the order line's change feed.

ExampleAn insert, a promised-date change, and a cancellation. The change sequence only ever grows.
day  op  order_id   line_no  ordered_qty  promised_date  line_status  change_seq
 1   I   SO-000017     1         40      2026-11-14     open            17
 2   U   SO-000017     1         40      2026-11-15     open           6204
 3   U   SO-000017     1         40      2026-11-15     cancelled     11873

The model keeps the truth beside this: the day the line was placed, the date first promised, the day it was cancelled. The oracle is built from that record, not from the feed.

Scale and repeatability

PropertyHowWhere
DeterministicThe seed in the world file is 11. Each concern has its own random stream, so changing how readings are made never changes an order.src/dataplatform/world/domains/tyre.py
ScalableA scale below 1.0 shrinks the number of orders and the frequency of readings. Dealers and presses do not shrink.The --scale option
Movable incidentsA named calendar moves every incident to other days. The compressed calendar puts all eleven on days 2 and 3 of a 3-day run.data-platform-catalog/slices/02-tyre/world.yaml
src/dataplatform/world/dwd.py:67Applying a named calendar. Every incident must be moved, and the order of days must hold.
def apply_calendar(world: dict[str, Any], name: str) -> dict[str, Any]:
    """A named calendar from the world's `calendars:` block moves incidents to other days (for a short run that still
    meets every incident). Every incident must be moved, a moved key must be `day` or one of its day params, and the
    order of days must hold; anything else is an error, never a silent default."""
    variants = world.get("calendars") or {}
    if name not in variants:
        raise DwdError(f"no calendar {name!r} in the world (have: {sorted(variants)})")
    moves = variants[name].get("moves") or {}
    ids = [e["id"] for e in world.get("calendar", [])]
    if sorted(moves) != sorted(ids):
        raise DwdError(f"calendar {name}: moves must name every incident; missing {sorted(set(ids) - set(moves))}, "
                       f"unknown {sorted(set(moves) - set(ids))}")
    out = []
    for entry in world["calendar"]:
        move, params = dict(moves[entry["id"]]), dict(entry.get("params") or {})
        unknown = [k for k in move if k != "day" and k not in params]
        if unknown:
            raise DwdError(f"calendar {name}, {entry['id']}: {unknown} are not this incident's params")
        day = int(move.pop("day", entry["day"]))
        params.update(move)
        wrong = [k for k in BEFORE if k in params and not int(params[k]) < day]
        wrong += [k for k in AFTER if k in params and not int(params[k]) > day]
        wrong += [k for k in NOT_BEFORE if k in params and not int(params[k]) >= day]
        if wrong or day < 1:
            raise DwdError(f"calendar {name}, {entry['id']}: day {day} breaks the order of {wrong or ['day']}")
        out.append({**entry, "day": day, "params": params})
    changed = _set_paths(world, variants[name].get("world") or {}, name)
    return {**changed, "calendar": out, "calendar_name": name}

The three rules at the bottom are the order of days. A day the incident reaches back to must be before the incident. A later effect must be after it. A last day must not be before it. A calendar that breaks one of these is refused, never quietly adjusted.

The settings that drive it

Setting in the world fileMeaningValue for the tyre company
seedThe seed of the random generator.11
clock.start, clock.daysThe first instant and the number of days.9 November 2026, 14 days
clock.business_timezoneThe zone a business day is counted in.Asia/Kolkata
population.dealers, population.materialsMaster data sizes.600 and 1800
population.orders_per_dayThe range of orders placed in a day.1500 to 3000
population.on_time_share, in_full_share, cancel_shareShares the generator aims for.0.93, 0.96, 0.03
calendarThe planted incidents with their days and exact values.Eleven entries
calendarsNamed variants of the calendar.One: compressed

How it is run

ScriptWhat it does
scripts/run_world.pyRuns the company alone for a number of days and writes the sources and the oracle files. It is the binding for the local stack: the only place that picks adapters for the sources.
scripts/run_platform.pyRuns the company and the platform together, day by day. After each world day it runs ingestion, silver and gold, compares each with the oracle, judges the day's incidents, and at the end writes a report and the operations page.
src/dataplatform/ingest/daybyday.py:102The day loop: world step, movers, ingestion, then bronze against the oracle.
def run_days(
    world: DayWorld,
    days: int,
    store: MetadataStore,
    writer: BronzeWriter,
    tokenizer: Tokenizer,
    sources: Sources,
    move: Callable[[int, str], None],
    domain: str,
    env: str = "dev",
    prefix: str = "ingest",
    on_day: Callable[[Day], None] | None = None,
    silver: CurrentKeys | None = None,
    run_ingest: Callable[[str, str], RunReport] | None = None,
) -> list[Day]:
    """`run_ingest(run_id, window)` runs the day's ingestion somewhere else (its own process or job) and returns its
    report; without it the ingestion runs here."""
    world.prepare()
    out = []
    for n in range(1, days + 1):
        oracle = world.step(n)
        window = str(oracle["date"])
        move(n, window)
        run_id = f"{prefix}-{window}"
        if run_ingest is not None:
            report = run_ingest(run_id, window)
        else:
            report = ingest(store, writer, tokenizer, sources, domain, env, run_id, Plan("normal", window),
                            silver=silver)
        by = report.by_name()
        bronze = {
            ds: (int(writer.fingerprint(ds, {"_run_id": run_id})["row_count"]), int(want))
            for ds, want in sorted(oracle["bronze"].items())
        }
        day = Day(n, window, report, bronze, [judge(n, f, by.get(f["dataset"])) for f in oracle["findings"]], oracle)
        out.append(day)
        if on_day:
            on_day(day)
    world.finish()
    return out

How the simulated company itself is tested

The golden data must be right before anything is judged against it. test_world_tyre.py (12) runs fourteen days at scale 0.05 once and checks many things on that one run. Each incident is found in the sources on its day. The oracle's silver numbers equal what the mock sources actually hold. Bronze never exceeds what the sources sent. The same seed gives the same world. The clean days are days 1, 2 and 14. test_world_calendar.py (5) covers the named calendars.

tests/dataplatform/test_world_tyre.py:208One row per incident: its id, its day, its detector and the function that checks the sources.
INCIDENTS = [
    ("readings-out-of-order", 3, "LATE-03", check_out_of_order_readings),
    ("order-lines-deleted", 4, "DEL-01", check_deletes),
    ("sellout-file-late", 5, "LATE-01", check_late_file),
    ("sellout-file-resent", 6, "DUP-01", check_resend),
    ("lab-column-renamed", 7, "SCH-02", check_rename),
    ("promised-date-out-of-order", 8, "SCD-02", check_late_change),
    ("material-copy-short", 9, None, check_short_copy),
    ("promised-date-corrected", 10, None, check_correction),
    ("new-dealer-late", 11, "LATE-02", check_late_dimension),
    ("change-feed-drops-rows", 12, "REC-05", check_silent_loss),
    ("stream-replayed", 13, "DUP-04", check_replay),
]
Not built yet, or known gaps
  • Volume is not realistic. The world file says so itself: thousands of rows a day, not millions.
  • Prices and margins are plausible but not modelled, and the shop floor beyond the presses is not simulated.
  • The database and the dealer interface have only local adapters. The file drop and the stream also have cloud adapters.
  • There is one domain model, the tyre company. A second company would need its own model.
  • A new kind of incident needs a handler in the model's code. The world file alone cannot add one.