Data PlatformZoom level 2
The simulated tyre company
The simulated company is a program that pretends to be a tyre maker. It runs for simulated days. Each day it takes orders, cures tyres, inspects them, receives dealer files and emits sensor readings, and writes all of it into mock source systems.
It exists because real data cannot be used to prove a platform. With real data nobody knows the right answer. With generated data the generator knows exactly what it produced, so every number the platform shows can be checked.
The world file
The company is described in one file, data-platform-catalog/slices/02-tyre/world.yaml. The file says what the sources are, how much they produce, when the clock starts, and which incidents happen on which day. It does not declare columns or keys. Those come from the dataset manifests and contracts that it cites, so the simulated company and the platform read the same specification.
data-platform-catalog/slices/02-tyre/world.yaml:34The clock: fourteen days from 9 November 2026, in the business time zone, with two plants.clock: start: 2026-11-09T00:00:00+05:30 # jobs read time only through the clock port (DPC-0032) # Partner files are due in the first 5 days of a month, but the clock starts on 9 November. Days 1 to 5 of the # clock (9 to 13 November) therefore stand in for October's file window: every dealer's October file is due on # day 1, and a file that arrives later is late (see sellout-file-late). November's files fall after the clock ends. days: 14 business_timezone: Asia/Kolkata plants: P-CHN: {timezone: Asia/Kolkata, presses: 48, shifts_per_day: 3} P-HUN: {timezone: Europe/Budapest, presses: 24, shifts_per_day: 2}
data-platform-catalog/slices/02-tyre/world.yaml:80The volumes and shares the generator aims for.population:
dealers: 600
materials: 1800
orders_per_day: {min: 1500, max: 3000}
lines_per_order: {min: 1, max: 6}
on_time_share: 0.93
in_full_share: 0.96
cancel_share: 0.03 # lines set to line_status cancelled; OTIF leaves them out (fact_otif)
tyres_cured_per_press_hour: {min: 20, max: 34}
scrap_share: {mean: 0.018, sd: 0.006}The six source systems
The table lists the six systems in the world file. The second column is the kind of source, the third the datasets it holds, and the fourth how it delivers.
| System | Kind | Datasets | How it delivers |
|---|---|---|---|
erp_db | Database with a change feed | Sales order lines, deliveries, materials | Order lines and deliveries every 15 minutes through the change feed. Materials as a full copy daily at 01:00. |
mes_db | Database with a change feed | Production batches, presses | Production every 15 minutes. Presses as a full copy daily at 01:00. |
lims_drop | File drop | Inspection results | One CSV file per plant per hour. |
partner_drop | File drop | Dealer sell-out | One CSV file per dealer per month, in the first 5 days of the month. |
dealer_api | Request and response interface | Dealers | Pages of 200 records since a cursor, read every 6 hours. |
press_stream | Event stream | Press readings | One reading per press every 10 seconds. |
A change feed is a list of every insert, update and delete in the order the database committed them. A full copy is the whole table sent again. Between them the six systems cover every way the platform can read a source.
How one simulated day runs
- The domain model produces the day. It works out everything the sources did that day and returns it as one object.
- The engine writes it through ports. Each output goes to the port that matches its system's kind. A write to a system the world file does not declare is refused.
- The oracle is saved. The model computes what a correct platform should show at the end of the day, and the engine writes it as one file for that day.
- The platform runs. The caller runs ingestion, then silver, then gold, and compares each with the oracle file.
src/dataplatform/world/engine.py:90One world day: write the sources, then save the oracle. def step(self, day: int) -> dict[str, Any]:
"""One world day: the sources' output written through the ports, then the oracle's totals saved. Days run in
order from 1 after prepare(), so a caller can run the platform between two days."""
(self.out / "oracle").mkdir(parents=True, exist_ok=True)
emission = self.model.day(day)
written = self.write(emission)
oracle = self.model.oracle(day)
oracle["written"] = written
(self.out / "oracle" / f"day-{day:02d}.json").write_text(json.dumps(oracle, indent=1, sort_keys=True))
for name, note in emission.notes.items():
(self.out / "world").mkdir(parents=True, exist_ok=True)
(self.out / "world" / name).write_text(json.dumps(note, indent=1, sort_keys=True))
return oraclesrc/dataplatform/world/domains/tyre.py:180What the tyre model does in one day, system by system. def day(self, d: int) -> Emission:
e = Emission()
self.ledger[d] = {"source": {}, "bronze": {}}
self.findings[d] = []
self._masters(d, e)
self._erp(d, e)
self._plant(d, e)
self._partner(d, e)
self._dealer_api(d, e)
e.events[(self.sys[READ], READ)] = self._readings(d)
e.corrections = self.pending_corrections
self.pending_corrections = []
return eThe engine knows nothing about tyres. It only checks that the model writes to declared systems through the right port, and that the model can plant every kind of incident on the calendar. If the calendar names a kind the model has no handler for, the engine refuses to start and names it.
src/dataplatform/world/engine.py:72The refusal at start-up.class World:
def __init__(self, dwd: Dwd, model: DomainModel, sinks: Sinks, out_dir: str | Path):
self.dwd, self.model, self.sinks, self.out = dwd, model, sinks, Path(out_dir)
unhandled = sorted({e["kind"] for e in dwd.world.get("calendar", [])} - set(model.handles))
if unhandled:
raise WorldError(f"the domain model cannot plant incident kind(s) {unhandled}; add them or fix the DWD")A small worked example
One order line through three days of the simulated company. The values are an example typed for this page, in the shape the generator writes to the order line's change feed.
ExampleAn insert, a promised-date change, and a cancellation. The change sequence only ever grows.day op order_id line_no ordered_qty promised_date line_status change_seq 1 I SO-000017 1 40 2026-11-14 open 17 2 U SO-000017 1 40 2026-11-15 open 6204 3 U SO-000017 1 40 2026-11-15 cancelled 11873
The model keeps the truth beside this: the day the line was placed, the date first promised, the day it was cancelled. The oracle is built from that record, not from the feed.
Scale and repeatability
| Property | How | Where |
|---|---|---|
| Deterministic | The seed in the world file is 11. Each concern has its own random stream, so changing how readings are made never changes an order. | src/dataplatform/world/domains/tyre.py |
| Scalable | A scale below 1.0 shrinks the number of orders and the frequency of readings. Dealers and presses do not shrink. | The --scale option |
| Movable incidents | A named calendar moves every incident to other days. The compressed calendar puts all eleven on days 2 and 3 of a 3-day run. | data-platform-catalog/slices/02-tyre/world.yaml |
src/dataplatform/world/dwd.py:67Applying a named calendar. Every incident must be moved, and the order of days must hold.def apply_calendar(world: dict[str, Any], name: str) -> dict[str, Any]:
"""A named calendar from the world's `calendars:` block moves incidents to other days (for a short run that still
meets every incident). Every incident must be moved, a moved key must be `day` or one of its day params, and the
order of days must hold; anything else is an error, never a silent default."""
variants = world.get("calendars") or {}
if name not in variants:
raise DwdError(f"no calendar {name!r} in the world (have: {sorted(variants)})")
moves = variants[name].get("moves") or {}
ids = [e["id"] for e in world.get("calendar", [])]
if sorted(moves) != sorted(ids):
raise DwdError(f"calendar {name}: moves must name every incident; missing {sorted(set(ids) - set(moves))}, "
f"unknown {sorted(set(moves) - set(ids))}")
out = []
for entry in world["calendar"]:
move, params = dict(moves[entry["id"]]), dict(entry.get("params") or {})
unknown = [k for k in move if k != "day" and k not in params]
if unknown:
raise DwdError(f"calendar {name}, {entry['id']}: {unknown} are not this incident's params")
day = int(move.pop("day", entry["day"]))
params.update(move)
wrong = [k for k in BEFORE if k in params and not int(params[k]) < day]
wrong += [k for k in AFTER if k in params and not int(params[k]) > day]
wrong += [k for k in NOT_BEFORE if k in params and not int(params[k]) >= day]
if wrong or day < 1:
raise DwdError(f"calendar {name}, {entry['id']}: day {day} breaks the order of {wrong or ['day']}")
out.append({**entry, "day": day, "params": params})
changed = _set_paths(world, variants[name].get("world") or {}, name)
return {**changed, "calendar": out, "calendar_name": name}The three rules at the bottom are the order of days. A day the incident reaches back to must be before the incident. A later effect must be after it. A last day must not be before it. A calendar that breaks one of these is refused, never quietly adjusted.
The settings that drive it
| Setting in the world file | Meaning | Value for the tyre company |
|---|---|---|
seed | The seed of the random generator. | 11 |
clock.start, clock.days | The first instant and the number of days. | 9 November 2026, 14 days |
clock.business_timezone | The zone a business day is counted in. | Asia/Kolkata |
population.dealers, population.materials | Master data sizes. | 600 and 1800 |
population.orders_per_day | The range of orders placed in a day. | 1500 to 3000 |
population.on_time_share, in_full_share, cancel_share | Shares the generator aims for. | 0.93, 0.96, 0.03 |
calendar | The planted incidents with their days and exact values. | Eleven entries |
calendars | Named variants of the calendar. | One: compressed |
How it is run
| Script | What it does |
|---|---|
scripts/run_world.py | Runs the company alone for a number of days and writes the sources and the oracle files. It is the binding for the local stack: the only place that picks adapters for the sources. |
scripts/run_platform.py | Runs the company and the platform together, day by day. After each world day it runs ingestion, silver and gold, compares each with the oracle, judges the day's incidents, and at the end writes a report and the operations page. |
src/dataplatform/ingest/daybyday.py:102The day loop: world step, movers, ingestion, then bronze against the oracle.def run_days(
world: DayWorld,
days: int,
store: MetadataStore,
writer: BronzeWriter,
tokenizer: Tokenizer,
sources: Sources,
move: Callable[[int, str], None],
domain: str,
env: str = "dev",
prefix: str = "ingest",
on_day: Callable[[Day], None] | None = None,
silver: CurrentKeys | None = None,
run_ingest: Callable[[str, str], RunReport] | None = None,
) -> list[Day]:
"""`run_ingest(run_id, window)` runs the day's ingestion somewhere else (its own process or job) and returns its
report; without it the ingestion runs here."""
world.prepare()
out = []
for n in range(1, days + 1):
oracle = world.step(n)
window = str(oracle["date"])
move(n, window)
run_id = f"{prefix}-{window}"
if run_ingest is not None:
report = run_ingest(run_id, window)
else:
report = ingest(store, writer, tokenizer, sources, domain, env, run_id, Plan("normal", window),
silver=silver)
by = report.by_name()
bronze = {
ds: (int(writer.fingerprint(ds, {"_run_id": run_id})["row_count"]), int(want))
for ds, want in sorted(oracle["bronze"].items())
}
day = Day(n, window, report, bronze, [judge(n, f, by.get(f["dataset"])) for f in oracle["findings"]], oracle)
out.append(day)
if on_day:
on_day(day)
world.finish()
return outHow the simulated company itself is tested
The golden data must be right before anything is judged against it. test_world_tyre.py (12) runs fourteen days at scale 0.05 once and checks many things on that one run. Each incident is found in the sources on its day. The oracle's silver numbers equal what the mock sources actually hold. Bronze never exceeds what the sources sent. The same seed gives the same world. The clean days are days 1, 2 and 14. test_world_calendar.py (5) covers the named calendars.
tests/dataplatform/test_world_tyre.py:208One row per incident: its id, its day, its detector and the function that checks the sources.INCIDENTS = [
("readings-out-of-order", 3, "LATE-03", check_out_of_order_readings),
("order-lines-deleted", 4, "DEL-01", check_deletes),
("sellout-file-late", 5, "LATE-01", check_late_file),
("sellout-file-resent", 6, "DUP-01", check_resend),
("lab-column-renamed", 7, "SCH-02", check_rename),
("promised-date-out-of-order", 8, "SCD-02", check_late_change),
("material-copy-short", 9, None, check_short_copy),
("promised-date-corrected", 10, None, check_correction),
("new-dealer-late", 11, "LATE-02", check_late_dimension),
("change-feed-drops-rows", 12, "REC-05", check_silent_loss),
("stream-replayed", 13, "DUP-04", check_replay),
]- Volume is not realistic. The world file says so itself: thousands of rows a day, not millions.
- Prices and margins are plausible but not modelled, and the shop floor beyond the presses is not simulated.
- The database and the dealer interface have only local adapters. The file drop and the stream also have cloud adapters.
- There is one domain model, the tyre company. A second company would need its own model.
- A new kind of incident needs a handler in the model's code. The world file alone cannot add one.