cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › One day, end to end

Data PlatformZoom level 1

One day, end to end: what happens to a day of data

The fastest way to understand the platform is to follow one day through it. This page walks the day in order. Each step links to the page that explains it in full.

The numbers match the steps below. Click a box to zoom in.

The day, in order

  1. Before the first day. The dataset manifests are checked and loaded into the metadata store. From this moment the store is the only place the code reads settings from. See Loader.
  2. 1 The sources change. Orders are placed, tyres are made, a dealer changes region. In the runs shown on these pages a simulated company does this, so the true totals for the day are known. See The simulated company.
  3. 2 Data reaches landing. Sources that push send change-feed files and plain files to the landing zone. Sources that are pulled are read by the platform. See Sources and movers.
  4. 3 One ingestion run takes everything new to bronze. The runner loops over the enabled datasets. For each it picks one of five pipelines, passes the batch through the entry steps, and the lander writes it to bronze. Bronze row counts are compared with the truth.
  5. 4 One silver run. For each dataset the reader picks the bronze batches loaded since silver last ran. They go through the thirteen stages, the history stage decides versions and deletes, and the writer applies it in one merge. Silver is compared with the truth.
  6. 5 One gold run for the business date. The harness builds the dimensions and the facts for the day's window. Gold is compared with the truth.
  7. 6 The semantic layer. After the last day the metrics are asked questions whose answers are known. See Semantic layer.
  8. Throughout. Every run, dataset and step is recorded. Freshness, alerts and the operations page are read from that record. See Operations.

The same thing in the code

One script runs the simulated company and the platform together, day by day, on a laptop. Its opening comment is the most compact description of a day that exists.

scripts/run_platform.py:1The script's own description of a day.
"""Run the tyre world and the platform together, day by day, locally: world -> sources -> landing -> bronze -> silver.
(--slice 03-clinical runs the clinical trial instead: the world in the slice picks the domain, models and gold judge.)

    python scripts/run_platform.py --days 14                 # local PostgreSQL for the sources and the metadata store
    python scripts/run_platform.py --days 3 --scale 0.05     # small and quick
    python scripts/run_platform.py --days 7 --db sqlite --meta memory   # nothing installed but Python packages
    python scripts/run_platform.py --days 3 --no-silver      # ingestion only (no Spark)

Each day: the world writes the day into the mock sources (scripts/run_world.py's local binding), the push movers
land the change feeds and drop-zone files, one ingestion run takes everything new to bronze (Delta tables under
build/run/bronze/), and bronze rows per dataset are compared with the oracle's bronze totals for the day. Each
planted incident is judged by the detector the world names. The metadata store is the run's only config source:
the manifest is loaded into it first (PostgreSQL database DP_RUN_META_DSN, default dbname=dp_meta_run, recreated
each run; or in memory). Silver (W1c): after each day's ingestion, one silver run on a local Spark (local[2], 1 GB
driver) takes the new bronze batches through the thirteen stages into Delta tables under build/run/silver/, and
silver is compared with the oracle's silver totals for the day; DEL-01, SCD-02, LATE-03 and DUP-04 are judged on what
silver holds. The mass-delete guard compares each full copy with silver's current keys. Gold (W1d): after silver,
one gold run for the day's business date builds the dimensions and the facts (models/<domain>/gold/, the domain the world names) into Delta tables
under build/run/gold/, and gold is compared with the oracle's gold for the day; LATE-01, LATE-02, HIST-05, DEL-01 and
DUP-01 are judged on gold. After the last day the metric views' known answers are asked of gold
(build/run/gold_metrics.json, the DP_GOLD_METRICS file W5's tests read). Report: build/run/report.json.

This file is the local binding: the only place that picks adapters for a run.
scripts/run_platform.py:574What runs after each day's ingestion: silver, then gold, each judged against the truth.
    def on_day(day: Day) -> None:
        show(day)
        clock.set(datetime.fromisoformat(day.date).replace(tzinfo=UTC) + timedelta(hours=2))
        if tables is not None:
            t = time.time()
            from dataplatform.silver.daybyday import judge_silver, silver_day

            if a.hops != "inline":
                s = judge_silver(day.day, day.oracle, run_hop("silver", f"silver-{day.date}", day.date), store,
                                 tables, dwd.platform["domain"], KeyedTokenizer.PATTERN, live_rows, tokenizer.token)
            else:
                s = silver_day(day.day, day.oracle, store, tables, stages, dwd.platform["domain"], "dev",
                               KeyedTokenizer.PATTERN, live=live_rows, tokenize=tokenizer.token)
            silver_days.append((s, round(time.time() - t, 1)))
            show_silver(s)
        if gold is not None:
            from dataplatform.gold.daybyday import gold_day, judge_gold

            clock.set(datetime.fromisoformat(day.date).replace(tzinfo=UTC) + timedelta(hours=3))
            t = time.time()
            if a.hops != "inline":
                g = judge_gold(day.day, day.oracle, run_hop("gold", f"gold-{day.date}", day.date), gold[2], gold[3],
                               [x for x, _ in gold_days])
            else:
                g = gold_day(day.day, day.oracle, store, gold[0], gold[1], gold[2], gold[3], dwd.platform["domain"],
                             "dev", tz, [x for x, _ in gold_days])
            gold_days.append((g, round(time.time() - t, 1)))
            show_gold(g)
        if a.housekeeping_days is not None:
            housekeeping(day)
        if table_store is not None and (day.day % a.upkeep_every == 0 or day.day == a.days):  # the port's upkeep
            # (ADR-027): compaction and version expiry
            t = time.time()
            done = upkeep_all(table_store)
            print(f"        upkeep    {done['tables']} tables, {done['rewritten_files']} files compacted, "
                  f"{done['expired_files']} old files expired ({time.time() - t:.0f}s)")
        if spark is not None and a.fresh_spark and a.spark == "local" and day.day % 5 == 0:  # a fresh session every 5th day only: the
            fresh_session()  # restart guards long runs on a small driver; a 3-day dev loop never pays for it

What is checked on a day

LayerCompared with the truthIncidents judged here
BronzeRows per dataset for the dayA file sent twice, a column renamed, a full copy that is far too short
SilverCurrent rows, versions, deletes marked, and each dataset's own measuresDeletes, a change arriving out of order, late readings, a replayed stream
GoldEach fact's measuresLate facts, history, deletes and duplicates seen from gold
SemanticKnown answersNone

An incident is a fault planted in the data on purpose on a known day. Planted incidents lists them all, with the rule that detects each one and the layer that handles it.

Where it has run

WhereWhatResult
A laptop, local Spark14 simulated days at full scale, 4 November 2026Silver equal to the truth on all 14 days, 294 numbers compared. Recorded in docs/W1C-SILVER.md.
A laptop, no SparkThe unit and contract tests of the public snapshot, 19 November 20261,316 passed and 219 skipped. The skipped tests need Spark.
DatabricksThree simulated days as a jobDescribed on Proof by layer.