cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Code map › Orchestration, end to end

Data PlatformZoom level 2

Orchestration: how a day runs, from the job to the last check

A platform needs something that decides what runs when. Here that is two things working together. Databricks Jobs start the work and retry it. Our own code decides what each run does and makes sure runs never collide or skip data.

One Databricks task runs the whole loop. Inside, each day runs ingestion, then silver, then gold, then the checks.

Three ways the same day runs

WayWhat starts itWhere each hop runsUsed for
Inlinepython scripts/run_platform.pyAll in one processThe 3-day loop on a laptop, and every test
One process per hoprun_platform.py --hops processEach hop is its own process: main scripts/run_hop.py:84Proving that each hop works as a deployed job task would, alone
Deployed jobThe Databricks job dp-<env>-engine-daysOne task runs the inline loop on the job's own SparkThe 3-day and 14-day proofs on Databricks

The job definition

realms/databricks/engine/databricks.yml:52The deployed job: a few parameters and one task.
    engine_days:
      name: dp-${var.env}-engine-days
      max_concurrent_runs: 1
      parameters:
        - name: slice           # which catalog slice to run (a folder under _catalog/slices/): 02-tyre or 03-clinical
          default: "02-tyre"
        - name: days
          default: "3"
        - name: calendar        # compressed = every planted incident on days 2-3 (the 3-day loop); "" = world's own
          default: "compressed"
        - name: housekeeping_days  # bronze housekeeping's correction window (B8); 1 on the compressed loop; "" = off
          default: "1"
      environments:
        - environment_key: default
          spec:
            environment_version: "3"
            dependencies:
              - "psycopg[binary]>=3.2"
              - "pyyaml>=6"
      tasks:
        - task_key: days
          environment_key: default
          spark_python_task:
            python_file: ./tasks/run_days.py
            parameters: [--slice, "{{job.parameters.slice}}", --days, "{{job.parameters.days}}", --calendar, "{{job.parameters.calendar}}",
                         --housekeeping-days, "{{job.parameters.housekeeping_days}}", --env, "${var.env}", --meta-instance,
                         "${var.meta_instance}", --run-id, "{{job.run_id}}", --root, "${workspace.file_path}"]

There is one task. The job's parameters are the number of days, the name of the world calendar, and how long bronze is kept before housekeeping. The task passes them to run_days.py, which sets its environment and runs run_platform.py inside the job. When the loop ends it copies the operations page and the report to a volume and publishes the operations tables.

realms/databricks/engine/tasks/run_days.py:31How the task turns the job's options into the day loop's options.
sys.argv = ["run_platform.py", "--slice", a.slice, "--days", a.days, "--scale", a.scale, "--db", "sqlite", "--meta", "lakebase",
            "--spark", "existing", "--table-format", "iceberg-uc", "--env", a.env, "--upkeep-every", "7", "--reset-tables",
            "--out", str(out), *(["--calendar", a.calendar] if a.calendar else []),
            *(["--housekeeping-days", a.housekeeping_days, "--prove-rebuild"] if a.housekeeping_days else [])]
code = 0

The loop itself

src/dataplatform/ingest/daybyday.py:102The loop: world acts, deliveries move, ingestion runs, the day is judged.
def run_days(
    world: DayWorld,
    days: int,
    store: MetadataStore,
    writer: BronzeWriter,
    tokenizer: Tokenizer,
    sources: Sources,
    move: Callable[[int, str], None],
    domain: str,
    env: str = "dev",
    prefix: str = "ingest",
    on_day: Callable[[Day], None] | None = None,
    silver: CurrentKeys | None = None,
    run_ingest: Callable[[str, str], RunReport] | None = None,
) -> list[Day]:
    """`run_ingest(run_id, window)` runs the day's ingestion somewhere else (its own process or job) and returns its
    report; without it the ingestion runs here."""
    world.prepare()
    out = []
    for n in range(1, days + 1):
        oracle = world.step(n)
        window = str(oracle["date"])
        move(n, window)
        run_id = f"{prefix}-{window}"
        if run_ingest is not None:
            report = run_ingest(run_id, window)
        else:
            report = ingest(store, writer, tokenizer, sources, domain, env, run_id, Plan("normal", window),
                            silver=silver)
        by = report.by_name()
        bronze = {
            ds: (int(writer.fingerprint(ds, {"_run_id": run_id})["row_count"]), int(want))
            for ds, want in sorted(oracle["bronze"].items())
        }
        day = Day(n, window, report, bronze, [judge(n, f, by.get(f["dataset"])) for f in oracle["findings"]], oracle)
        out.append(day)
        if on_day:
            on_day(day)
    world.finish()
  1. world.step. The simulated company acts for the day and returns the true totals (the oracle).
  2. move. The day's change-capture output and file drops are moved to landing.
  3. ingest. One ingestion run takes everything new to bronze: ingest ingest/runner.py:112.
  4. judge. Bronze row counts are compared with the truth, and each planted incident is judged.
  5. on_day. The caller's hook runs silver, gold, housekeeping and upkeep for the same day, then the next day starts.

What keeps the order safe

MechanismWhat it preventsWhere
A lease per datasetTwo runs writing one table at the same time. The second finds it busy and writes nothing.busy leases.py:20
Bookmarks that move only on successSkipping data after a failure, or reading it twice. A failed run is taken again whole.Reader: picking batches, Runtime tables
The file logLosing track of what was delivered, held, voided or loaded.Runtime tables
The run logNot knowing what a run did. Every step of every run is a row.Run log

The first slice's job

realms/databricks/databricks.yml:45The earlier four-task job for one slice: prepare, land, silver, finish, each waiting for the one before.
      name: dp-${var.env}-slice1-day
      max_concurrent_runs: 1
      parameters:
        - name: day
          default: "1"
      environments:
        - environment_key: default
          spec:
            environment_version: "3"
      tasks:
        - task_key: prepare
          environment_key: default
          spark_python_task:
            python_file: ./tasks/main.py
Not built yet, or known gaps
  • No schedule or file-arrival trigger is defined. The jobs are started by a person or a command.
  • The platform job with three tasks (ingestion, silver, gold) that --hops job expects is not defined in this snapshot, so that way has not been shown here.
  • Retries are Databricks' job setting plus our source guard for pulls. There is no retry of a failed hop by our code.
  • A rule that jobs wait for their data and not for the clock (ORCH-01) is written in code comments but has no test.