Data PlatformZoom level 2
Orchestration: how a day runs, from the job to the last check
A platform needs something that decides what runs when. Here that is two things working together. Databricks Jobs start the work and retry it. Our own code decides what each run does and makes sure runs never collide or skip data.
Three ways the same day runs
| Way | What starts it | Where each hop runs | Used for |
|---|---|---|---|
| Inline | python scripts/run_platform.py | All in one process | The 3-day loop on a laptop, and every test |
| One process per hop | run_platform.py --hops process | Each hop is its own process: main scripts/run_hop.py:84 | Proving that each hop works as a deployed job task would, alone |
| Deployed job | The Databricks job dp-<env>-engine-days | One task runs the inline loop on the job's own Spark | The 3-day and 14-day proofs on Databricks |
The job definition
realms/databricks/engine/databricks.yml:52The deployed job: a few parameters and one task. engine_days:
name: dp-${var.env}-engine-days
max_concurrent_runs: 1
parameters:
- name: slice # which catalog slice to run (a folder under _catalog/slices/): 02-tyre or 03-clinical
default: "02-tyre"
- name: days
default: "3"
- name: calendar # compressed = every planted incident on days 2-3 (the 3-day loop); "" = world's own
default: "compressed"
- name: housekeeping_days # bronze housekeeping's correction window (B8); 1 on the compressed loop; "" = off
default: "1"
environments:
- environment_key: default
spec:
environment_version: "3"
dependencies:
- "psycopg[binary]>=3.2"
- "pyyaml>=6"
tasks:
- task_key: days
environment_key: default
spark_python_task:
python_file: ./tasks/run_days.py
parameters: [--slice, "{{job.parameters.slice}}", --days, "{{job.parameters.days}}", --calendar, "{{job.parameters.calendar}}",
--housekeeping-days, "{{job.parameters.housekeeping_days}}", --env, "${var.env}", --meta-instance,
"${var.meta_instance}", --run-id, "{{job.run_id}}", --root, "${workspace.file_path}"]There is one task. The job's parameters are the number of days, the name of the world calendar, and how long bronze is kept before housekeeping. The task passes them to run_days.py, which sets its environment and runs run_platform.py inside the job. When the loop ends it copies the operations page and the report to a volume and publishes the operations tables.
realms/databricks/engine/tasks/run_days.py:31How the task turns the job's options into the day loop's options.sys.argv = ["run_platform.py", "--slice", a.slice, "--days", a.days, "--scale", a.scale, "--db", "sqlite", "--meta", "lakebase",
"--spark", "existing", "--table-format", "iceberg-uc", "--env", a.env, "--upkeep-every", "7", "--reset-tables",
"--out", str(out), *(["--calendar", a.calendar] if a.calendar else []),
*(["--housekeeping-days", a.housekeeping_days, "--prove-rebuild"] if a.housekeeping_days else [])]
code = 0The loop itself
src/dataplatform/ingest/daybyday.py:102The loop: world acts, deliveries move, ingestion runs, the day is judged.def run_days(
world: DayWorld,
days: int,
store: MetadataStore,
writer: BronzeWriter,
tokenizer: Tokenizer,
sources: Sources,
move: Callable[[int, str], None],
domain: str,
env: str = "dev",
prefix: str = "ingest",
on_day: Callable[[Day], None] | None = None,
silver: CurrentKeys | None = None,
run_ingest: Callable[[str, str], RunReport] | None = None,
) -> list[Day]:
"""`run_ingest(run_id, window)` runs the day's ingestion somewhere else (its own process or job) and returns its
report; without it the ingestion runs here."""
world.prepare()
out = []
for n in range(1, days + 1):
oracle = world.step(n)
window = str(oracle["date"])
move(n, window)
run_id = f"{prefix}-{window}"
if run_ingest is not None:
report = run_ingest(run_id, window)
else:
report = ingest(store, writer, tokenizer, sources, domain, env, run_id, Plan("normal", window),
silver=silver)
by = report.by_name()
bronze = {
ds: (int(writer.fingerprint(ds, {"_run_id": run_id})["row_count"]), int(want))
for ds, want in sorted(oracle["bronze"].items())
}
day = Day(n, window, report, bronze, [judge(n, f, by.get(f["dataset"])) for f in oracle["findings"]], oracle)
out.append(day)
if on_day:
on_day(day)
world.finish()- world.step. The simulated company acts for the day and returns the true totals (the oracle).
- move. The day's change-capture output and file drops are moved to landing.
- ingest. One ingestion run takes everything new to bronze:
ingestingest/runner.py:112. - judge. Bronze row counts are compared with the truth, and each planted incident is judged.
- on_day. The caller's hook runs silver, gold, housekeeping and upkeep for the same day, then the next day starts.
What keeps the order safe
| Mechanism | What it prevents | Where |
|---|---|---|
| A lease per dataset | Two runs writing one table at the same time. The second finds it busy and writes nothing. | busy leases.py:20 |
| Bookmarks that move only on success | Skipping data after a failure, or reading it twice. A failed run is taken again whole. | Reader: picking batches, Runtime tables |
| The file log | Losing track of what was delivered, held, voided or loaded. | Runtime tables |
| The run log | Not knowing what a run did. Every step of every run is a row. | Run log |
The first slice's job
realms/databricks/databricks.yml:45The earlier four-task job for one slice: prepare, land, silver, finish, each waiting for the one before. name: dp-${var.env}-slice1-day
max_concurrent_runs: 1
parameters:
- name: day
default: "1"
environments:
- environment_key: default
spec:
environment_version: "3"
tasks:
- task_key: prepare
environment_key: default
spark_python_task:
python_file: ./tasks/main.pyNot built yet, or known gaps
- No schedule or file-arrival trigger is defined. The jobs are started by a person or a command.
- The platform job with three tasks (ingestion, silver, gold) that
--hops jobexpects is not defined in this snapshot, so that way has not been shown here. - Retries are Databricks' job setting plus our source guard for pulls. There is no retry of a failed hop by our code.
- A rule that jobs wait for their data and not for the clock (ORCH-01) is written in code comments but has no test.