Data PlatformZoom level 2
The run loop: one ingestion run over a domain
The runner is the function that performs one ingestion run. It asks the metadata store which datasets to load, hands each one to its pipeline, and records what happened.
It holds no knowledge of any dataset. Everything it needs comes from the store's merged view of the dataset, and everything it learns goes back into the store. A run that is not recorded did not happen as far as the platform is concerned.
The steps, with every metadata store call in order
- Open the run.
start_runwrites the run with its id, domain, environment, trigger and run type. - List the datasets.
datasetsreturns the domain's datasets. The loop skips one that is switched off, one that is a gold table, and one that is not in the optional list of names to run. - Read the configuration.
dataset_viewreturns the dataset's merged configuration as one document. - Open the dataset run.
start_dataset_runstores the configuration snapshot under the dataset's fingerprint. - Take the lease.
acquire_leaseon the namebronze:source.dataset. If another run holds it, a failed step is logged, the dataset run is closed as skipped, and the loop moves on. - Choose the pipeline. The pair of ingest mode and ingest read picks one of five functions. An unknown pair is logged as an error and the dataset run fails.
- Run the pipeline. The pipeline finds the new batches. For each it calls
dataset_files,bookmarksorwatermarkto know where it stands, and the entry steps callbatch_statusto spot a batch already loaded. - Flush. The new batches are written to bronze in one commit. Then
log_fileis called once per batch, held and skipped ones included. - Record errors.
log_erroronce per held batch, with an error class that names the kind of hold. - Record the steps.
log_stepfor the pipeline itself, then one per detector that met something, then one for the profile, which also callslast_profileand, in a normal run,save_profile. - Close the dataset run.
finish_dataset_runwith the status, the counts and, in a normal run only, the new watermark and bookmarks. - Give the lease back.
release_lease, always, even after an error. - Close the run.
finish_runworks out the run's status from its dataset runs and returns it.
The code
src/dataplatform/ingest/runner.py:112The outer loop. Three calls to the store and one helper per dataset.def ingest(
store: MetadataStore,
writer: BronzeWriter,
tokenizer: Tokenizer,
sources: Sources,
domain: str,
env: str,
run_id: str,
plan: Plan,
trigger: str = "schedule",
only: set[str] | None = None,
silver: CurrentKeys | None = None,
) -> RunReport:
"""Run every enabled dataset of a domain (or the named `source.dataset`s) once, by the plan's run type."""
store.start_run(run_id, domain, env, trigger, plan.run_type)
reports = []
for ref in store.datasets(domain):
name = f"{ref.source}.{ref.dataset}"
if not ref.is_enabled or ref.kind in GOLD_KINDS or (only is not None and name not in only):
continue
reports.append(
_one(store, writer, tokenizer, sources, run_id, plan, store.dataset_view(ref.source, ref.dataset), silver)
)
return RunReport(run_id, store.finish_run(run_id), reports)The plan argument carries the run type and the business date. The silver argument is optional and gives the mass-delete guard a way to ask silver for its current keys. Entry steps explains the guard.
src/dataplatform/ingest/runner.py:138One dataset: open its run, take the lease, and make sure the lease is given back.def _one(
store: MetadataStore,
writer: BronzeWriter,
tokenizer: Tokenizer,
sources: Sources,
run_id: str,
plan: Plan,
view: dict[str, Any],
silver: CurrentKeys | None = None,
) -> DatasetReport:
spec = Spec.from_view(view)
store.start_dataset_run(run_id, spec.source, spec.dataset, spec.fingerprint, view)
lease = f"bronze:{spec.source}.{spec.dataset}" # shared with housekeeping (ORCH-04)
holder = store.acquire_lease(lease, run_id, LEASE_SECONDS)
if holder is not None:
msg = busy(store, run_id, spec.source, spec.dataset, lease, holder)
return DatasetReport(spec.source, spec.dataset, "skipped", {}, {}, [msg])
try:
return _one_leased(store, writer, tokenizer, sources, run_id, plan, view, silver, spec)
finally:
store.release_lease(lease, run_id)Notice the order. The dataset run is opened before the lease is asked for. That is deliberate: a step can only be logged for a dataset that was started in the run, and a busy table must leave a step behind that says who held it.
src/dataplatform/leases.py:20What happens when the table is busy.def busy(store: MetadataStore, run_id: str, source: str, dataset: str, lease: str, holder: str) -> str:
"""Record that this run did not write because another holds the lease; the dataset run must be started."""
msg = f"{lease} is being written by run {holder}; this run does not write it (ORCH-04)"
store.log_step(StepRecord(run_id, source, dataset, STEP, "failed", {}, msg))
store.finish_dataset_run(run_id, source, dataset, "skipped", message=msg)
return msgsrc/dataplatform/ingest/runner.py:161One dataset with the lease held: choose, run, flush, record, close.def _one_leased( # noqa: PLR0917 -- the runner's own arguments, passed through to the dataset's run store: MetadataStore, writer: BronzeWriter, tokenizer: Tokenizer, sources: Sources, run_id: str, plan: Plan, view: dict[str, Any], silver: CurrentKeys | None, spec: Spec, ) -> DatasetReport: pipeline = PIPELINES.get((spec.mode, spec.read)) if pipeline is None: msg = f"no ingestion pipeline for ingest mode {spec.mode!r} and read {spec.read!r}" store.log_error(run_id, spec.source, spec.dataset, "ingest", "NoPipeline", msg) store.finish_dataset_run(run_id, spec.source, spec.dataset, "failed", message=msg) return DatasetReport(spec.source, spec.dataset, "failed", {}, {}, [msg]) try: ctx = Ctx(spec, store, writer, tokenizer, run_id, silver=silver, pending=[]) result = pipeline(ctx, sources, plan) flush(ctx) # the run's new batches in one bronze commit, then logged except Exception as exc: msg = f"{type(exc).__name__}: {exc}" store.log_error(run_id, spec.source, spec.dataset, "ingest", type(exc).__name__, msg) store.finish_dataset_run(run_id, spec.source, spec.dataset, "failed", message=msg) return DatasetReport(spec.source, spec.dataset, "failed", {}, {}, [msg]) counts, status = counts_of(result), status_of(result) reasons = [f"{o.batch.name}: {o.reason}" for o in result.outcomes if o.status in HELD] for o in result.outcomes: if o.status in HELD: store.log_error( run_id, spec.source, spec.dataset, "land", ERROR_CLASS[o.status], f"{o.batch.name}: {o.reason}" ) noticed = "; ".join([f"{k}: {v}" for k, v in sorted(result.noticed.items())] + result.notes) step_status = "warned" if status == "succeeded" and result.noticed else status store.log_step( StepRecord( run_id, spec.source, spec.dataset, f"ingest.{spec.mode}.{spec.read}", step_status, {**counts, **{f"noticed.{k}": v for k, v in result.noticed.items()}}, noticed or ("nothing new" if status == "skipped" else ""), ) ) _detect_steps(store, run_id, spec, result) _profile_step(store, run_id, spec, result, plan.run_type == "normal") store.finish_dataset_run( run_id, spec.source, spec.dataset, status, counts, result.watermark if plan.run_type == "normal" else None, result.bookmarks if plan.run_type == "normal" else None, "; ".join(reasons)[:4000], ) return DatasetReport(spec.source, spec.dataset, status, counts, dict(result.noticed), reasons)
Walk through it. The lookup in PIPELINES comes first. The pipeline then runs inside a try. Any exception, such as a source that cannot be reached, is turned into one error row and a failed dataset run, and the function returns so the loop can go on to the next dataset. The context object is created with an empty pending list, which tells the entry steps to hold the batches back until flush writes them together.
After the pipeline, the counts and the status are worked out from the outcomes. The step for the pipeline is named after the mode and the read, for example ingest.push.change_feed. Its status is warned when the dataset succeeded but something was noticed, such as delete rows in a change feed.
The last call matters most. The watermark and the bookmarks are passed only when the run type is normal. A reload or a backfill goes over old ground and must not move the dataset's position.
src/dataplatform/ingest/runner.py:86The status of a dataset run, from its batches.def status_of(result: Result) -> str:
if any(o.status in HELD for o in result.outcomes):
return "failed"
return "succeeded" if result.outcomes else "skipped"| Situation | Dataset run status |
|---|---|
| At least one batch was held | failed |
| Batches were handled and none was held | succeeded |
| No batch at all | skipped |
| The bronze table was busy | skipped, with a failed orch.lease step |
| The pipeline raised an error | failed |
src/dataplatform/ingest/runner.py:27The three statuses that count as held, and the rule each outcome is reported under.HELD = ("rejected_contract", "rejected_volume", "held_mass_delete")
# the detector (DAC id) that holds or skips a batch: one `detect.<id>` step row each (ADR-024 one step format)
DETECTOR = {
"skipped_duplicate": "DUP-01",
"rejected_contract": "SCH-02",
"rejected_volume": "ING-01", # the volume guard is one of the entry steps every way into bronze passes
"held_mass_delete": "DEL-04",
}src/dataplatform/ingest/runner.py:92One step row per rule that met something in this run.def detect_rows(result: Result) -> list[tuple[str, str, dict[str, int], str]]:
"""(detector, status, counts, message) per detector that met this run: a held batch is `failed` (blocked), a
skipped duplicate or anything noticed is `warned` (handled or flagged for the layer that applies it)."""
held: dict[str, list[str]] = {}
for o in result.outcomes:
if o.status in DETECTOR:
held.setdefault(o.status, []).append(f"{o.batch.name}: {o.reason}")
out = [
(DETECTOR[st], "warned" if st == "skipped_duplicate" else "failed", {"batches": len(why)}, "; ".join(why))
for st, why in sorted(held.items())
]
out += [(det, "warned", {"rows": n}, f"{n} noticed") for det, n in sorted(result.noticed.items())]
return outA held batch gives a step with status failed. A skipped duplicate gives a step with status warned. Anything a pipeline noticed, such as readings that arrived out of order, also gives a warned step with the count.
A worked example: the day the material copy came short
On day 9 of the simulated tyre company the nightly copy of the material master is missing a large share of its rows. The table shows what the run records that day. The first column is the dataset, the second its status, the third what the store holds.
| Dataset | Status | What is recorded |
|---|---|---|
erp.material | failed | A file log row with status held_mass_delete, an error row of class MassDelete, a failed step detect.DEL-04. The bookmarks do not move. Bronze is unchanged. |
| Every other dataset | Its own status | Each is loaded and recorded on its own. One dataset's hold does not stop another. |
The run as a whole ends as partial, because some datasets failed and some did not. The 14-day test asserts exactly this for that day.
The settings the runner reads
| Name | Where it comes from | Meaning | Default |
|---|---|---|---|
is_enabled | The dataset row | A dataset that is switched off is not run. | Required in the manifest |
kind | The dataset row | Gold tables are skipped. | dataset |
ingest_mode, ingest_read | The dataset row | Together they pick the pipeline. | None. A missing pair fails the dataset. |
run_type | The plan passed by the caller | normal, reload or backfill. | normal |
window | The plan passed by the caller | The business date stamped on pulled batches. | Empty |
only | An argument | A list of dataset names to limit the run to. | All enabled datasets |
| Lease length | LEASE_SECONDS | How long the bronze lease lasts. | 4 hours |
How it is tested
tests/dataplatform/test_ingest_pipelines.py drives the runner against in-memory sources, an in-memory metadata store and an in-memory bronze. Each test is a short story: what the sources do, which runs happen, and what the store and bronze must show afterwards.
tests/dataplatform/test_ingest_pipelines.py:447One run over five small datasets must leave a complete record: every dataset, a fingerprint each, one step per pipeline, a profile each, six files.def test_every_run_goes_through_the_run_log(tmp_path):
r = Rig(tmp_path)
everything(r)
r.run(W1)
rep = r.store.run_report("r1")
assert rep["run"]["status"] == "succeeded"
assert sorted(f"{d['source']}.{d['dataset']}" for d in rep["datasets"]) == sorted(f"{s}.{d}" for s, d in DATASETS)
assert all(d["config_fingerprint"] for d in rep["datasets"])
steps = [s["step"] for s in rep["steps"]]
assert sorted(s for s in steps if s != "profile.batches") == sorted(
f"ingest.{m}.{rd}" for m, rd, *_ in DATASETS.values())
assert steps.count("profile.batches") == len(DATASETS) # every dataset that loaded is profiled (B9)
assert len(rep["files"]) == 6 # one change delivery, one copy, one file, one page, two shards- Datasets are processed in sequence, in the order the store lists them. A slow source delays the ones after it.
- There is no retry of a whole dataset inside a run. A source call can be retried by the source guard. See Five pipelines.
- The lease is asked for once and not renewed during a long load.
- If the process dies in the middle of a dataset, its dataset run and the run stay recorded as running. The bronze write and the file log are still consistent, because a batch is logged as loaded only after the bronze commit. The stale run rows have to be dealt with by hand.
- When a pipeline raises an error, batches that had already passed the entry steps in that dataset are not written. They are taken again by the next run.