cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Ingestion › Run loop

Data PlatformZoom level 2

The run loop: one ingestion run over a domain

The runner is the function that performs one ingestion run. It asks the metadata store which datasets to load, hands each one to its pipeline, and records what happened.

It holds no knowledge of any dataset. Everything it needs comes from the store's merged view of the dataset, and everything it learns goes back into the store. A run that is not recorded did not happen as far as the platform is concerned.

start_runfor each enabled datasetstartdataset runleasetake itOpen: pipelinepipelinethen flushzoom in ›log, finishdataset runfinish_runan error in one dataset is recorded and the loop goes on
The dashed frame is repeated per dataset. The grey boxes happen once per run.

The steps, with every metadata store call in order

  1. Open the run. start_run writes the run with its id, domain, environment, trigger and run type.
  2. List the datasets. datasets returns the domain's datasets. The loop skips one that is switched off, one that is a gold table, and one that is not in the optional list of names to run.
  3. Read the configuration. dataset_view returns the dataset's merged configuration as one document.
  4. Open the dataset run. start_dataset_run stores the configuration snapshot under the dataset's fingerprint.
  5. Take the lease. acquire_lease on the name bronze:source.dataset. If another run holds it, a failed step is logged, the dataset run is closed as skipped, and the loop moves on.
  6. Choose the pipeline. The pair of ingest mode and ingest read picks one of five functions. An unknown pair is logged as an error and the dataset run fails.
  7. Run the pipeline. The pipeline finds the new batches. For each it calls dataset_files, bookmarks or watermark to know where it stands, and the entry steps call batch_status to spot a batch already loaded.
  8. Flush. The new batches are written to bronze in one commit. Then log_file is called once per batch, held and skipped ones included.
  9. Record errors. log_error once per held batch, with an error class that names the kind of hold.
  10. Record the steps. log_step for the pipeline itself, then one per detector that met something, then one for the profile, which also calls last_profile and, in a normal run, save_profile.
  11. Close the dataset run. finish_dataset_run with the status, the counts and, in a normal run only, the new watermark and bookmarks.
  12. Give the lease back. release_lease, always, even after an error.
  13. Close the run. finish_run works out the run's status from its dataset runs and returns it.

The code

src/dataplatform/ingest/runner.py:112The outer loop. Three calls to the store and one helper per dataset.
def ingest(
    store: MetadataStore,
    writer: BronzeWriter,
    tokenizer: Tokenizer,
    sources: Sources,
    domain: str,
    env: str,
    run_id: str,
    plan: Plan,
    trigger: str = "schedule",
    only: set[str] | None = None,
    silver: CurrentKeys | None = None,
) -> RunReport:
    """Run every enabled dataset of a domain (or the named `source.dataset`s) once, by the plan's run type."""
    store.start_run(run_id, domain, env, trigger, plan.run_type)
    reports = []
    for ref in store.datasets(domain):
        name = f"{ref.source}.{ref.dataset}"
        if not ref.is_enabled or ref.kind in GOLD_KINDS or (only is not None and name not in only):
            continue
        reports.append(
            _one(store, writer, tokenizer, sources, run_id, plan, store.dataset_view(ref.source, ref.dataset), silver)
        )
    return RunReport(run_id, store.finish_run(run_id), reports)

The plan argument carries the run type and the business date. The silver argument is optional and gives the mass-delete guard a way to ask silver for its current keys. Entry steps explains the guard.

src/dataplatform/ingest/runner.py:138One dataset: open its run, take the lease, and make sure the lease is given back.
def _one(
    store: MetadataStore,
    writer: BronzeWriter,
    tokenizer: Tokenizer,
    sources: Sources,
    run_id: str,
    plan: Plan,
    view: dict[str, Any],
    silver: CurrentKeys | None = None,
) -> DatasetReport:
    spec = Spec.from_view(view)
    store.start_dataset_run(run_id, spec.source, spec.dataset, spec.fingerprint, view)
    lease = f"bronze:{spec.source}.{spec.dataset}"  # shared with housekeeping (ORCH-04)
    holder = store.acquire_lease(lease, run_id, LEASE_SECONDS)
    if holder is not None:
        msg = busy(store, run_id, spec.source, spec.dataset, lease, holder)
        return DatasetReport(spec.source, spec.dataset, "skipped", {}, {}, [msg])
    try:
        return _one_leased(store, writer, tokenizer, sources, run_id, plan, view, silver, spec)
    finally:
        store.release_lease(lease, run_id)

Notice the order. The dataset run is opened before the lease is asked for. That is deliberate: a step can only be logged for a dataset that was started in the run, and a busy table must leave a step behind that says who held it.

src/dataplatform/leases.py:20What happens when the table is busy.
def busy(store: MetadataStore, run_id: str, source: str, dataset: str, lease: str, holder: str) -> str:
    """Record that this run did not write because another holds the lease; the dataset run must be started."""
    msg = f"{lease} is being written by run {holder}; this run does not write it (ORCH-04)"
    store.log_step(StepRecord(run_id, source, dataset, STEP, "failed", {}, msg))
    store.finish_dataset_run(run_id, source, dataset, "skipped", message=msg)
    return msg
src/dataplatform/ingest/runner.py:161One dataset with the lease held: choose, run, flush, record, close.
def _one_leased(  # noqa: PLR0917 -- the runner's own arguments, passed through to the dataset's run
    store: MetadataStore,
    writer: BronzeWriter,
    tokenizer: Tokenizer,
    sources: Sources,
    run_id: str,
    plan: Plan,
    view: dict[str, Any],
    silver: CurrentKeys | None,
    spec: Spec,
) -> DatasetReport:
    pipeline = PIPELINES.get((spec.mode, spec.read))
    if pipeline is None:
        msg = f"no ingestion pipeline for ingest mode {spec.mode!r} and read {spec.read!r}"
        store.log_error(run_id, spec.source, spec.dataset, "ingest", "NoPipeline", msg)
        store.finish_dataset_run(run_id, spec.source, spec.dataset, "failed", message=msg)
        return DatasetReport(spec.source, spec.dataset, "failed", {}, {}, [msg])
    try:
        ctx = Ctx(spec, store, writer, tokenizer, run_id, silver=silver, pending=[])
        result = pipeline(ctx, sources, plan)
        flush(ctx)  # the run's new batches in one bronze commit, then logged
    except Exception as exc:
        msg = f"{type(exc).__name__}: {exc}"
        store.log_error(run_id, spec.source, spec.dataset, "ingest", type(exc).__name__, msg)
        store.finish_dataset_run(run_id, spec.source, spec.dataset, "failed", message=msg)
        return DatasetReport(spec.source, spec.dataset, "failed", {}, {}, [msg])
    counts, status = counts_of(result), status_of(result)
    reasons = [f"{o.batch.name}: {o.reason}" for o in result.outcomes if o.status in HELD]
    for o in result.outcomes:
        if o.status in HELD:
            store.log_error(
                run_id, spec.source, spec.dataset, "land", ERROR_CLASS[o.status], f"{o.batch.name}: {o.reason}"
            )
    noticed = "; ".join([f"{k}: {v}" for k, v in sorted(result.noticed.items())] + result.notes)
    step_status = "warned" if status == "succeeded" and result.noticed else status
    store.log_step(
        StepRecord(
            run_id,
            spec.source,
            spec.dataset,
            f"ingest.{spec.mode}.{spec.read}",
            step_status,
            {**counts, **{f"noticed.{k}": v for k, v in result.noticed.items()}},
            noticed or ("nothing new" if status == "skipped" else ""),
        )
    )
    _detect_steps(store, run_id, spec, result)
    _profile_step(store, run_id, spec, result, plan.run_type == "normal")
    store.finish_dataset_run(
        run_id,
        spec.source,
        spec.dataset,
        status,
        counts,
        result.watermark if plan.run_type == "normal" else None,
        result.bookmarks if plan.run_type == "normal" else None,
        "; ".join(reasons)[:4000],
    )
    return DatasetReport(spec.source, spec.dataset, status, counts, dict(result.noticed), reasons)

Walk through it. The lookup in PIPELINES comes first. The pipeline then runs inside a try. Any exception, such as a source that cannot be reached, is turned into one error row and a failed dataset run, and the function returns so the loop can go on to the next dataset. The context object is created with an empty pending list, which tells the entry steps to hold the batches back until flush writes them together.

After the pipeline, the counts and the status are worked out from the outcomes. The step for the pipeline is named after the mode and the read, for example ingest.push.change_feed. Its status is warned when the dataset succeeded but something was noticed, such as delete rows in a change feed.

The last call matters most. The watermark and the bookmarks are passed only when the run type is normal. A reload or a backfill goes over old ground and must not move the dataset's position.

src/dataplatform/ingest/runner.py:86The status of a dataset run, from its batches.
def status_of(result: Result) -> str:
    if any(o.status in HELD for o in result.outcomes):
        return "failed"
    return "succeeded" if result.outcomes else "skipped"
SituationDataset run status
At least one batch was heldfailed
Batches were handled and none was heldsucceeded
No batch at allskipped
The bronze table was busyskipped, with a failed orch.lease step
The pipeline raised an errorfailed
src/dataplatform/ingest/runner.py:27The three statuses that count as held, and the rule each outcome is reported under.
HELD = ("rejected_contract", "rejected_volume", "held_mass_delete")
# the detector (DAC id) that holds or skips a batch: one `detect.<id>` step row each (ADR-024 one step format)
DETECTOR = {
    "skipped_duplicate": "DUP-01",
    "rejected_contract": "SCH-02",
    "rejected_volume": "ING-01",  # the volume guard is one of the entry steps every way into bronze passes
    "held_mass_delete": "DEL-04",
}
src/dataplatform/ingest/runner.py:92One step row per rule that met something in this run.
def detect_rows(result: Result) -> list[tuple[str, str, dict[str, int], str]]:
    """(detector, status, counts, message) per detector that met this run: a held batch is `failed` (blocked), a
    skipped duplicate or anything noticed is `warned` (handled or flagged for the layer that applies it)."""
    held: dict[str, list[str]] = {}
    for o in result.outcomes:
        if o.status in DETECTOR:
            held.setdefault(o.status, []).append(f"{o.batch.name}: {o.reason}")
    out = [
        (DETECTOR[st], "warned" if st == "skipped_duplicate" else "failed", {"batches": len(why)}, "; ".join(why))
        for st, why in sorted(held.items())
    ]
    out += [(det, "warned", {"rows": n}, f"{n} noticed") for det, n in sorted(result.noticed.items())]
    return out

A held batch gives a step with status failed. A skipped duplicate gives a step with status warned. Anything a pipeline noticed, such as readings that arrived out of order, also gives a warned step with the count.

A worked example: the day the material copy came short

On day 9 of the simulated tyre company the nightly copy of the material master is missing a large share of its rows. The table shows what the run records that day. The first column is the dataset, the second its status, the third what the store holds.

DatasetStatusWhat is recorded
erp.materialfailedA file log row with status held_mass_delete, an error row of class MassDelete, a failed step detect.DEL-04. The bookmarks do not move. Bronze is unchanged.
Every other datasetIts own statusEach is loaded and recorded on its own. One dataset's hold does not stop another.

The run as a whole ends as partial, because some datasets failed and some did not. The 14-day test asserts exactly this for that day.

The settings the runner reads

NameWhere it comes fromMeaningDefault
is_enabledThe dataset rowA dataset that is switched off is not run.Required in the manifest
kindThe dataset rowGold tables are skipped.dataset
ingest_mode, ingest_readThe dataset rowTogether they pick the pipeline.None. A missing pair fails the dataset.
run_typeThe plan passed by the callernormal, reload or backfill.normal
windowThe plan passed by the callerThe business date stamped on pulled batches.Empty
onlyAn argumentA list of dataset names to limit the run to.All enabled datasets
Lease lengthLEASE_SECONDSHow long the bronze lease lasts.4 hours

How it is tested

tests/dataplatform/test_ingest_pipelines.py drives the runner against in-memory sources, an in-memory metadata store and an in-memory bronze. Each test is a short story: what the sources do, which runs happen, and what the store and bronze must show afterwards.

tests/dataplatform/test_ingest_pipelines.py:447One run over five small datasets must leave a complete record: every dataset, a fingerprint each, one step per pipeline, a profile each, six files.
def test_every_run_goes_through_the_run_log(tmp_path):
    r = Rig(tmp_path)
    everything(r)
    r.run(W1)
    rep = r.store.run_report("r1")
    assert rep["run"]["status"] == "succeeded"
    assert sorted(f"{d['source']}.{d['dataset']}" for d in rep["datasets"]) == sorted(f"{s}.{d}" for s, d in DATASETS)
    assert all(d["config_fingerprint"] for d in rep["datasets"])
    steps = [s["step"] for s in rep["steps"]]
    assert sorted(s for s in steps if s != "profile.batches") == sorted(
        f"ingest.{m}.{rd}" for m, rd, *_ in DATASETS.values())
    assert steps.count("profile.batches") == len(DATASETS)  # every dataset that loaded is profiled (B9)
    assert len(rep["files"]) == 6  # one change delivery, one copy, one file, one page, two shards
Not built yet, or known gaps
  • Datasets are processed in sequence, in the order the store lists them. A slow source delays the ones after it.
  • There is no retry of a whole dataset inside a run. A source call can be retried by the source guard. See Five pipelines.
  • The lease is asked for once and not renewed during a long load.
  • If the process dies in the middle of a dataset, its dataset run and the run stay recorded as running. The bronze write and the file log are still consistent, because a batch is logged as loaded only after the bronze commit. The stale run rows have to be dealt with by hand.
  • When a pipeline raises an error, batches that had already passed the entry steps in that dataset are not written. They are taken again by the next run.