cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Operations › Run log

Data PlatformZoom level 2

The run log: what every run writes down

The run log is the platform's diary. Every run of every layer writes what it did: which datasets it touched, which steps ran, how many rows, which files were loaded or held, and which errors it met.

Without it, a question such as why yesterday's order lines are missing can only be answered by reading job output. With it, the answer is one query, and the same records feed freshness, alerts and every screen.

runone per run of a domaindataset runone per dataset in the runstepone per stage or checkfileone per batch handlederrorone per error caughtEvery record carries the run id.Join any two tables on it.
A run has dataset runs. A dataset run has steps, files and errors. All five share the run id.

What is recorded, level by level

The table lists the five record kinds. The first column is the record. The second is what one row of it describes. The third lists its fields as the read side returns them.

RecordOne row isFields
RunOne run of one domain in one environmentrun id, domain, environment, trigger, run type, status, start time, end time
Dataset runOne dataset handled inside a runrun id, source, dataset, status, configuration fingerprint, watermark before and after, counts, message, start time, end time
StepOne stage of the pipeline, or one check, for one datasetrun id, source, dataset, step name, status, counts, message, time
FileOne batch that ingestion handledrun id, source, dataset, batch key, file name, window, rows, status, reason, time
ErrorOne error that was caughtrun id, source and dataset when known, step, error class, message, time

Three terms in that table need a definition. A configuration fingerprint is a hash of the dataset's settings at the time of the run, so a later reader can tell which settings a run used. A watermark is the highest change position or time already taken from the source. A batch key is the identity of one delivery, used to recognise the same delivery sent twice.

src/dataplatform/obs/runlog.py:32The dataset run as operations reads it back. It is frozen: a reader cannot change it.
@dataclass(frozen=True)
class DatasetRunRow:
    run_id: str
    source: str
    dataset: str
    status: str
    config_fingerprint: str
    watermark_before: str | None
    watermark_after: str | None
    counts: dict[str, int]
    message: str
    started_at: datetime
    ended_at: datetime | None

Reading the log back

Writing happens through the metadata store's own functions, which the runtime tables page explains. Reading for operations happens through a separate, read-only port with a single method. It returns every run of a domain that started in a time window, with everything recorded for those runs.

src/dataplatform/obs/runlog.py:93The whole read side: one method.
class RunLogReader(Protocol):
    def snapshot(self, domain: str, since: datetime, until: datetime) -> RunLogSnapshot:
        """Runs of the domain that started in [since, until), with everything recorded for them, in record order."""
        ...
AdapterFileHow it answers
In memorysrc/dataplatform/adapters/memory_runlog.pyWraps any metadata store, passes every call through, and stamps each run log write with the time from the clock port. A simulated week therefore runs on simulated time.
PostgreSQLsrc/dataplatform/adapters/postgres_runlog.pyOne read-only database function over the store's run tables, using the store's own timestamps.
No logsrc/dataplatform/adapters/null_runlog.pyReturns nothing. It proves that every screen still renders when a customer has no run log connected.
A customer's own logsrc/dataplatform/adapters/customer_runlog.pyAn example skeleton only. Its row in the contract test is skipped.

A worked example

One ingestion run loads the tyre company's sales order lines. One delivery arrives twice. A delete is seen in the change feed. The run log then holds these rows. The values are an example typed for this page, in the shape the code writes.

ExampleOne run, one dataset run, four steps and two files. Every row carries the run id.
run          run_id=ingest-2026-11-12  domain=tyre  env=dev  trigger=schedule  status=succeeded

dataset run  erp.sales_order_line  status=succeeded  counts={loaded: 1, skipped_duplicate: 1, rows_loaded: 412}

step         profile.batches           succeeded  {rows: 412, promised_date.nulls: 0}
step         detect.DUP-01             warned     {batches: 1}   "sol_2026-11-12_resent.jsonl: already loaded"
step         detect.DEL-01             warned     {rows: 5}      "5 noticed"
step         ingest.push.change_feed   warned     {loaded: 1, skipped_duplicate: 1, rows_loaded: 412, noticed.DEL-01: 5}

file         sol_2026-11-12.jsonl          loaded             rows=412
file         sol_2026-11-12_resent.jsonl   skipped_duplicate  rows=0    reason="already loaded"

The one step format

A step is either a pipeline step, such as one ingestion read or one silver stage, or a check result. Check results follow one format so that operations can treat them all alike. The name starts with one of four words and a dot.

Name starts withWritten byMeaningScreen with the detail
detect.The engine: ingestion and the gold harness todayA detector met something in a load, for example detect.DEL-01 for deletes at the sourceSources and ingestion
quality.Data expectation servicesThe verdict of one quality ruleData expectations
recon.Data expectation services, and the live source check after silverOne reconciliation entryData expectations
profile.Ingestion's profiling stepOne profile and whether it driftedData expectations

The status of a check row is the severity operations reads. succeeded reads as passed. warned means the condition was handled, or flagged for a later layer. failed reads as blocked: the load was stopped.

src/dataplatform/obs/steps.py:23The four kinds, who owns each, which screen holds the detail, and how a status is read.
DETECT = "detect"
CHECK_KINDS = ("quality", "recon", "profile")
KINDS = (DETECT, *CHECK_KINDS)
OWNER = {DETECT: "engine", "quality": "des", "recon": "des", "profile": "des"}
SCREEN = {"engine": "sources_ingestion", "des": "des_data_expectations"}  # where the detail lives
SEVERITY = {"succeeded": "passed", "warned": "warned", "failed": "blocked"}
MAX_MESSAGE = 4000
_NAME = re.compile(r"^(detect|quality|recon|profile)\.[A-Za-z0-9][A-Za-z0-9_.:-]{0,127}$")
src/dataplatform/obs/steps.py:48The contract every check row must keep. It returns what is wrong, or an empty list.
def violations(step: str, status: str, counts: Mapping[str, object], message: str) -> list[str]:
    """What breaks the one step format in this row (empty when it keeps it)."""
    out = []
    if not _NAME.match(step):
        out.append(f"step {step!r} is not <detect|quality|recon|profile>.<id>")
    if status not in SEVERITY:
        out.append(f"status {status!r} is not one of {sorted(SEVERITY)}")
    bad = [k for k, v in counts.items() if not isinstance(k, str) or isinstance(v, bool) or not isinstance(v, int)]
    if bad:
        out.append(f"counts {bad} are not name -> integer")
    if not isinstance(message, str) or len(message) > MAX_MESSAGE:
        out.append(f"message is not text of at most {MAX_MESSAGE} characters")
    if status == "failed" and not message:
        out.append("a blocked step says why")
    return out

Walk through it. The name must be one of the four kinds, a dot, and an identifier. The status must be one of the three known words. Every count must be a whole number under a text name, and a true or false value is refused even though Python treats it as a number. The message is text of at most 4000 characters. The last rule is the one people forget: a blocked step must say why.

The recorder: one id from start to finish

The recorder is a helper around the store for code that wants the run, the dataset and the step as nested blocks. Opening a run opens a root trace span. A trace is the record a tracing tool keeps of one piece of work, and a span is one timed part of it. The trace id of that root span becomes the run id.

src/dataplatform/obs/recorder.py:187Opening and closing a run. The run id is the root span's trace id.
    @contextmanager
    def run(self, domain: str, env: str, trigger: str = "manual", run_type: str = "normal") -> Iterator[RunHandle]:
        span = self.tracer.start(f"run {domain}", {"dp.domain": domain, "dp.env": env, "dp.trigger": trigger,
                                                   "dp.run_type": run_type})
        self.store.start_run(span.trace_id, domain, env, trigger, run_type)
        handle = RunHandle(self, span, domain, env)
        try:
            yield handle
        except Exception as exc:
            if id(exc) not in handle.logged:
                self.store.log_error(handle.run_id, None, None, "run", type(exc).__name__, str(exc))
            for ds in list(handle.open):
                if ds.status is None:
                    ds.fail(f"run stopped: {type(exc).__name__}")
            handle.status = self.store.finish_run(handle.run_id)
            span.set({"dp.status": handle.status})
            span.end("error", type(exc).__name__)
            self._finished(handle.run_id)
            raise
        handle.status = self.store.finish_run(handle.run_id)
        span.set({"dp.status": handle.status})
        span.end("error" if handle.status in ("failed", "partial") else "ok")
        self._finished(handle.run_id)

Walk through it. A root span starts, and its trace id is passed to the store as the run id. The block's body then runs. If the body raises, the error is logged against the run unless a dataset already logged it, every dataset still open is closed as failed, the run is finished, the hooks are called, and the error is raised again. If the body ends normally, the store works out the run's status from its dataset runs. A run with one failed dataset out of two comes back as partial. Either way the on_finish hooks receive the run id last.

src/dataplatform/obs/recorder.py:76A step as a block. An exception inside it becomes a failed step row, then travels on.
    @contextmanager
    def step(self, name: str) -> Iterator[StepHandle]:
        handle, span = StepHandle(), self.span.child(f"step {name}", {"dp.step": name})
        try:
            yield handle
        except Exception as exc:
            self._log(name, "failed", handle.counts, f"{type(exc).__name__}: {exc}")
            span.end("error", type(exc).__name__)
            raise
        self._log(name, handle.status, handle.counts, handle.message)
        span.set({f"dp.count.{k}": v for k, v in handle.counts.items()})
        span.end()
src/dataplatform/obs/recorder.py:89How a check and a detector become step rows in the one format.
    def check(self, kind: str, name: str, passed: bool, counts: dict[str, int] | None = None,
              message: str = "", warn_only: bool = False) -> None:
        """A quality rule, a reconciliation entry or a profile: its verdict as a step of this dataset run."""
        if kind not in CHECK_KINDS:
            raise ValueError(f"check kind {kind!r} is not one of {CHECK_KINDS}")
        status = "succeeded" if passed else ("warned" if warn_only else "failed")
        self._log(f"{kind}.{name}", status, counts, message or ("" if passed else f"{kind}.{name} did not pass"))

    def detect(self, detector: str, counts: dict[str, int] | None = None, message: str = "",
               blocked: bool = False) -> None:
        """A detector fired (DAC id, e.g. DEL-01). Handled is warned; blocked (the load stopped) is failed."""
        self._log(f"{DETECT}.{detector}", "failed" if blocked else "warned", counts,
                  message or f"{detector} {'blocked the load' if blocked else 'handled'}")
        self.span.set({f"dp.detect.{detector}": "blocked" if blocked else "handled"})

A check that did not pass is failed unless the caller said it should only warn. A detector that fired is warned when the platform handled the condition and failed when it blocked the load.

Tracing wrapped around the store

The ingestion, silver and gold runners were written before the recorder and call the store directly. To trace them without changing them, the store itself is wrapped. The wrapper forwards every call and adds a span for each run, dataset run and step.

src/dataplatform/obs/traced_store.py:29Five calls are intercepted. Every other call goes straight through.
    def __getattr__(self, name: str) -> Any:  # every other operation goes straight through
        return getattr(self._store, name)

    def start_run(self, run_id: str, domain: str, env: str, trigger: str = "manual", run_type: str = "normal") -> None:
        self._store.start_run(run_id, domain, env, trigger, run_type)
        self._runs[run_id] = self._tracer.start(f"run.{trigger}", {RUN_TAG: run_id, "domain": domain, "env": env,
                                                                   "run_type": run_type})

    def start_dataset_run(self, run_id: str, source: str, dataset: str, config_fingerprint: str,
                          config: dict[str, Any]) -> None:
        self._store.start_dataset_run(run_id, source, dataset, config_fingerprint, config)
        parent = self._runs.get(run_id)
        if parent is not None:
            self._datasets[(run_id, source, dataset)] = parent.child(
                f"dataset.{source}.{dataset}", {RUN_TAG: run_id, "fingerprint": config_fingerprint})

    def log_step(self, step: StepRecord) -> None:
        self._store.log_step(step)
        parent = self._datasets.get((step.run_id, step.source, step.dataset))
        if parent is not None:
            span = parent.child(f"step.{step.step}", {RUN_TAG: step.run_id, **{f"count.{k}": v for k, v in
                                                                              step.counts.items()}})
            span.end("error" if step.status == "failed" else "ok", step.message[:200])
src/dataplatform/obs/traced_store.py:59Finishing a run ends its span and pushes the run id to the hooks.
    def finish_run(self, run_id: str) -> str:
        status = self._store.finish_run(run_id)
        span = self._runs.pop(run_id, None)
        if span is not None:
            span.set({"status": status})
            span.end("error" if status == "failed" else "ok", status)
        for hook in self._on_finish:
            hook(run_id)
        return status

Two details matter. First, here the run id is the one the runner chose, and it is put on every span as the tag dp_run_id. The trace id is the tracer's own. Second, the hooks are called after the store has finished the run, so the monitor reads a complete record.

src/dataplatform/obs/trace.py:63The same 128 bits written as a UUID, the form lineage events want for a run id.
def as_uuid(trace_id: str) -> str:
    """The run id in the UUID form lineage events want for their run id (same 128 bits)."""
    if not valid_trace_id(trace_id):
        raise ValueError(f"not a trace id: {trace_id!r}")
    return str(uuid.UUID(hex=trace_id))

The settings that drive it

The run log reads no settings from the metadata store. What it records is fixed by the code. The only choices are made by the binding, which is the script that picks adapters: which store, which clock, and which tracer.

How it is tested

tests/dataplatform/test_obs_recorder.py is one table of cases. Each case plays a small run against the in-memory store and checks one fact about what was recorded.

tests/dataplatform/test_obs_recorder.py:109The start of the recorder's case table: a name, an action, and the expected result.
CASES = [
    ("run id is the root span's trace id",
     lambda e: (r := happy(e)).run_id == r.span.trace_id == e["tracer"].finished[-1].trace_id, True),
    ("every span of the run shares its trace id",
     lambda e: len({s.trace_id for s in (happy(e), e["tracer"].finished)[1]}), 1),
    ("span tree: steps, dataset, run", lambda e: (happy(e), [s.name for s in e["tracer"].finished])[1],
     ["step land", "dataset erp.orders", "run shop"]),
    ("run status from its datasets", lambda e: happy(e).status, "succeeded"),
    ("steps follow the naming convention", lambda e: [s["step"] for s in report(e, happy(e))["steps"]],
     ["land", "quality.not_null", "recon.src_vs_silver", "detect.DEL-01"]),
    ("check and detector verdicts", lambda e: [s["status"] for s in report(e, happy(e))["steps"]],
     ["succeeded", "succeeded", "failed", "warned"]),
    ("watermark moves on success", lambda e: (happy(e), e["store"].watermark("erp", "orders"))[1], "9"),
    ("a dataset left open succeeds when its block ends",

tests/dataplatform/test_obs_runlog.py is the contract table for the read port. The same cases run against the in-memory adapter and the PostgreSQL adapter. The PostgreSQL rows are skipped when no local server answers, and the customer adapter's rows are always skipped for now.

tests/dataplatform/test_obs_runlog.py:134What every adapter of the read port must answer.
CASES = [
    ("only the domain's runs, in start order", runs,
     [(0, "schedule", "normal", "partial"), (1, "manual", "reload", "succeeded")]),
    ("dataset runs with status, counts and watermarks", lambda snap, ids: plain(snap.datasets),
     [{"source": "erp", "dataset": "orders", "status": "succeeded", "config_fingerprint": "fp",
       "watermark_before": None, "watermark_after": "10", "counts": {"rows_written": 4}, "message": ""},
      {"source": "erp", "dataset": "returns", "status": "failed", "config_fingerprint": "fp",
       "watermark_before": None, "watermark_after": None, "counts": {}, "message": "held"},
      {"source": "erp", "dataset": "orders", "status": "succeeded", "config_fingerprint": "fp",
       "watermark_before": "10", "watermark_after": "11", "counts": {"rows_written": 1}, "message": ""}]),
    ("steps in record order", lambda snap, ids: [(s.step, s.status) for s in snap.steps],
     [("land", "succeeded"), ("quality.not_null", "succeeded"), ("detect.DEL-01", "warned")]),
    ("files with outcome and reason", lambda snap, ids: [(f.file_name, f.status, f.reason) for f in snap.files],
     [("a.csv", "loaded", ""), ("a2.csv", "skipped_duplicate", "same batch")]),
    ("errors with dataset and class", lambda snap, ids: [(e.dataset, e.error_class) for e in snap.errors],
     [("returns", "ContractMismatch")]),
    ("every record keyed by a run in the window",
     lambda snap, ids: {x.run_id for x in [*snap.datasets, *snap.steps, *snap.files, *snap.errors]} <= set(ids),
     True),
    ("times are aware and in order", lambda snap, ids: ordered(snap), True),
]

tests/dataplatform/test_obs_steps.py runs the step contract over the rows of every owner: the recorder, ingestion, the fixture runner and the gold harness. The silver row and the data expectation services row are skipped as pending. tests/dataplatform/test_obs_traced_store.py checks that a traced run produces one trace and one push. tests/dataplatform/test_obs_trace.py checks the shape of the ids.

tests/dataplatform/test_obs_steps.py:117The refusals of the step contract, one row each.
@pytest.mark.parametrize(
    ("case", "row", "problem"),
    [
        ("a pipeline step is not an owner row", ("silver.dedupe", "succeeded", {}, ""), "is not <detect"),
        ("no id after the kind", ("quality.", "succeeded", {}, ""), "is not <detect"),
        ("an unknown status", ("quality.a", "ok", {}, ""), "status 'ok'"),
        ("counts are integers", ("recon.a", "warned", {"keys": 1.5}, "x"), "counts ['keys']"),
        ("a boolean is not a count", ("recon.a", "warned", {"ok": True}, "x"), "counts ['ok']"),
        ("a blocked step says why", ("detect.SCH-02", "failed", {}, ""), "says why"),
        ("a long message", ("profile.a", "succeeded", {}, "x" * 4001), "message"),
    ],
)
Not built yet, or known gaps
  • The recorder, with its run id equal to the trace id, is not used by the ingestion, silver or gold runners. They record through the store with their own run ids, and the tracing wrapper links the two by a tag.
  • The tracing wrapper's start_dataset_run does not hand back the watermark that the store returns. No runner reads that return value today, so nothing breaks, but the wrapper is not a perfect stand-in for the store.
  • Silver does not yet write detect. rows for the conditions it handles. They are judged on what the silver tables hold.
  • Silver records all its stages together after the run, so the time of a whole silver run shows on its first stage in the step timings.