cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Operations

Data PlatformZoom level 1

Operations: how the platform is watched

Where this starts in the code
PartStarting pointCalled by (found by reading the code)
The monitor that alerts, pushed by every runMonitor obs/monitor.py:63main · scripts/run_platform.py:423
FixtureRunner.__init__ · obs/fixture.py:96
Render the operations pagerender obs/page.py:73write_ops_page · scripts/run_platform.py:141

Operations answers four questions about the platform every day. Did the loads run? Is the data as fresh as promised? Was anything held back or wrong, and who was told? Where did the time go?

The design has one rule. Everything a run does is written to the run log, a set of tables in the metadata store. Every screen, every alert and every freshness number is then worked out from the run log by pure functions. A pure function takes rows in and gives rows back, and touches nothing else. So there is one record of what happened, and no screen can disagree with another.

Start at the run log in the middle. Runs write to it on the left. Everything on the right only reads. Click a box to zoom in.

The pages under this one

PageWhat it explains
Run logWhat is recorded for a run, for each dataset in the run and for each step, how the records are read back, and how tracing is wrapped around the store.
FreshnessHow the platform decides that a dataset, or one dealer's file, is late. Also the twenty success criteria a run is scored against.
AlertsWhich conditions raise an alert, how the severity is chosen, who is told, and the four rules that keep alerts few enough to be read.
LineageWhat lineage the platform records itself, and at what grain.
Operations page and dashboardsThe operations tables, the page each run writes, and the metric views and dashboards published on Databricks.
Reconciliation and profilingReconciliation between layers, the live source compared with silver, the report each hop hands over, and profiling.
Recovery and housekeepingWithdrawing a bad batch and rebuilding by rerun, how long each copy is kept, and the housekeeping job that removes old bronze.
ResilienceRetry, the circuit breaker and the connection limit that protect calls to a source.

The design decisions that shape operations

DecisionWhat it meansWhy
One run log, read through a portRuns write through the metadata store. Operations reads through a small read-only interface that returns every run in a time window.A port is an interface with more than one implementation behind it. The same screens work on the in-memory store in tests and on PostgreSQL in a deployment.
Push, not pollWhen a run finishes, the store tells the monitor. The monitor evaluates that run's datasets at once. No job wakes on a fixed interval to look for trouble.An alert goes out when the run ends, not minutes later, and nothing runs when nothing happened.
One format for every check resultA detector, a quality rule, a reconciliation and a profile all write the same kind of step row: a name, a status, counts and a message.Operations can alert on any check and show it on a screen without knowing what the check does.
Operations never judges the dataThe part that owns a check decides whether it passed. Operations reads the status and reports it.One place decides. An alert and its owner can never disagree.
Adopt what the platform already showsJob runs, job failure messages, lineage from bronze onward and table usage are the platform's own screens. Operations builds only what the platform cannot know.Less to build and keep correct. The adopted list is in src/dataplatform/obs/native.py.
Cost is not hereNo billing data is read. Every job carries tags with the domain, the data product and the run id.Spend belongs to a separate cost tool. The tags let that tool attribute spend to a run.

How it is wired in a run

The local run script builds the wiring in four lines. It wraps the metadata store twice. The inner wrapper stamps a time on every run log write. The outer wrapper opens a trace span for each run, dataset and step, and calls the monitor when a run finishes. The ingestion, silver and gold runners receive the wrapped store and do not know it is wrapped.

scripts/run_platform.py:397The wiring: a stamping wrapper, then a tracing wrapper that pushes each finished run to the monitor.
    stamped = StampingRunLog(raw_store, clock)  # the run log with times, for the ops tables (B10)
    run_log: Any = stamped
    if a.hops != "inline":
        from dataplatform.adapters import postgres_runlog

        postgres_runlog.install(raw_store.conn)
        run_log = postgres_runlog.PostgresRunLogReader(raw_store.conn)
    monitor_box: list[Any] = []  # the monitor exists once the manifest is loaded; runs push to it as they end

    def pushed(run_id: str) -> None:
        if monitor_box:
            monitor_box[0].run_finished(run_id, clock.now())

    store = TracedStore(stamped, tracer, on_finish=(pushed,))  # every run traced and pushed (B10, ADR-024)
Not built yet, or known gaps
  • The alert state and the alert history are kept in memory on the monitor. A deployment needs a table for them, and none exists yet.
  • Alerts are delivered to an in-memory notifier in every run this page describes. The adapter for a workspace's notification destinations exists and is tested with an injected transport, but sending through a real workspace has not been done.
  • The data expectation services, which are meant to own quality rules and reconciliation results with their own tables and screen, are not built. Their screen exists and reads the shared step rows.
  • The heartbeat rule is in the alert configuration, but no code produces a heartbeat signal.
  • The run recorder that makes the run id equal to the trace id is used by the fixture and by tests. The ingestion, silver and gold runners record through the store directly with run ids such as ingest-2026-11-09.

Look inside