cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Operations › Operations page and dashboards

Data PlatformZoom level 2

The operations page, the operations tables and the dashboards

After every run the platform writes one HTML page that shows what happened: the runs, the files held back, the checks that warned, the freshness results and the alerts. The same rows are published as tables on Databricks, where dashboards read them.

The page and the dashboards never compute anything themselves. One builder turns the run log into fifteen small tables, and everything shown is a row of one of them. That is why the page on a laptop and the dashboard in the workspace show the same numbers.

Open: Run logRun logzoom in ›Open: Freshness resultsFreshness resultszoom in ›Open: Alert eventsAlert eventszoom in ›Platform telemetrydaily copyops.buildpure: rows in, rows out15 ops tablesplain rowsops.htmlone file per runops_tables.jsontyped, portableCatalog tables,views, dashboards
The builder in the middle is the only place where numbers are worked out. Follow its output to the three places it is shown.

The fifteen operations tables

Each table is a list of plain rows. Every table has a day column holding the business day, and the tables about runs carry the run id so they join.

TableOne row is
ops_runsOne run: status, duration, number of datasets, number failed
ops_dataset_runsOne dataset run: status, duration, rows read and written, attempt number, whether it was slow
ops_run_status_dailyA day and a status, with the number of dataset runs
ops_errors_dailyA day and an error class, with the number of errors
ops_stepsOne check result in the one step format, with its owner, severity and the owner's screen
ops_step_durationsOne step of one dataset run, with the seconds it took
ops_ingestion_dailyA day and a dataset: files loaded, duplicates skipped, files rejected, rows loaded
ops_files_heldOne file that was not loaded, with its reason
ops_restatementsOne published gold window that changed, per measure: the number before and after
ops_freshnessOne freshness result
ops_slo_dailyA day and a dataset: was the freshness target met at every check
ops_usage_dailyA day and a table: reads and distinct readers
ops_lineageOne table-to-table edge with the run that wrote it
ops_alertsOne alert event: fired, grouped, snoozed or resolved
ops_alert_countsA rule and a day, with the number of alerts sent
src/dataplatform/obs/ops.py:293The builder. Its inputs are the four boxes on the left of the picture.
def build(log: RunLogSnapshot, system: SystemSnapshot, freshness: list[FreshnessResult], events: list[Event],
          tz: str, pipeline: list[TableUpdateRow] | None = None) -> Tables:
    ds_runs = dataset_runs_table(log, system, tz, pipeline)
    ingestion, held = ingestion_tables(log, tz)
    fresh, slo = freshness_tables(freshness, tz)
    alerts, alert_counts = alert_tables(events, tz)
    errors = [{"day": business_day(e.at, tz), "error_class": e.error_class} for e in log.errors]
    return {
        "ops_runs": runs_table(log, system, tz),
        "ops_dataset_runs": ds_runs,
        "ops_run_status_daily": _count_by(ds_runs, ("day", "status")),
        "ops_errors_daily": _count_by(errors, ("day", "error_class")),
        "ops_steps": steps_table(log, tz),
        "ops_step_durations": step_durations_table(log, tz),
        "ops_ingestion_daily": ingestion,
        "ops_files_held": held,
        "ops_restatements": restatements_table(log, tz),
        "ops_freshness": fresh,
        "ops_slo_daily": slo,
        "ops_usage_daily": usage_daily(system, tz),
        "ops_lineage": lineage_table(system, tz),
        "ops_alerts": alerts,
        "ops_alert_counts": alert_counts,
    }

Two derived columns worth knowing

src/dataplatform/obs/ops.py:159Attempts and slow-downs, worked out per dataset from the run log alone.
def _performance(rows: list[Row]) -> list[Row]:
    """Attempts per dataset per day (a retry is any attempt after the first) and the time against the dataset's own
    median so far (a slow-down is SLOW_RATIO or more). Only what needs the run log: per dataset, not per job."""
    seen: dict[str, list[float]] = defaultdict(list)
    attempts: dict[tuple[str, str], int] = defaultdict(int)
    for r in sorted(rows, key=lambda x: (x["started_at"], x["dataset"])):
        key = (r["day"], r["dataset"])
        attempts[key] += 1
        r["attempt"], r["retry"] = attempts[key], int(attempts[key] > 1)
        past, took = seen[r["dataset"]], r["duration_s"]
        ratio = round(took / median(past), 3) if took is not None and past and median(past) > 0 else None
        r["duration_ratio"], r["slow"] = ratio, ratio is not None and ratio >= SLOW_RATIO
        if took is not None and r["status"] == "succeeded":
            past.append(took)
    return rows

An attempt counts the runs of one dataset on one business day, and any attempt after the first is a retry. A run is slow when it took at least twice the median time of that dataset's own earlier successful runs. The median is the middle value, so one odd run does not move it much. The first run of a dataset has nothing to compare with and is never slow.

A worked example

Three rows of ops_dataset_runs for the material dataset, with example values. The third run takes more than twice the usual time.

ExampleThe ratio on the third day is 30 divided by the median of 12 and 14, which is 13.
day         dataset        status     duration_s  attempt  retry  duration_ratio  slow
2026-11-09  erp.material   succeeded    12.0        1        0                      false
2026-11-10  erp.material   succeeded    14.0        1        0      1.167           false
2026-11-11  erp.material   succeeded    30.0        1        0      2.308           true

The page each run writes

The page is one self-contained file: white, no scripts, no external files. It opens from disk. At the top are eight tiles, then one section per table.

src/dataplatform/obs/page.py:87The eight tiles at the top of the page.
    tiles = "".join([
        _tile(len(runs), "runs"),
        _tile(len(ds_runs), "dataset runs"),
        _tile(failed, "dataset runs failed", "bad" if failed else "ok"),
        _tile(int(_sum(tables.get("ops_ingestion_daily", []), "rows_loaded")), "rows loaded"),
        _tile(len(held), "files held or skipped", "warn" if held else "ok"),
        _tile(len(alerts), "alert events", "warn" if alerts else "ok"),
        _tile(f"{len(incidents) - missed}/{len(incidents)}", "planted incidents caught", "bad" if missed else "ok"),
        _tile("yes" if checks_ok else ("no" if checks_ok is False else "n/a"), "every layer equals the oracle",
              "ok" if checks_ok else "bad"),
    ])
Section of the pageWhere its rows come from
Every layer against the world's own totalsThe day-by-day comparison with the oracle, passed in by the run script
Planted incidents and whether the platform caught themThe incident verdicts, passed in by the run script
Runs, and dataset runsops_runs, ops_dataset_runs
Where the time goesops_step_durations, summed per step, the twelve largest
Delay per hopPercentiles from the time stamps silver keeps, passed in by the run script
Files held or skipped, with the reasonops_files_held
Steps that warned or failedops_steps
Ingestion per day, freshness checks, alertsops_ingestion_daily, ops_freshness, ops_alerts

The first two sections exist because the local run uses the simulated company. They show whether each layer equalled the truth on each day, which The oracle explains.

scripts/run_platform.py:123The run script gathers the inputs, renders the page, and writes the tables as a file.
def write_ops_page(a: Any, days: list[Any], silver_days: list[Any], gold_days: list[Any], monitor: Any, clock: Any,
                   tables: Any, store: Any, domain: str = "tyre") -> Path:
    """The local ops page (B10): the ops tables the dashboards read, plus every layer against the oracle, the
    planted incidents' verdicts and the delay per hop, in one HTML file beside the run."""
    from dataplatform.obs.page import render

    if days:
        clock.set(clock.now() + timedelta(hours=20))
        monitor.wake(clock.now())
    ops = monitor.tables(clock.now())
    checks = [{"day": d.day, "layer": "bronze", "equals_oracle": d.bronze_ok} for d in days]
    checks += [{"day": s.day, "layer": "silver", "equals_oracle": s.ok} for s, _ in silver_days]
    checks += [{"day": g.day, "layer": "gold", "equals_oracle": g.ok} for g, _ in gold_days]
    incidents = [{"layer": layer, **v.__dict__} for layer, items in
                 (("bronze", [v for d in days for v in d.verdicts]),
                  ("silver", [v for s, _ in silver_days for v in s.verdicts]),
                  ("gold", [v for g, _ in gold_days for v in g.verdicts])) for v in items]
    hops = hop_delays(a, tables, store, monitor.domain)
    html = render(ops, f"{domain.capitalize()} platform: local run", f"{len(days)} day(s), scale {a.scale}, "
                  f"engine {a.engine}, table format {a.table_format}, change capture {a.cdc_shape}. This local run "
                  "loads once per simulated day, so datasets promised hourly freshness show breaches between days: "
                  "that is the schedule, not a fault.", checks, incidents, hops)
    path = a.out / "ops.html"
    path.write_text(html)
    from dataplatform.obs.ops import COLUMNS
    from dataplatform.obs.ops_export import export

    (a.out / "ops_tables.json").write_text(json.dumps(export(ops, COLUMNS)))  # a realm publishes these (dashboards)
    return path

The portable file

Plain rows show a column's type only through their values. A catalog table needs each column's type stated. The export gives every table its columns with one of six types, and values a file can carry.

src/dataplatform/obs/ops_export.py:59Declared columns win, so a table with no rows today still has its shape.
def export(tables: dict[str, list[dict[str, Any]]],
           declared: dict[str, tuple[tuple[str, str], ...]] | None = None) -> dict[str, Any]:
    """Declared columns win (so an empty table keeps its shape); a column the rows have beyond them is added, typed
    by its values."""
    out: dict[str, Any] = {}
    for name in sorted(tables):
        cols = list((declared or {}).get(name, ()))
        cols += [c for c in column_types(tables[name]) if c[0] not in {d for d, _ in cols}]
        out[name] = {"columns": [list(c) for c in cols],
                     "rows": [[_carry(r.get(c), t) for c, t in cols] for r in tables[name]]}
    return out

The declared columns matter on a quiet day. If no file was held back, ops_files_held has no rows. Without declared columns it would be published with no columns, and the dashboard reading it would break.

Certified views over the tables

A dashboard that aggregates must not invent its own definition of a number. Twelve metric views sit over the tables. A metric view is a view that names its dimensions and its measures, so every reader computes a measure the same way. Each dashboard query that aggregates must name a measure its view certifies, with the same aggregate and the same column, and may group or filter only on the view's dimensions.

src/dataplatform/obs/views.py:71One view definition: its table, its dimensions and its one measure.
    OpsView("mv_ops_freshness", "ops_freshness", "Freshness checks against the manifest's targets.",
            ("day", "dataset", "key", "state", "tier", "breach"),
            (_vm("breaches", "count", comment="Checks (filter breach = true for breaches)."),)),
src/dataplatform/obs/views.py:91Why a dashboard query is refused.
def problem(table: str, group_by: tuple[str, ...], filtered: tuple[str, ...],
            measures: tuple[tuple[str, str, str], ...]) -> str | None:
    """Why an aggregate query cannot be answered by the certified view over its table (None when it can)."""
    v = view_for(table)
    if v is None:
        return f"no ops metric view over {table}"
    loose = [c for c in (*group_by, *filtered) if c not in v.dimensions]
    if loose:
        return f"{v.name} has no dimension(s) {loose}"
    for name, agg, column in measures:
        m = v.measure(name)
        if m is None or (m.agg, m.column) != (agg, column):
            return f"{v.name} certifies no measure {name} = {agg}({column})"
    return None
realms/databricks/dashboards/views/mv_ops_freshness.sql:3The same view as generated for Databricks. The file is generated and checked for drift.
CREATE OR REPLACE VIEW `dev_ops`.`ops`.`mv_ops_freshness`
WITH METRICS
LANGUAGE YAML
AS $$
version: 1.1
comment: "Freshness checks against the manifest's targets. Certified ops definition (ADR-024 rule 7)."
source: dev_ops.ops.ops_freshness
dimensions:
  - name: day
    expr: "`day`"
  - name: dataset
    expr: "`dataset`"
  - name: key
    expr: "`key`"
  - name: state
    expr: "`state`"
  - name: tier
    expr: "`tier`"
  - name: breach
    expr: "`breach`"
measures:
  - name: breaches
    expr: "COUNT(1)"
    comment: "Checks (filter breach = true for breaches)."
$$;

The dashboards

Dashboards are defined as data: queries and widgets with no product named. An adapter compiles each one to a dashboard file for Databricks. The table lists what is in the dashboards folder.

Dashboard fileOwnerThe question it answers
platform_health.lvdash.jsonOperationsDid the loads run, and did they succeed?
freshness_slos.lvdash.jsonOperationsIs the data as fresh as each dataset promises?
performance.lvdash.jsonOperationsWhere does each load spend its time, and is it getting slower?
sources_ingestion.lvdash.jsonOperationsWhat arrived from each source, and what was held back?
des_data_expectations.lvdash.jsonData expectation servicesIs the data right?
native/lakeflow_job_health.lvdash.jsonThe platformJob runs and their health. Adopted unchanged.
src/dataplatform/obs/dashboards.py:212One dashboard definition in full: three queries and three widgets.
FRESHNESS = Dashboard(
    "freshness_slos", "Freshness and SLOs",
    "Is the data as fresh as each dataset's contract promises? Breaches of the manifest's own targets, per dataset "
    "and per feed, and how often each target was met. Learned freshness and completeness alerts per table are the "
    "platform's anomaly detection (adopted, not rebuilt); this screen keeps only the promised targets it cannot know.",
    queries=(
        Query("breaches", "ops_freshness", columns=("day", "dataset", "key", "age_hours", "error_hours", "tier"),
              where=(("breach", "=", True),), order_by=("day",)),
        Query("breaches_daily", "ops_freshness", ("day",), (_m("breaches", "count"),),
              where=(("breach", "=", True),)),
        Query("slo", "ops_slo_daily", ("dataset",), (_m("met_pct", "avg", "met_pct"),)),
    ),
    widgets=(
        Widget("slo_met", "Share of checks within target (%)", "bar", "slo", "dataset", "met_pct"),
        Widget("breaches_per_day", "Freshness breaches per day", "bar", "breaches_daily", "day", "breaches"),
        Widget("breach_list", "Breaches (dataset or feed past its error target)", "table", "breaches",
               columns=("day", "dataset", "key", "age_hours", "error_hours", "tier")),
    ),
    discharges=("DPC-0057", "FRESH-01", "OPS-01", "LATE-01"),
)

Because a query is so small, the same definition can be evaluated in memory. That is how a test proves that every widget has data before anything is deployed.

Publishing on Databricks

On Databricks the run's task copies the page, the report and the tables file to a volume, then publishes the tables. Each table is written whole, replacing the last publish, because the tables are rebuilt from the whole run log on every run. The metric views are created after the tables. The generator realms/databricks/dashboards/generate.py writes the dashboard files, the view files and the bundle resources, with the catalog dev_ops and the schema ops as defaults.

src/dataplatform/adapters/spark_ops_tables.py:19Tables first, then the views. A view the session refuses is reported, not raised.
def publish(spark: Any, doc: dict[str, Any], catalog: str, schema: str) -> list[str]:
    from pyspark.sql import types as T  # noqa: PLC0415 - Spark only where it runs

    from dataplatform.adapters.databricks_ops_views import view_sql  # noqa: PLC0415
    from dataplatform.obs.ops_export import restore  # noqa: PLC0415
    from dataplatform.obs.views import VIEWS  # noqa: PLC0415

    bad = [n for n in (catalog, schema) if not NAME.match(n)]
    if bad:
        raise ValueError(f"not a plain catalog or schema name: {bad}")
    notes = []
    # shiftleft: allow SL-SEC-01 -- DDL takes no bind parameters; both names are checked against NAME above
    spark.sql(f"CREATE SCHEMA IF NOT EXISTS `{catalog}`.`{schema}`")
    for name, table in sorted(doc.items()):
        cols, rows = restore(table)
        if not cols:
            notes.append(f"{name}: no rows and no columns, left as it was")
            continue
        struct = T.StructType([T.StructField(c, getattr(T, SPARK_TYPES[k])(), True) for c, k in cols])
        (spark.createDataFrame(rows, struct).write.mode("overwrite").option("overwriteSchema", "true")
         .saveAsTable(f"{catalog}.{schema}.{name}"))
        notes.append(f"{name}: {len(rows)} rows")
    for view in VIEWS:
        try:
            spark.sql(view_sql(view, catalog, schema).rstrip().rstrip(";"))
            notes.append(f"{view.name}: view created")
        except Exception as exc:
            notes.append(f"{view.name}: view NOT created: {str(exc).splitlines()[0][:200]}")
    return notes

The settings that drive it

No dataset setting is read here. The business time zone comes from the alert configuration and decides which day a row belongs to. The catalog and schema of the published tables are arguments of the generator and of the publish function.

How it is tested

tests/dataplatform/test_obs_page.py checks the page's rendering with a small table of cases and one test of the tiles. tests/dataplatform/test_ops_export.py has three tables for the column types, the round trip and the declared columns. tests/dataplatform/test_ops_restatements.py covers the restatement table. tests/dataplatform/test_obs_dashboards.py is the largest: it plays a week and a fortnight of the simulated company on a fixture run log and checks that every widget of every dashboard has rows, that the declared columns match what the builder produces, that every aggregate reads a certified view, and that the generated files are up to date.

tests/dataplatform/test_obs_page.py:9The page's rendering cases.
CASES = [
    ("no rows says so", [], "Nothing recorded."),
    ("a failed status is marked", [{"status": "failed"}], 'class="bad"'),
    ("a success is marked", [{"status": "succeeded"}], 'class="ok"'),
    ("values are escaped", [{"reason": "<script>x</script>"}], "&lt;script&gt;"),
]
Not built yet, or known gaps
  • The dashboard tests run on a fixture run log, which is a run log written by a stand-in runner that follows the simulated company's calendar. They do not run on the run log the real engine writes.
  • The operations folder's own notes still say five dashboards and fourteen tables. The code defines four operations dashboards and fifteen tables. The usage dashboard was dropped in favour of the platform's own.
  • ops_usage_daily and ops_lineage are built but no screen reads them.
  • Each publish replaces the tables. There is no history of earlier publishes beyond what the run log itself holds.
  • The bundle resources file must be included from the deployment bundle by hand. I could not verify from this repository that the dashboards themselves have been deployed to a workspace.