Data PlatformZoom level 2
The operations page, the operations tables and the dashboards
After every run the platform writes one HTML page that shows what happened: the runs, the files held back, the checks that warned, the freshness results and the alerts. The same rows are published as tables on Databricks, where dashboards read them.
The page and the dashboards never compute anything themselves. One builder turns the run log into fifteen small tables, and everything shown is a row of one of them. That is why the page on a laptop and the dashboard in the workspace show the same numbers.
The fifteen operations tables
Each table is a list of plain rows. Every table has a day column holding the business day, and the tables about runs carry the run id so they join.
| Table | One row is |
|---|---|
ops_runs | One run: status, duration, number of datasets, number failed |
ops_dataset_runs | One dataset run: status, duration, rows read and written, attempt number, whether it was slow |
ops_run_status_daily | A day and a status, with the number of dataset runs |
ops_errors_daily | A day and an error class, with the number of errors |
ops_steps | One check result in the one step format, with its owner, severity and the owner's screen |
ops_step_durations | One step of one dataset run, with the seconds it took |
ops_ingestion_daily | A day and a dataset: files loaded, duplicates skipped, files rejected, rows loaded |
ops_files_held | One file that was not loaded, with its reason |
ops_restatements | One published gold window that changed, per measure: the number before and after |
ops_freshness | One freshness result |
ops_slo_daily | A day and a dataset: was the freshness target met at every check |
ops_usage_daily | A day and a table: reads and distinct readers |
ops_lineage | One table-to-table edge with the run that wrote it |
ops_alerts | One alert event: fired, grouped, snoozed or resolved |
ops_alert_counts | A rule and a day, with the number of alerts sent |
src/dataplatform/obs/ops.py:293The builder. Its inputs are the four boxes on the left of the picture.def build(log: RunLogSnapshot, system: SystemSnapshot, freshness: list[FreshnessResult], events: list[Event],
tz: str, pipeline: list[TableUpdateRow] | None = None) -> Tables:
ds_runs = dataset_runs_table(log, system, tz, pipeline)
ingestion, held = ingestion_tables(log, tz)
fresh, slo = freshness_tables(freshness, tz)
alerts, alert_counts = alert_tables(events, tz)
errors = [{"day": business_day(e.at, tz), "error_class": e.error_class} for e in log.errors]
return {
"ops_runs": runs_table(log, system, tz),
"ops_dataset_runs": ds_runs,
"ops_run_status_daily": _count_by(ds_runs, ("day", "status")),
"ops_errors_daily": _count_by(errors, ("day", "error_class")),
"ops_steps": steps_table(log, tz),
"ops_step_durations": step_durations_table(log, tz),
"ops_ingestion_daily": ingestion,
"ops_files_held": held,
"ops_restatements": restatements_table(log, tz),
"ops_freshness": fresh,
"ops_slo_daily": slo,
"ops_usage_daily": usage_daily(system, tz),
"ops_lineage": lineage_table(system, tz),
"ops_alerts": alerts,
"ops_alert_counts": alert_counts,
}Two derived columns worth knowing
src/dataplatform/obs/ops.py:159Attempts and slow-downs, worked out per dataset from the run log alone.def _performance(rows: list[Row]) -> list[Row]:
"""Attempts per dataset per day (a retry is any attempt after the first) and the time against the dataset's own
median so far (a slow-down is SLOW_RATIO or more). Only what needs the run log: per dataset, not per job."""
seen: dict[str, list[float]] = defaultdict(list)
attempts: dict[tuple[str, str], int] = defaultdict(int)
for r in sorted(rows, key=lambda x: (x["started_at"], x["dataset"])):
key = (r["day"], r["dataset"])
attempts[key] += 1
r["attempt"], r["retry"] = attempts[key], int(attempts[key] > 1)
past, took = seen[r["dataset"]], r["duration_s"]
ratio = round(took / median(past), 3) if took is not None and past and median(past) > 0 else None
r["duration_ratio"], r["slow"] = ratio, ratio is not None and ratio >= SLOW_RATIO
if took is not None and r["status"] == "succeeded":
past.append(took)
return rowsAn attempt counts the runs of one dataset on one business day, and any attempt after the first is a retry. A run is slow when it took at least twice the median time of that dataset's own earlier successful runs. The median is the middle value, so one odd run does not move it much. The first run of a dataset has nothing to compare with and is never slow.
A worked example
Three rows of ops_dataset_runs for the material dataset, with example values. The third run takes more than twice the usual time.
ExampleThe ratio on the third day is 30 divided by the median of 12 and 14, which is 13.day dataset status duration_s attempt retry duration_ratio slow 2026-11-09 erp.material succeeded 12.0 1 0 false 2026-11-10 erp.material succeeded 14.0 1 0 1.167 false 2026-11-11 erp.material succeeded 30.0 1 0 2.308 true
The page each run writes
The page is one self-contained file: white, no scripts, no external files. It opens from disk. At the top are eight tiles, then one section per table.
src/dataplatform/obs/page.py:87The eight tiles at the top of the page. tiles = "".join([
_tile(len(runs), "runs"),
_tile(len(ds_runs), "dataset runs"),
_tile(failed, "dataset runs failed", "bad" if failed else "ok"),
_tile(int(_sum(tables.get("ops_ingestion_daily", []), "rows_loaded")), "rows loaded"),
_tile(len(held), "files held or skipped", "warn" if held else "ok"),
_tile(len(alerts), "alert events", "warn" if alerts else "ok"),
_tile(f"{len(incidents) - missed}/{len(incidents)}", "planted incidents caught", "bad" if missed else "ok"),
_tile("yes" if checks_ok else ("no" if checks_ok is False else "n/a"), "every layer equals the oracle",
"ok" if checks_ok else "bad"),
])| Section of the page | Where its rows come from |
|---|---|
| Every layer against the world's own totals | The day-by-day comparison with the oracle, passed in by the run script |
| Planted incidents and whether the platform caught them | The incident verdicts, passed in by the run script |
| Runs, and dataset runs | ops_runs, ops_dataset_runs |
| Where the time goes | ops_step_durations, summed per step, the twelve largest |
| Delay per hop | Percentiles from the time stamps silver keeps, passed in by the run script |
| Files held or skipped, with the reason | ops_files_held |
| Steps that warned or failed | ops_steps |
| Ingestion per day, freshness checks, alerts | ops_ingestion_daily, ops_freshness, ops_alerts |
The first two sections exist because the local run uses the simulated company. They show whether each layer equalled the truth on each day, which The oracle explains.
scripts/run_platform.py:123The run script gathers the inputs, renders the page, and writes the tables as a file.def write_ops_page(a: Any, days: list[Any], silver_days: list[Any], gold_days: list[Any], monitor: Any, clock: Any,
tables: Any, store: Any, domain: str = "tyre") -> Path:
"""The local ops page (B10): the ops tables the dashboards read, plus every layer against the oracle, the
planted incidents' verdicts and the delay per hop, in one HTML file beside the run."""
from dataplatform.obs.page import render
if days:
clock.set(clock.now() + timedelta(hours=20))
monitor.wake(clock.now())
ops = monitor.tables(clock.now())
checks = [{"day": d.day, "layer": "bronze", "equals_oracle": d.bronze_ok} for d in days]
checks += [{"day": s.day, "layer": "silver", "equals_oracle": s.ok} for s, _ in silver_days]
checks += [{"day": g.day, "layer": "gold", "equals_oracle": g.ok} for g, _ in gold_days]
incidents = [{"layer": layer, **v.__dict__} for layer, items in
(("bronze", [v for d in days for v in d.verdicts]),
("silver", [v for s, _ in silver_days for v in s.verdicts]),
("gold", [v for g, _ in gold_days for v in g.verdicts])) for v in items]
hops = hop_delays(a, tables, store, monitor.domain)
html = render(ops, f"{domain.capitalize()} platform: local run", f"{len(days)} day(s), scale {a.scale}, "
f"engine {a.engine}, table format {a.table_format}, change capture {a.cdc_shape}. This local run "
"loads once per simulated day, so datasets promised hourly freshness show breaches between days: "
"that is the schedule, not a fault.", checks, incidents, hops)
path = a.out / "ops.html"
path.write_text(html)
from dataplatform.obs.ops import COLUMNS
from dataplatform.obs.ops_export import export
(a.out / "ops_tables.json").write_text(json.dumps(export(ops, COLUMNS))) # a realm publishes these (dashboards)
return pathThe portable file
Plain rows show a column's type only through their values. A catalog table needs each column's type stated. The export gives every table its columns with one of six types, and values a file can carry.
src/dataplatform/obs/ops_export.py:59Declared columns win, so a table with no rows today still has its shape.def export(tables: dict[str, list[dict[str, Any]]],
declared: dict[str, tuple[tuple[str, str], ...]] | None = None) -> dict[str, Any]:
"""Declared columns win (so an empty table keeps its shape); a column the rows have beyond them is added, typed
by its values."""
out: dict[str, Any] = {}
for name in sorted(tables):
cols = list((declared or {}).get(name, ()))
cols += [c for c in column_types(tables[name]) if c[0] not in {d for d, _ in cols}]
out[name] = {"columns": [list(c) for c in cols],
"rows": [[_carry(r.get(c), t) for c, t in cols] for r in tables[name]]}
return outThe declared columns matter on a quiet day. If no file was held back, ops_files_held has no rows. Without declared columns it would be published with no columns, and the dashboard reading it would break.
Certified views over the tables
A dashboard that aggregates must not invent its own definition of a number. Twelve metric views sit over the tables. A metric view is a view that names its dimensions and its measures, so every reader computes a measure the same way. Each dashboard query that aggregates must name a measure its view certifies, with the same aggregate and the same column, and may group or filter only on the view's dimensions.
src/dataplatform/obs/views.py:71One view definition: its table, its dimensions and its one measure. OpsView("mv_ops_freshness", "ops_freshness", "Freshness checks against the manifest's targets.",
("day", "dataset", "key", "state", "tier", "breach"),
(_vm("breaches", "count", comment="Checks (filter breach = true for breaches)."),)),src/dataplatform/obs/views.py:91Why a dashboard query is refused.def problem(table: str, group_by: tuple[str, ...], filtered: tuple[str, ...],
measures: tuple[tuple[str, str, str], ...]) -> str | None:
"""Why an aggregate query cannot be answered by the certified view over its table (None when it can)."""
v = view_for(table)
if v is None:
return f"no ops metric view over {table}"
loose = [c for c in (*group_by, *filtered) if c not in v.dimensions]
if loose:
return f"{v.name} has no dimension(s) {loose}"
for name, agg, column in measures:
m = v.measure(name)
if m is None or (m.agg, m.column) != (agg, column):
return f"{v.name} certifies no measure {name} = {agg}({column})"
return Nonerealms/databricks/dashboards/views/mv_ops_freshness.sql:3The same view as generated for Databricks. The file is generated and checked for drift.CREATE OR REPLACE VIEW `dev_ops`.`ops`.`mv_ops_freshness`
WITH METRICS
LANGUAGE YAML
AS $$
version: 1.1
comment: "Freshness checks against the manifest's targets. Certified ops definition (ADR-024 rule 7)."
source: dev_ops.ops.ops_freshness
dimensions:
- name: day
expr: "`day`"
- name: dataset
expr: "`dataset`"
- name: key
expr: "`key`"
- name: state
expr: "`state`"
- name: tier
expr: "`tier`"
- name: breach
expr: "`breach`"
measures:
- name: breaches
expr: "COUNT(1)"
comment: "Checks (filter breach = true for breaches)."
$$;The dashboards
Dashboards are defined as data: queries and widgets with no product named. An adapter compiles each one to a dashboard file for Databricks. The table lists what is in the dashboards folder.
| Dashboard file | Owner | The question it answers |
|---|---|---|
platform_health.lvdash.json | Operations | Did the loads run, and did they succeed? |
freshness_slos.lvdash.json | Operations | Is the data as fresh as each dataset promises? |
performance.lvdash.json | Operations | Where does each load spend its time, and is it getting slower? |
sources_ingestion.lvdash.json | Operations | What arrived from each source, and what was held back? |
des_data_expectations.lvdash.json | Data expectation services | Is the data right? |
native/lakeflow_job_health.lvdash.json | The platform | Job runs and their health. Adopted unchanged. |
src/dataplatform/obs/dashboards.py:212One dashboard definition in full: three queries and three widgets.FRESHNESS = Dashboard(
"freshness_slos", "Freshness and SLOs",
"Is the data as fresh as each dataset's contract promises? Breaches of the manifest's own targets, per dataset "
"and per feed, and how often each target was met. Learned freshness and completeness alerts per table are the "
"platform's anomaly detection (adopted, not rebuilt); this screen keeps only the promised targets it cannot know.",
queries=(
Query("breaches", "ops_freshness", columns=("day", "dataset", "key", "age_hours", "error_hours", "tier"),
where=(("breach", "=", True),), order_by=("day",)),
Query("breaches_daily", "ops_freshness", ("day",), (_m("breaches", "count"),),
where=(("breach", "=", True),)),
Query("slo", "ops_slo_daily", ("dataset",), (_m("met_pct", "avg", "met_pct"),)),
),
widgets=(
Widget("slo_met", "Share of checks within target (%)", "bar", "slo", "dataset", "met_pct"),
Widget("breaches_per_day", "Freshness breaches per day", "bar", "breaches_daily", "day", "breaches"),
Widget("breach_list", "Breaches (dataset or feed past its error target)", "table", "breaches",
columns=("day", "dataset", "key", "age_hours", "error_hours", "tier")),
),
discharges=("DPC-0057", "FRESH-01", "OPS-01", "LATE-01"),
)Because a query is so small, the same definition can be evaluated in memory. That is how a test proves that every widget has data before anything is deployed.
Publishing on Databricks
On Databricks the run's task copies the page, the report and the tables file to a volume, then publishes the tables. Each table is written whole, replacing the last publish, because the tables are rebuilt from the whole run log on every run. The metric views are created after the tables. The generator realms/databricks/dashboards/generate.py writes the dashboard files, the view files and the bundle resources, with the catalog dev_ops and the schema ops as defaults.
src/dataplatform/adapters/spark_ops_tables.py:19Tables first, then the views. A view the session refuses is reported, not raised.def publish(spark: Any, doc: dict[str, Any], catalog: str, schema: str) -> list[str]:
from pyspark.sql import types as T # noqa: PLC0415 - Spark only where it runs
from dataplatform.adapters.databricks_ops_views import view_sql # noqa: PLC0415
from dataplatform.obs.ops_export import restore # noqa: PLC0415
from dataplatform.obs.views import VIEWS # noqa: PLC0415
bad = [n for n in (catalog, schema) if not NAME.match(n)]
if bad:
raise ValueError(f"not a plain catalog or schema name: {bad}")
notes = []
# shiftleft: allow SL-SEC-01 -- DDL takes no bind parameters; both names are checked against NAME above
spark.sql(f"CREATE SCHEMA IF NOT EXISTS `{catalog}`.`{schema}`")
for name, table in sorted(doc.items()):
cols, rows = restore(table)
if not cols:
notes.append(f"{name}: no rows and no columns, left as it was")
continue
struct = T.StructType([T.StructField(c, getattr(T, SPARK_TYPES[k])(), True) for c, k in cols])
(spark.createDataFrame(rows, struct).write.mode("overwrite").option("overwriteSchema", "true")
.saveAsTable(f"{catalog}.{schema}.{name}"))
notes.append(f"{name}: {len(rows)} rows")
for view in VIEWS:
try:
spark.sql(view_sql(view, catalog, schema).rstrip().rstrip(";"))
notes.append(f"{view.name}: view created")
except Exception as exc:
notes.append(f"{view.name}: view NOT created: {str(exc).splitlines()[0][:200]}")
return notesThe settings that drive it
No dataset setting is read here. The business time zone comes from the alert configuration and decides which day a row belongs to. The catalog and schema of the published tables are arguments of the generator and of the publish function.
How it is tested
tests/dataplatform/test_obs_page.py checks the page's rendering with a small table of cases and one test of the tiles. tests/dataplatform/test_ops_export.py has three tables for the column types, the round trip and the declared columns. tests/dataplatform/test_ops_restatements.py covers the restatement table. tests/dataplatform/test_obs_dashboards.py is the largest: it plays a week and a fortnight of the simulated company on a fixture run log and checks that every widget of every dashboard has rows, that the declared columns match what the builder produces, that every aggregate reads a certified view, and that the generated files are up to date.
tests/dataplatform/test_obs_page.py:9The page's rendering cases.CASES = [
("no rows says so", [], "Nothing recorded."),
("a failed status is marked", [{"status": "failed"}], 'class="bad"'),
("a success is marked", [{"status": "succeeded"}], 'class="ok"'),
("values are escaped", [{"reason": "<script>x</script>"}], "<script>"),
]- The dashboard tests run on a fixture run log, which is a run log written by a stand-in runner that follows the simulated company's calendar. They do not run on the run log the real engine writes.
- The operations folder's own notes still say five dashboards and fourteen tables. The code defines four operations dashboards and fifteen tables. The usage dashboard was dropped in favour of the platform's own.
ops_usage_dailyandops_lineageare built but no screen reads them.- Each publish replaces the tables. There is no history of earlier publishes beyond what the run log itself holds.
- The bundle resources file must be included from the deployment bundle by hand. I could not verify from this repository that the dashboards themselves have been deployed to a workspace.