Data PlatformZoom level 2
Freshness: how late is too late
Freshness says how old the newest data is, compared with what the dataset promised. A dataset that promises data no older than two hours is late when its last successful load is older than that.
The promise is not written in code. Each dataset's manifest states it, and the freshness functions read it from the metadata store. A dataset with no target is not checked at all.
Two measures, because one is not enough
| Measure | The question | What it is based on | What it catches |
|---|---|---|---|
| Dataset freshness | How long since this dataset last loaded successfully? | The end time of the newest dataset run with status succeeded | A source that stopped sending, or a job that stopped running |
| Feed freshness | Is one feed of this dataset behind the others? | Each bookmark of the dataset whose value is a time, compared with the newest such bookmark | One dealer's monthly file missing while every other dealer's arrived |
A bookmark is a named position the metadata store keeps for a dataset. Ingestion keeps one bookmark per dealer for the dealer sell-out files, holding the newest data time in that dealer's file. A run that found nothing new still counts as a successful load for dataset freshness: the platform looked, and it is current with its source.
A worked example
The targets below are the tyre company's, as the test reads them from the loaded manifest. The first column is the dataset. The next two are its targets in hours. The tier and the rota follow from them and are explained further down.
| Dataset | Warning after | Error after | Tier | Alerts go to |
|---|---|---|---|---|
erp.sales_order_line | 1 hour | 2 hours | 1 | erp-data-rota |
erp.material | 24 hours | 48 hours | 2 | the rota named in its manifest |
partner.dealer_sellout | 72 hours | 120 hours | 3 | partner-data-rota |
Now a check with example values. The order lines last loaded 1.5 hours ago, so the state is warn and nobody is told. The dealer sell-out dataset loaded 0.2 hours ago, so the dataset is ok. But its bookmarks tell a second story.
ExampleDealer D-42's bookmark stands at the end of September while the others stand at the end of October. 744 hours is 31 days.dataset key basis age in hours state erp.sales_order_line last_success 1.5 warn partner.dealer_sellout last_success 0.2 ok partner.dealer_sellout D-1 bookmark_lag 0.0 ok partner.dealer_sellout D-2 bookmark_lag 0.0 ok partner.dealer_sellout D-42 bookmark_lag 744.0 error
The dataset loaded today, yet one dealer is a month behind. Dataset freshness alone would have said all is well. This is the late dealer file that the simulated company plants, and the planted incidents page follows it through the layers.
The settings that drive it
The table lists every setting freshness reads. The registry is the list of known settings in src/dataplatform/meta/settings_registry.json.
| Setting | Where it is set | Meaning | Default |
|---|---|---|---|
sla_freshness_hours | Dataset settings. Registered in the registry as a whole number. | Hours after which the dataset is in the warn state. | None. Without it the dataset has no target. |
sla_freshness_error_hours | Dataset settings. In the registry with status proposed. | Hours after which the dataset is in the error state. | Twice the warning target. |
owners.alerts | The dataset manifest | The rota that alerts go to. | The dataset owner. |
owners.dataset | The dataset manifest | The dataset's owner. | Empty. |
tiers | The alert configuration, src/dataplatform/obs/rules/alerts.yaml | A tier forced for a named dataset. | Empty: the tier follows the error target. |
data-platform-catalog/slices/02-tyre/manifest/datasets/partner.dealer_sellout.yaml:14The dealer sell-out dataset's two targets, as its manifest states them.sla_freshness_hours: "72" sla_freshness_error_hours: "120"
The code
src/dataplatform/obs/freshness.py:73A dataset's targets, read from the merged view the metadata store gives.def target_from_view(view: dict[str, Any], tiers: dict[str, int] | None = None) -> FreshnessTarget | None:
"""A dataset's targets from the store's merged view; None when the dataset declares no freshness target."""
settings = view.get("settings") or {}
if "sla_freshness_hours" not in settings:
return None
warn = float(settings["sla_freshness_hours"])
error = float(settings.get("sla_freshness_error_hours", warn * 2))
owners = (view.get("manifest") or {}).get("owners") or {}
name = f"{view['source']}.{view['dataset']}"
return FreshnessTarget(view["source"], view["dataset"], warn, error, tier_for(error, (tiers or {}).get(name)),
str(owners.get("alerts") or owners.get("dataset") or ""),
str(owners.get("dataset") or ""))The function returns nothing when the dataset has no warning target, and that dataset is then left out of every freshness check. The error target falls back to twice the warning target. The rota falls back to the dataset owner.
src/dataplatform/obs/freshness.py:64The tier from the error target, unless the alert configuration forces one.def tier_for(error_hours: float, override: int | None = None) -> int:
if override is not None:
return override
for limit, tier in TIER_BY_ERROR_HOURS:
if error_hours <= limit:
return tier
return 3A tier says how urgent a dataset is. An error target of 4 hours or less is tier 1. An error target of 48 hours or less is tier 2. Anything longer is tier 3. The alerts page shows what each tier means for who is told.
src/dataplatform/obs/freshness.py:98The whole rule for a state.def state_of(age_hours: float | None, warn: float, error: float) -> str:
if age_hours is None:
return "never"
if age_hours > error:
return "error"
return "warn" if age_hours > warn else "ok"src/dataplatform/obs/freshness.py:127Both measures. It returns one result per dataset, then one per time-valued bookmark.def check(targets: list[FreshnessTarget], datasets: list[DatasetRunRow],
bookmarks: dict[tuple[str, str], dict[str, str]], now: datetime) -> list[FreshnessResult]:
"""Every target's dataset freshness, then each time-valued bookmark's lag behind its newest sibling."""
seen = last_success(datasets, now)
out = []
for t in targets:
at = seen.get((t.source, t.dataset))
age = _hours(now, at) if at else None
out.append(FreshnessResult(t.source, t.dataset, "", "last_success" if at else "none", age,
t.warn_hours, t.error_hours, state_of(age, t.warn_hours, t.error_hours),
t.tier, now))
times = {k: v for k, raw in sorted(bookmarks.get((t.source, t.dataset), {}).items())
if (v := _as_time(raw)) is not None}
if len(times) < 2: # noqa: PLR2004 - a lag needs a sibling to lag behind
continue
newest = max(times.values())
for key, value in times.items():
lag = _hours(newest, value)
out.append(FreshnessResult(t.source, t.dataset, key, "bookmark_lag", lag, t.warn_hours,
t.error_hours, state_of(lag, t.warn_hours, t.error_hours), t.tier, now))
return outWalk through it. The helper last_success finds, for each dataset, the newest run that succeeded, has an end time, and ended before now. A run still open does not count, and a failed run does not count. The age is the hours from that end time to now. Then the bookmarks: only values that parse as a time with a time zone are kept, so a shard position such as 1234 is ignored. With fewer than two such bookmarks there is nothing to lag behind, and the loop moves on. Otherwise each bookmark's lag is its distance from the newest one, and the same two targets decide its state.
src/dataplatform/obs/freshness.py:154The one moment at which a lateness check is worth running.def next_due(targets: list[FreshnessTarget], datasets: list[DatasetRunRow], now: datetime) -> datetime | None:
"""When the next dataset would pass its error target if nothing loads before then: the one moment a lateness
check is worth running (ADR-024 rule 4, no fixed interval). None when nothing can become late (never loaded
datasets are already reported as `never`; a bookmark lag does not grow with the clock)."""
seen = last_success(datasets, now)
due = [at + timedelta(hours=t.error_hours) for t in targets if (at := seen.get((t.source, t.dataset)))]
later = [d for d in due if d > now]
return min(later) if later else NoneLateness has no event. Nothing happens when data fails to arrive. So instead of checking on a fixed interval, the monitor asks this function when the next dataset would pass its error target if nothing loads, and checks only then. A dataset that is already late is skipped, because checking it again changes nothing.
The success criteria and the scorecard
Freshness is one promise. The platform also keeps one list of twenty criteria that say what a good run means, each with a single metric and a target. After a run, a scorecard gives each criterion one of three states: met, not met, or not measured. A criterion the run cannot see is never counted as met.
src/dataplatform/obs/success.py:35The twenty criteria. Timeliness is C7 and C8.CRITERIA = (
Criterion("C1", "Correctness", "Days on which every layer equals the oracle", "100% of days, every layer"),
Criterion("C2", "Correctness", "Certified metric answers equal to gold and the world", "100% of known answers"),
Criterion("C3", "Correctness", "Live source agrees with silver, or the missing keys and hop are named", "100%"),
Criterion("C4", "Correctness", "Gold fact runs whose row ledger balances", "100%"),
Criterion("C5", "Fault handling", "Planted faults handled at their layer, none silent", "every planted fault"),
Criterion("C6", "Fault handling", "False alarms on clean days", "0"),
Criterion("C7", "Timeliness", "Days whose gold is complete by 06:00 business time", "95% of days"),
Criterion("C8", "Timeliness", "Hop latency p95 the platform owns (bronze to silver)", "30 minutes"),
Criterion("C9", "Performance", "One day's load, all layers, scale 0.1", "6 minutes or less"),
Criterion("C10", "Cost", "Cost per daily load per data product", "within budget (after a 14-day baseline)"),
Criterion("C11", "Reliability", "Rerunning a day changes nothing", "0 duplicate rows, 0 changed windows"),
Criterion("C12", "Reliability", "Two runs writing one table at once", "never"),
Criterion("C13", "Recovery", "A bad batch withdrawn from every layer, changed windows logged", "one run cycle"),
Criterion("C14", "Explainability", "A past window rebuilt from its run log equals what it published", "100%"),
Criterion("C15", "Privacy", "Restricted values in clear in bronze, silver or gold", "0"),
Criterion("C16", "Privacy", "Right-to-be-forgotten requests completed", "within 30 days"),
Criterion("C17", "Security", "Drift between the access model and catalog grants", "0"),
Criterion("C18", "Security", "Secrets in code, logs or the run log", "0"),
Criterion("C19", "Change safety", "Unexplained data differences that reach production", "0"),
Criterion("C20", "Coverage", "DAC rules with a passing test or a written waiver", "89 of 89"),
)Eight of the twenty are measured by a run from its own evidence: C1 to C5, C8, C9 and C15. The other twelve are reported as not measured, with the note that they are measured outside the run. The two timeliness numbers in the code are these. The hop from bronze to silver must take 30 minutes or less at the 95th percentile. One simulated day through all layers at one tenth of full volume must take 6 minutes or less.
src/dataplatform/obs/success.py:122The hop latency criterion. The hop from source commit to bronze is shown as context and not judged.def _c8(e: Evidence) -> tuple[str, str, str]:
seen = {h: s for h, s in e.hops.items() if h in HOP_TARGETS}
if not seen:
return "not_measured", "", "no hop stamps in this run"
over = {h: s for h, s in seen.items() if s > HOP_TARGETS[h]}
value = ", ".join(f"{h} {s / 60:.1f} min" for h, s in sorted(seen.items()))
context = "; ".join(f"{h} {e.hops[h] / 60:.0f} min (context, not judged)" for h in CONTEXT_HOPS if h in e.hops)
why = "worst dataset p95, from the world's clock (simulated schedule)" + (f"; {context}" if context else "")
return ("not_met" if over else "met"), value, whyThe scorecard rows travel with the operations tables as a sixteenth table, ops_success, so the run's score can be published beside its other tables.
How it is tested
tests/dataplatform/test_obs_freshness.py is table-driven. One table covers the states, one the tiers, one the full check, one the targets read from a view, and one the next due time. A further test loads the tyre manifest into the in-memory store and checks that nine datasets have targets with the values shown above. tests/dataplatform/test_obs_success.py has one case per measured criterion for met, not met and not seen.
tests/dataplatform/test_obs_freshness.py:48The cases for the full check: runs, bookmarks, and the results expected.DS = [
("never loaded", [], {}, [("", "none", None, "never")]),
("fresh", [ran(0.5)], {}, [("", "last_success", 0.5, "ok")]),
("newest success counts, failures do not", [ran(5), ran(1.5), ran(0.1, "failed")], {},
[("", "last_success", 1.5, "warn")]),
("a run still open does not count", [ran(3), ran(0.1, ended=False)], {}, [("", "last_success", 3.0, "error")]),
("a run after the check time does not count", [ran(-1), ran(3)], {}, [("", "last_success", 3.0, "error")]),
("one lagging bookmark is an error, its siblings ok",
[ran(0.2)], {"D-1": "2026-10-31T23:59:59+05:30", "D-2": "2026-10-31T23:59:59+05:30",
"D-42": "2026-09-30T23:59:59+05:30"},
[("", "last_success", 0.2, "ok"), ("D-1", "bookmark_lag", 0.0, "ok"), ("D-2", "bookmark_lag", 0.0, "ok"),
("D-42", "bookmark_lag", 744.0, "error")]),
("bookmarks that are not times are ignored", [ran(0.2)], {"shard-0": "1234", "shard-1": "1300"},
[("", "last_success", 0.2, "ok")]),
("a single bookmark has nothing to lag behind", [ran(0.2)], {"D-1": "2026-10-01T00:00:00+00:00"},
[("", "last_success", 0.2, "ok")]),
]- Criterion C7, the share of days whose gold is complete by 06:00 business time, is in the list but no code measures it.
- The freshness target is measured from the end of the last successful load, not from the age of the newest row in gold. A freshness target measured end to end is an obligation in the catalog that no test cites.
- In a local simulated run the platform loads once per simulated day, so datasets that promise hourly freshness show breaches between days. The operations page says so in its subtitle.
- Criterion C20's target text says 89 of 89 rules. The catalog snapshot beside this code holds 105 obligations, as Proof by layer reports.
- Hop latency from a simulated run uses the simulated clock, so it describes the schedule and not the speed of a real network.