cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Operations › Freshness

Data PlatformZoom level 2

Freshness: how late is too late

Freshness says how old the newest data is, compared with what the dataset promised. A dataset that promises data no older than two hours is late when its last successful load is older than that.

The promise is not written in code. Each dataset's manifest states it, and the freshness functions read it from the metadata store. A dataset with no target is not checked at all.

Hours since the dataset last loaded successfullyokup to the warning targetwarnpast the warning targeterrorpast the error targetsla_freshness_hourssla_freshness_error_hoursOnly the error state raises an alert. A dataset that never loaded has the state never.
Read left to right as time passes without a load. The two arrows are the two settings.

Two measures, because one is not enough

MeasureThe questionWhat it is based onWhat it catches
Dataset freshnessHow long since this dataset last loaded successfully?The end time of the newest dataset run with status succeededA source that stopped sending, or a job that stopped running
Feed freshnessIs one feed of this dataset behind the others?Each bookmark of the dataset whose value is a time, compared with the newest such bookmarkOne dealer's monthly file missing while every other dealer's arrived

A bookmark is a named position the metadata store keeps for a dataset. Ingestion keeps one bookmark per dealer for the dealer sell-out files, holding the newest data time in that dealer's file. A run that found nothing new still counts as a successful load for dataset freshness: the platform looked, and it is current with its source.

Goes inThe dataset's targets from the metadata store, the dataset runs from the run log, and the dataset's bookmarks.
What happensFor each dataset with a target, the age of the last success is compared with the two targets. Then each time-valued bookmark is compared with its newest sibling.
Comes outOne result per dataset and one per bookmark, each with a state: ok, warn, error or never.

A worked example

The targets below are the tyre company's, as the test reads them from the loaded manifest. The first column is the dataset. The next two are its targets in hours. The tier and the rota follow from them and are explained further down.

DatasetWarning afterError afterTierAlerts go to
erp.sales_order_line1 hour2 hours1erp-data-rota
erp.material24 hours48 hours2the rota named in its manifest
partner.dealer_sellout72 hours120 hours3partner-data-rota

Now a check with example values. The order lines last loaded 1.5 hours ago, so the state is warn and nobody is told. The dealer sell-out dataset loaded 0.2 hours ago, so the dataset is ok. But its bookmarks tell a second story.

ExampleDealer D-42's bookmark stands at the end of September while the others stand at the end of October. 744 hours is 31 days.
dataset                   key     basis          age in hours   state
erp.sales_order_line              last_success     1.5          warn
partner.dealer_sellout            last_success     0.2          ok
partner.dealer_sellout    D-1     bookmark_lag     0.0          ok
partner.dealer_sellout    D-2     bookmark_lag     0.0          ok
partner.dealer_sellout    D-42    bookmark_lag   744.0          error

The dataset loaded today, yet one dealer is a month behind. Dataset freshness alone would have said all is well. This is the late dealer file that the simulated company plants, and the planted incidents page follows it through the layers.

The settings that drive it

The table lists every setting freshness reads. The registry is the list of known settings in src/dataplatform/meta/settings_registry.json.

SettingWhere it is setMeaningDefault
sla_freshness_hoursDataset settings. Registered in the registry as a whole number.Hours after which the dataset is in the warn state.None. Without it the dataset has no target.
sla_freshness_error_hoursDataset settings. In the registry with status proposed.Hours after which the dataset is in the error state.Twice the warning target.
owners.alertsThe dataset manifestThe rota that alerts go to.The dataset owner.
owners.datasetThe dataset manifestThe dataset's owner.Empty.
tiersThe alert configuration, src/dataplatform/obs/rules/alerts.yamlA tier forced for a named dataset.Empty: the tier follows the error target.
data-platform-catalog/slices/02-tyre/manifest/datasets/partner.dealer_sellout.yaml:14The dealer sell-out dataset's two targets, as its manifest states them.
  sla_freshness_hours: "72"
  sla_freshness_error_hours: "120"

The code

src/dataplatform/obs/freshness.py:73A dataset's targets, read from the merged view the metadata store gives.
def target_from_view(view: dict[str, Any], tiers: dict[str, int] | None = None) -> FreshnessTarget | None:
    """A dataset's targets from the store's merged view; None when the dataset declares no freshness target."""
    settings = view.get("settings") or {}
    if "sla_freshness_hours" not in settings:
        return None
    warn = float(settings["sla_freshness_hours"])
    error = float(settings.get("sla_freshness_error_hours", warn * 2))
    owners = (view.get("manifest") or {}).get("owners") or {}
    name = f"{view['source']}.{view['dataset']}"
    return FreshnessTarget(view["source"], view["dataset"], warn, error, tier_for(error, (tiers or {}).get(name)),
                           str(owners.get("alerts") or owners.get("dataset") or ""),
                           str(owners.get("dataset") or ""))

The function returns nothing when the dataset has no warning target, and that dataset is then left out of every freshness check. The error target falls back to twice the warning target. The rota falls back to the dataset owner.

src/dataplatform/obs/freshness.py:64The tier from the error target, unless the alert configuration forces one.
def tier_for(error_hours: float, override: int | None = None) -> int:
    if override is not None:
        return override
    for limit, tier in TIER_BY_ERROR_HOURS:
        if error_hours <= limit:
            return tier
    return 3

A tier says how urgent a dataset is. An error target of 4 hours or less is tier 1. An error target of 48 hours or less is tier 2. Anything longer is tier 3. The alerts page shows what each tier means for who is told.

src/dataplatform/obs/freshness.py:98The whole rule for a state.
def state_of(age_hours: float | None, warn: float, error: float) -> str:
    if age_hours is None:
        return "never"
    if age_hours > error:
        return "error"
    return "warn" if age_hours > warn else "ok"
src/dataplatform/obs/freshness.py:127Both measures. It returns one result per dataset, then one per time-valued bookmark.
def check(targets: list[FreshnessTarget], datasets: list[DatasetRunRow],
          bookmarks: dict[tuple[str, str], dict[str, str]], now: datetime) -> list[FreshnessResult]:
    """Every target's dataset freshness, then each time-valued bookmark's lag behind its newest sibling."""
    seen = last_success(datasets, now)
    out = []
    for t in targets:
        at = seen.get((t.source, t.dataset))
        age = _hours(now, at) if at else None
        out.append(FreshnessResult(t.source, t.dataset, "", "last_success" if at else "none", age,
                                   t.warn_hours, t.error_hours, state_of(age, t.warn_hours, t.error_hours),
                                   t.tier, now))
        times = {k: v for k, raw in sorted(bookmarks.get((t.source, t.dataset), {}).items())
                 if (v := _as_time(raw)) is not None}
        if len(times) < 2:  # noqa: PLR2004 - a lag needs a sibling to lag behind
            continue
        newest = max(times.values())
        for key, value in times.items():
            lag = _hours(newest, value)
            out.append(FreshnessResult(t.source, t.dataset, key, "bookmark_lag", lag, t.warn_hours,
                                       t.error_hours, state_of(lag, t.warn_hours, t.error_hours), t.tier, now))
    return out

Walk through it. The helper last_success finds, for each dataset, the newest run that succeeded, has an end time, and ended before now. A run still open does not count, and a failed run does not count. The age is the hours from that end time to now. Then the bookmarks: only values that parse as a time with a time zone are kept, so a shard position such as 1234 is ignored. With fewer than two such bookmarks there is nothing to lag behind, and the loop moves on. Otherwise each bookmark's lag is its distance from the newest one, and the same two targets decide its state.

src/dataplatform/obs/freshness.py:154The one moment at which a lateness check is worth running.
def next_due(targets: list[FreshnessTarget], datasets: list[DatasetRunRow], now: datetime) -> datetime | None:
    """When the next dataset would pass its error target if nothing loads before then: the one moment a lateness
    check is worth running (ADR-024 rule 4, no fixed interval). None when nothing can become late (never loaded
    datasets are already reported as `never`; a bookmark lag does not grow with the clock)."""
    seen = last_success(datasets, now)
    due = [at + timedelta(hours=t.error_hours) for t in targets if (at := seen.get((t.source, t.dataset)))]
    later = [d for d in due if d > now]
    return min(later) if later else None

Lateness has no event. Nothing happens when data fails to arrive. So instead of checking on a fixed interval, the monitor asks this function when the next dataset would pass its error target if nothing loads, and checks only then. A dataset that is already late is skipped, because checking it again changes nothing.

The success criteria and the scorecard

Freshness is one promise. The platform also keeps one list of twenty criteria that say what a good run means, each with a single metric and a target. After a run, a scorecard gives each criterion one of three states: met, not met, or not measured. A criterion the run cannot see is never counted as met.

src/dataplatform/obs/success.py:35The twenty criteria. Timeliness is C7 and C8.
CRITERIA = (
    Criterion("C1", "Correctness", "Days on which every layer equals the oracle", "100% of days, every layer"),
    Criterion("C2", "Correctness", "Certified metric answers equal to gold and the world", "100% of known answers"),
    Criterion("C3", "Correctness", "Live source agrees with silver, or the missing keys and hop are named", "100%"),
    Criterion("C4", "Correctness", "Gold fact runs whose row ledger balances", "100%"),
    Criterion("C5", "Fault handling", "Planted faults handled at their layer, none silent", "every planted fault"),
    Criterion("C6", "Fault handling", "False alarms on clean days", "0"),
    Criterion("C7", "Timeliness", "Days whose gold is complete by 06:00 business time", "95% of days"),
    Criterion("C8", "Timeliness", "Hop latency p95 the platform owns (bronze to silver)", "30 minutes"),
    Criterion("C9", "Performance", "One day's load, all layers, scale 0.1", "6 minutes or less"),
    Criterion("C10", "Cost", "Cost per daily load per data product", "within budget (after a 14-day baseline)"),
    Criterion("C11", "Reliability", "Rerunning a day changes nothing", "0 duplicate rows, 0 changed windows"),
    Criterion("C12", "Reliability", "Two runs writing one table at once", "never"),
    Criterion("C13", "Recovery", "A bad batch withdrawn from every layer, changed windows logged", "one run cycle"),
    Criterion("C14", "Explainability", "A past window rebuilt from its run log equals what it published", "100%"),
    Criterion("C15", "Privacy", "Restricted values in clear in bronze, silver or gold", "0"),
    Criterion("C16", "Privacy", "Right-to-be-forgotten requests completed", "within 30 days"),
    Criterion("C17", "Security", "Drift between the access model and catalog grants", "0"),
    Criterion("C18", "Security", "Secrets in code, logs or the run log", "0"),
    Criterion("C19", "Change safety", "Unexplained data differences that reach production", "0"),
    Criterion("C20", "Coverage", "DAC rules with a passing test or a written waiver", "89 of 89"),
)

Eight of the twenty are measured by a run from its own evidence: C1 to C5, C8, C9 and C15. The other twelve are reported as not measured, with the note that they are measured outside the run. The two timeliness numbers in the code are these. The hop from bronze to silver must take 30 minutes or less at the 95th percentile. One simulated day through all layers at one tenth of full volume must take 6 minutes or less.

src/dataplatform/obs/success.py:122The hop latency criterion. The hop from source commit to bronze is shown as context and not judged.
def _c8(e: Evidence) -> tuple[str, str, str]:
    seen = {h: s for h, s in e.hops.items() if h in HOP_TARGETS}
    if not seen:
        return "not_measured", "", "no hop stamps in this run"
    over = {h: s for h, s in seen.items() if s > HOP_TARGETS[h]}
    value = ", ".join(f"{h} {s / 60:.1f} min" for h, s in sorted(seen.items()))
    context = "; ".join(f"{h} {e.hops[h] / 60:.0f} min (context, not judged)" for h in CONTEXT_HOPS if h in e.hops)
    why = "worst dataset p95, from the world's clock (simulated schedule)" + (f"; {context}" if context else "")
    return ("not_met" if over else "met"), value, why

The scorecard rows travel with the operations tables as a sixteenth table, ops_success, so the run's score can be published beside its other tables.

How it is tested

tests/dataplatform/test_obs_freshness.py is table-driven. One table covers the states, one the tiers, one the full check, one the targets read from a view, and one the next due time. A further test loads the tyre manifest into the in-memory store and checks that nine datasets have targets with the values shown above. tests/dataplatform/test_obs_success.py has one case per measured criterion for met, not met and not seen.

tests/dataplatform/test_obs_freshness.py:48The cases for the full check: runs, bookmarks, and the results expected.
DS = [
    ("never loaded", [], {}, [("", "none", None, "never")]),
    ("fresh", [ran(0.5)], {}, [("", "last_success", 0.5, "ok")]),
    ("newest success counts, failures do not", [ran(5), ran(1.5), ran(0.1, "failed")], {},
     [("", "last_success", 1.5, "warn")]),
    ("a run still open does not count", [ran(3), ran(0.1, ended=False)], {}, [("", "last_success", 3.0, "error")]),
    ("a run after the check time does not count", [ran(-1), ran(3)], {}, [("", "last_success", 3.0, "error")]),
    ("one lagging bookmark is an error, its siblings ok",
     [ran(0.2)], {"D-1": "2026-10-31T23:59:59+05:30", "D-2": "2026-10-31T23:59:59+05:30",
                  "D-42": "2026-09-30T23:59:59+05:30"},
     [("", "last_success", 0.2, "ok"), ("D-1", "bookmark_lag", 0.0, "ok"), ("D-2", "bookmark_lag", 0.0, "ok"),
      ("D-42", "bookmark_lag", 744.0, "error")]),
    ("bookmarks that are not times are ignored", [ran(0.2)], {"shard-0": "1234", "shard-1": "1300"},
     [("", "last_success", 0.2, "ok")]),
    ("a single bookmark has nothing to lag behind", [ran(0.2)], {"D-1": "2026-10-01T00:00:00+00:00"},
     [("", "last_success", 0.2, "ok")]),
]
Not built yet, or known gaps
  • Criterion C7, the share of days whose gold is complete by 06:00 business time, is in the list but no code measures it.
  • The freshness target is measured from the end of the last successful load, not from the age of the newest row in gold. A freshness target measured end to end is an obligation in the catalog that no test cites.
  • In a local simulated run the platform loads once per simulated day, so datasets that promise hourly freshness show breaches between days. The operations page says so in its subtitle.
  • Criterion C20's target text says 89 of 89 rules. The catalog snapshot beside this code holds 105 obligations, as Proof by layer reports.
  • Hop latency from a simulated run uses the simulated clock, so it describes the schedule and not the speed of a real network.