Data PlatformZoom level 2
Alerts: what raises one, how severe it is, and who is told
An alert is a message to a person that something needs attention. The platform raises one when a dataset run fails, when data is older than its error target, when a detector fires, or when a quality or reconciliation check fails.
The hard part of alerting is not sending. It is sending few enough that people still read them. So the rules are configuration reviewed like any other, and four controls decide whether a failing condition becomes a new message.
Which conditions raise an alert
A signal is one subject's verdict at one check: failing or not. A subject is the thing the verdict is about. The table lists the five signal kinds, the subject of each, and the exact condition that makes it failing.
| Rule id | Signal | Subject | Failing when |
|---|---|---|---|
run-failed | run_failed | The dataset, such as erp.material | The dataset run ended with status failed. |
freshness-breach | freshness | The dataset, or the dataset and a bookmark key, such as partner.dealer_sellout/D-0042 | The freshness state is error. The warn state does not alert. |
detector-fired | detector | The dataset and the detector, such as erp.sales_order_line/DEL-01 | A detect. step has any status other than succeeded. Both a handled condition and a blocked load alert. |
check-failed | check_failed | The dataset and the step name | A quality. or recon. step has status failed. A warned check does not alert, and a profile never does. |
heartbeat-missing | heartbeat | Not defined in code | Nothing produces this signal yet. |
src/dataplatform/obs/rules/alerts.yaml:16The shipped rules, routes and limits. A domain may replace this file.rules:
- {id: run-failed, signal: run_failed, title: A dataset run failed}
- {id: freshness-breach, signal: freshness, title: Data is older than its error target}
- {id: detector-fired, signal: detector, title: A detector fired on a load}
- {id: check-failed, signal: check_failed, title: A quality or reconciliation check failed}
- {id: heartbeat-missing, signal: heartbeat, title: The heartbeat did not arrive}
routes:
1: {channel: page, to: owners.alerts}
2: {channel: ticket, to: owners.alerts}
3: {channel: digest, to: owners.dataset}
tiers: {}
snooze_max_days: 30
snoozes: []Severity, and who is told
Severity is the dataset's tier. The tier picks a channel, and the dataset's manifest says who receives on that channel. The tier comes from the dataset's freshness error target, as the freshness page explains, unless the configuration forces one.
| Tier | Error target | Channel | Goes to |
|---|---|---|---|
| 1 | 4 hours or less | page | The rota in owners.alerts |
| 2 | 48 hours or less | ticket | The rota in owners.alerts |
| 3 | Longer, or the dataset has no freshness target | digest | The owner in owners.dataset |
src/dataplatform/obs/alerts.py:211Turning a tier into a channel and a recipient.def route_for(config: AlertConfig, signal: Signal) -> Route:
route = config.routes[signal.tier]
if route.to.startswith("owners."):
who = signal.owners.get(route.to.split(".", 1)[1]) or signal.owners.get("dataset") or "unrouted"
return replace(route, to=who)
return routeWhen the route names the rota but the manifest has none, the dataset owner is used. When the manifest names nobody at all, the recipient is the word unrouted, so the gap is visible in the alert log and the message is not lost. A dataset with no freshness target has no owners attached to its signals, so its run failures are tier 3 and unrouted.
The four controls that keep alerts few
| Control | Rule | Example with the tyre company |
|---|---|---|
| Change of state | A subject alerts when it starts failing and resolves when it recovers. Still failing at the next check sends nothing. | Material copies stay held for six hours. One alert, not one per check. |
| One per rule, route and business day | A second subject failing the same day under the same rule and route joins the day's alert. | Two dealers' files are late on the same day. One message lists both. |
| Routed by tier | The channel and recipient come from the tier and the manifest. | Order lines page the rota. Dealer sell-out goes into the owner's digest. |
| Snoozes expire | A snooze needs a rule, a subject pattern, a reason of at least 10 characters, an owner, and an end time with a time zone no more than 30 days ahead. | The lab system is down for a planned upgrade until Friday. After Friday a subject still failing alerts again. |
A business day is counted in the domain's business time zone, which the shipped rules set to Asia/Kolkata. A resolution message is sent once per recovery, and only for a subject that was actually alerted.
A worked example
The freshness rule is checked every two hours for dealer D-0042's file and dealer D-0107's file, both tier 3. The first column is the hour, the second is which subjects are failing, and the third is what is sent.
| Hour | Failing | Sent |
|---|---|---|
| 0 | D-0042 | One alert for freshness-breach, digest, to the dataset owner |
| 2 | D-0042 and D-0107 | Nothing. D-0107 joins the day's alert and is recorded as grouped. |
| 4 | D-0042 and D-0107 | Nothing. Both are still failing. |
| 6 | D-0107 | One resolution for D-0042 |
| 24 | D-0107 | Nothing. It never stopped failing, so there is no change of state. |
The code
src/dataplatform/obs/alerts.py:230One evaluation: the signals of this check against the state kept from the last one.def evaluate(config: AlertConfig, signals: list[Signal], state: dict[str, Any], now: datetime,
scope: Scope | None = None) -> Evaluation:
"""One check. Subjects a rule knew as failing that are absent from this check's signals have recovered (within
the scope: by default every signal and every subject)."""
scope = scope or Scope()
subjects: dict[str, Any] = {k: dict(v) for k, v in (state.get("subjects") or {}).items()}
day = business_day(now, config.business_timezone)
daily: dict[str, Any] = {k: v for k, v in (state.get("daily") or {}).items() if k.split("|")[1] == day}
by_rule: dict[str, dict[str, Signal]] = {r.id: {} for r in config.rules if r.enabled and r.signal in scope.signals}
signal_of = {r.id: r.signal for r in config.rules}
for sig in signals:
for r in config.rules:
if r.id in by_rule and r.signal == sig.signal:
by_rule[r.id][sig.subject] = sig
sent: list[Alert] = []
events: list[Event] = []
for rule_id, current in by_rule.items():
failing = {s: sig for s, sig in current.items() if sig.failing}
for _, sig in sorted(failing.items()):
_fail(config, rule_id, sig, subjects, daily, day, now, sent, events)
for key in sorted(k for k in subjects if k.startswith(f"{rule_id}|")):
subject = key.split("|", 1)[1]
if subject not in failing and scope.covers(signal_of[rule_id], subject):
_recover(rule_id, subject, subjects.pop(key), now, sent, events, day)
return Evaluation({"subjects": subjects, "daily": daily}, sent, events)Walk through it. The state has two parts. subjects holds every subject known to be failing, keyed by rule and subject. daily holds the messages already sent today, and entries from earlier business days are dropped on the first line that builds it. The signals are sorted into rules. For each rule, every failing subject goes to _fail. Then every subject the state knew as failing, which is not failing now and which the scope covers, goes to _recover.
The scope is what makes pushed evaluation safe. A finished run speaks only for the datasets it ran. Without a scope, a run of the order lines would find no signal for the material dataset and wrongly resolve its alert.
src/dataplatform/obs/alerts.py:257What happens to one failing subject. The four controls are four early returns.def _fail(config: AlertConfig, rule_id: str, sig: Signal, subjects: dict[str, Any], daily: dict[str, Any],
day: str, now: datetime, sent: list[Alert], events: list[Event]) -> None:
key = _key(rule_id, sig.subject)
entry = subjects.setdefault(key, {"since": now.isoformat(), "notified": False})
if entry["notified"]:
return # still failing: change of state only
route = route_for(config, sig)
entry.update(channel=route.channel, to=route.to)
if snoozed(config, rule_id, sig.subject, now):
events.append(Event(now, rule_id, sig.subject, "snoozed", route.channel, route.to, sig.run_id))
return
entry["notified"] = True
group = f"{rule_id}|{day}|{route.channel}|{route.to}"
item = {"subject": sig.subject, "run_id": sig.run_id, "detail": sig.detail}
if group in daily:
daily[group]["subjects"].append(item)
events.append(Event(now, rule_id, sig.subject, "grouped", route.channel, route.to, sig.run_id))
return
daily[group] = {"first_at": now.isoformat(), "subjects": [item]}
sent.append(Alert(rule_id, day, route.channel, route.to, now, [item]))
events.append(Event(now, rule_id, sig.subject, "fired", route.channel, route.to, sig.run_id))- Already notified. The subject was alerted and is still failing. Return. This is change of state.
- Find the route. The tier gives the channel and the recipient, and both are stored on the subject.
- Snoozed. A snooze that has not expired matches the rule and subject. Record a snoozed event and return. The subject is not marked notified, so it alerts once the snooze ends.
- Join today's alert. A message for this rule, day, channel and recipient already went out. Add the subject to it, record a grouped event, and return.
- Send. Otherwise create the day's alert, add it to the list to send, and record a fired event.
src/dataplatform/obs/ops.py:333How a finished run's records become signals.def run_signals(run: RunLogSnapshot, targets: list[FreshnessTarget]) -> list[Signal]:
"""A finished run's signals (pushed as it ends): every dataset it ran, failing or not, so recoveries resolve.
Detectors and checks come only from the step rows (the one step format); details stay with the owner, and the
alert carries the owner's screen."""
make = _maker(targets)
out = []
for d in run.datasets:
name = f"{d.source}.{d.dataset}"
out.append(make("run_failed", name, name, d.status == "failed", d.run_id, d.message))
for s in run.steps:
kind, name = kind_of(s.step), f"{s.source}.{s.dataset}"
see = f" (see {SCREEN[owner_of(s.step)]})" if kind else ""
if kind == DETECT:
out.append(make("detector", f"{name}/{s.step.split('.', 1)[1]}", name, s.status != "succeeded",
s.run_id, s.message + see))
elif kind in ("quality", "recon"):
out.append(make("check_failed", f"{name}/{s.step}", name, s.status == "failed", s.run_id,
s.message + see))
return outEvery dataset the run touched produces a run_failed signal, failing or not. The not-failing ones matter: they are how a dataset that failed yesterday resolves today. Detector and check signals come only from step rows in the one format, and each message carries the name of the screen where the owner keeps the detail.
src/dataplatform/obs/monitor.py:96The monitor's entry point, called as each run ends. def run_finished(self, run_id: str, now: datetime) -> CheckOutput:
"""Pushed by the recorder as a run ends: the run's own datasets, detectors and checks, then lateness (a run
that loads can end a lateness)."""
log = self.reader.snapshot(self.domain, self.since, now)
run = _only(log, run_id)
targets = fresh.targets_from_store(self.store, self.domain, self.config.tiers)
names = frozenset(f"{d.source}.{d.dataset}" for d in run.datasets)
out = self._evaluate(ops.run_signals(run, targets), now, Scope(RUN_SIGNALS, names))
late = self._lateness(log, now)
return CheckOutput(out.sent + late.sent, out.events + late.events, late.freshness)The monitor reads the run log, keeps only this run's records, and evaluates them inside a scope naming this run's datasets and the three run signal kinds. It then runs a lateness check, because a run that loaded data can end a lateness. The lateness check also stores the next due time, and wake does nothing until that time has come.
Delivery
Sending is behind a port with one method, send. Two adapters exist. The in-memory one keeps the alerts in a list and is what every run and test on this site uses. The other, in src/dataplatform/adapters/databricks_notify.py, maps each channel to one of the workspace's notification destinations and posts one payload shape. A channel with no destination bound is refused with an error, never dropped silently.
How it is tested
tests/dataplatform/test_obs_alerts.py drives the evaluator with timelines. Each case is a list of checks, each an hour and the set of failing subjects, and the expected result is the list of messages sent with their hour, rule, subjects, channel and recipient.
tests/dataplatform/test_obs_alerts.py:53The noise controls as timelines. The first two rows are the core promise.TIMELINES = [
("OPS-06: failing for a day fires once", cfg(), [(h, {"a"}) for h in range(0, 24, 2)],
[(0, "fresh", ("a",), "page", "erp-rota")]),
("OPS-06: recovery resolves once", cfg(), [(0, {"a"}), (2, {"a"}), (4, set()), (6, set())],
[(0, "fresh", ("a",), "page", "erp-rota"), (4, "fresh:resolved", ("a",), "page", "erp-rota")]),
("a second subject the same day joins the day's alert", cfg(), [(0, {"a"}), (2, {"a", "b"})],
[(0, "fresh", ("a",), "page", "erp-rota")]),
("flapping the same day sends no second alert", cfg(), [(0, {"a"}), (1, set()), (2, {"a"})],
[(0, "fresh", ("a",), "page", "erp-rota"), (1, "fresh:resolved", ("a",), "page", "erp-rota")]),
("failing again on a new business day alerts again", cfg(), [(0, {"a"}), (1, set()), (24, {"a"})],
[(0, "fresh", ("a",), "page", "erp-rota"), (1, "fresh:resolved", ("a",), "page", "erp-rota"),
(24, "fresh", ("a",), "page", "erp-rota")]),
("business day, not UTC day: 23:00 and 01:00 IST are two days", cfg(),
[(17, {"a"}), (18, set()), (19, {"a"})], # 23:30 IST day 1, then 01:30 IST day 2
[(17, "fresh", ("a",), "page", "erp-rota"), (18, "fresh:resolved", ("a",), "page", "erp-rota"),
(19, "fresh", ("a",), "page", "erp-rota")]),
("OPS-06: a snooze holds the alert", cfg(snoozes=[SNOOZE]), [(0, {"a"}), (4, {"a"})], []),
("OPS-06: an expired snooze lets a still-failing alert return", cfg(snoozes=[SNOOZE]),
[(0, {"a"}), (4, {"a"}), (11, {"a"}), (12, {"a"})], [(11, "fresh", ("a",), "page", "erp-rota")]),
("a snoozed subject that recovers sends no resolution", cfg(snoozes=[SNOOZE]), [(0, {"a"}), (4, set())], []),
("a snooze is per subject: others still alert", cfg(snoozes=[SNOOZE]), [(0, {"a", "b"})],
[(0, "fresh", ("b",), "page", "erp-rota")]),
("a disabled rule sends nothing", cfg(rules=[{"id": "fresh", "signal": "freshness", "enabled": False}]),
[(0, {"a"})], []),
]The same file holds eleven configuration refusals, the routing table by tier, and the scope cases. tests/dataplatform/test_obs_notify.py is the contract table for the notifier port, run against both adapters.
- No code produces the heartbeat signal. The rule is configuration only.
- Alert state lives in memory on the monitor. Restarting the process forgets which subjects were already notified.
- Sending through a real workspace destination is not done. The transport is injected in tests.
- The platform's own job failure notification, meant to be set on every deployed job, is written as a snippet but not merged into the bundle generator.
- A dataset with no freshness target has no owners on its signals, so its alerts go to
unrouted. Nothing warns about that at load time.