cleandataengineering.aiData Platform, from the big picture to the code
You are here: Start here › Operations › Alerts

Data PlatformZoom level 2

Alerts: what raises one, how severe it is, and who is told

An alert is a message to a person that something needs attention. The platform raises one when a dataset run fails, when data is older than its error target, when a detector fires, or when a quality or reconciliation check fails.

The hard part of alerting is not sending. It is sending few enough that people still read them. So the rules are configuration reviewed like any other, and four controls decide whether a failing condition becomes a new message.

A run finishespushed by the storeLateness is duethe monitor wakesSignalsone per subjectEvaluatestate and noise rulesTier 1: pagethe rotaTier 2: ticketthe rotaTier 3: digestthe dataset owner
Two things start an evaluation, on the left. The Evaluate box decides what is sent. The tier picks the channel.

Which conditions raise an alert

A signal is one subject's verdict at one check: failing or not. A subject is the thing the verdict is about. The table lists the five signal kinds, the subject of each, and the exact condition that makes it failing.

Rule idSignalSubjectFailing when
run-failedrun_failedThe dataset, such as erp.materialThe dataset run ended with status failed.
freshness-breachfreshnessThe dataset, or the dataset and a bookmark key, such as partner.dealer_sellout/D-0042The freshness state is error. The warn state does not alert.
detector-fireddetectorThe dataset and the detector, such as erp.sales_order_line/DEL-01A detect. step has any status other than succeeded. Both a handled condition and a blocked load alert.
check-failedcheck_failedThe dataset and the step nameA quality. or recon. step has status failed. A warned check does not alert, and a profile never does.
heartbeat-missingheartbeatNot defined in codeNothing produces this signal yet.
src/dataplatform/obs/rules/alerts.yaml:16The shipped rules, routes and limits. A domain may replace this file.
rules:
  - {id: run-failed, signal: run_failed, title: A dataset run failed}
  - {id: freshness-breach, signal: freshness, title: Data is older than its error target}
  - {id: detector-fired, signal: detector, title: A detector fired on a load}
  - {id: check-failed, signal: check_failed, title: A quality or reconciliation check failed}
  - {id: heartbeat-missing, signal: heartbeat, title: The heartbeat did not arrive}
routes:
  1: {channel: page, to: owners.alerts}
  2: {channel: ticket, to: owners.alerts}
  3: {channel: digest, to: owners.dataset}
tiers: {}
snooze_max_days: 30
snoozes: []

Severity, and who is told

Severity is the dataset's tier. The tier picks a channel, and the dataset's manifest says who receives on that channel. The tier comes from the dataset's freshness error target, as the freshness page explains, unless the configuration forces one.

TierError targetChannelGoes to
14 hours or lesspageThe rota in owners.alerts
248 hours or lessticketThe rota in owners.alerts
3Longer, or the dataset has no freshness targetdigestThe owner in owners.dataset
src/dataplatform/obs/alerts.py:211Turning a tier into a channel and a recipient.
def route_for(config: AlertConfig, signal: Signal) -> Route:
    route = config.routes[signal.tier]
    if route.to.startswith("owners."):
        who = signal.owners.get(route.to.split(".", 1)[1]) or signal.owners.get("dataset") or "unrouted"
        return replace(route, to=who)
    return route

When the route names the rota but the manifest has none, the dataset owner is used. When the manifest names nobody at all, the recipient is the word unrouted, so the gap is visible in the alert log and the message is not lost. A dataset with no freshness target has no owners attached to its signals, so its run failures are tier 3 and unrouted.

The four controls that keep alerts few

ControlRuleExample with the tyre company
Change of stateA subject alerts when it starts failing and resolves when it recovers. Still failing at the next check sends nothing.Material copies stay held for six hours. One alert, not one per check.
One per rule, route and business dayA second subject failing the same day under the same rule and route joins the day's alert.Two dealers' files are late on the same day. One message lists both.
Routed by tierThe channel and recipient come from the tier and the manifest.Order lines page the rota. Dealer sell-out goes into the owner's digest.
Snoozes expireA snooze needs a rule, a subject pattern, a reason of at least 10 characters, an owner, and an end time with a time zone no more than 30 days ahead.The lab system is down for a planned upgrade until Friday. After Friday a subject still failing alerts again.

A business day is counted in the domain's business time zone, which the shipped rules set to Asia/Kolkata. A resolution message is sent once per recovery, and only for a subject that was actually alerted.

A worked example

The freshness rule is checked every two hours for dealer D-0042's file and dealer D-0107's file, both tier 3. The first column is the hour, the second is which subjects are failing, and the third is what is sent.

HourFailingSent
0D-0042One alert for freshness-breach, digest, to the dataset owner
2D-0042 and D-0107Nothing. D-0107 joins the day's alert and is recorded as grouped.
4D-0042 and D-0107Nothing. Both are still failing.
6D-0107One resolution for D-0042
24D-0107Nothing. It never stopped failing, so there is no change of state.

The code

src/dataplatform/obs/alerts.py:230One evaluation: the signals of this check against the state kept from the last one.
def evaluate(config: AlertConfig, signals: list[Signal], state: dict[str, Any], now: datetime,
             scope: Scope | None = None) -> Evaluation:
    """One check. Subjects a rule knew as failing that are absent from this check's signals have recovered (within
    the scope: by default every signal and every subject)."""
    scope = scope or Scope()
    subjects: dict[str, Any] = {k: dict(v) for k, v in (state.get("subjects") or {}).items()}
    day = business_day(now, config.business_timezone)
    daily: dict[str, Any] = {k: v for k, v in (state.get("daily") or {}).items() if k.split("|")[1] == day}
    by_rule: dict[str, dict[str, Signal]] = {r.id: {} for r in config.rules if r.enabled and r.signal in scope.signals}
    signal_of = {r.id: r.signal for r in config.rules}
    for sig in signals:
        for r in config.rules:
            if r.id in by_rule and r.signal == sig.signal:
                by_rule[r.id][sig.subject] = sig
    sent: list[Alert] = []
    events: list[Event] = []
    for rule_id, current in by_rule.items():
        failing = {s: sig for s, sig in current.items() if sig.failing}
        for _, sig in sorted(failing.items()):
            _fail(config, rule_id, sig, subjects, daily, day, now, sent, events)
        for key in sorted(k for k in subjects if k.startswith(f"{rule_id}|")):
            subject = key.split("|", 1)[1]
            if subject not in failing and scope.covers(signal_of[rule_id], subject):
                _recover(rule_id, subject, subjects.pop(key), now, sent, events, day)
    return Evaluation({"subjects": subjects, "daily": daily}, sent, events)

Walk through it. The state has two parts. subjects holds every subject known to be failing, keyed by rule and subject. daily holds the messages already sent today, and entries from earlier business days are dropped on the first line that builds it. The signals are sorted into rules. For each rule, every failing subject goes to _fail. Then every subject the state knew as failing, which is not failing now and which the scope covers, goes to _recover.

The scope is what makes pushed evaluation safe. A finished run speaks only for the datasets it ran. Without a scope, a run of the order lines would find no signal for the material dataset and wrongly resolve its alert.

src/dataplatform/obs/alerts.py:257What happens to one failing subject. The four controls are four early returns.
def _fail(config: AlertConfig, rule_id: str, sig: Signal, subjects: dict[str, Any], daily: dict[str, Any],
          day: str, now: datetime, sent: list[Alert], events: list[Event]) -> None:
    key = _key(rule_id, sig.subject)
    entry = subjects.setdefault(key, {"since": now.isoformat(), "notified": False})
    if entry["notified"]:
        return  # still failing: change of state only
    route = route_for(config, sig)
    entry.update(channel=route.channel, to=route.to)
    if snoozed(config, rule_id, sig.subject, now):
        events.append(Event(now, rule_id, sig.subject, "snoozed", route.channel, route.to, sig.run_id))
        return
    entry["notified"] = True
    group = f"{rule_id}|{day}|{route.channel}|{route.to}"
    item = {"subject": sig.subject, "run_id": sig.run_id, "detail": sig.detail}
    if group in daily:
        daily[group]["subjects"].append(item)
        events.append(Event(now, rule_id, sig.subject, "grouped", route.channel, route.to, sig.run_id))
        return
    daily[group] = {"first_at": now.isoformat(), "subjects": [item]}
    sent.append(Alert(rule_id, day, route.channel, route.to, now, [item]))
    events.append(Event(now, rule_id, sig.subject, "fired", route.channel, route.to, sig.run_id))
  1. Already notified. The subject was alerted and is still failing. Return. This is change of state.
  2. Find the route. The tier gives the channel and the recipient, and both are stored on the subject.
  3. Snoozed. A snooze that has not expired matches the rule and subject. Record a snoozed event and return. The subject is not marked notified, so it alerts once the snooze ends.
  4. Join today's alert. A message for this rule, day, channel and recipient already went out. Add the subject to it, record a grouped event, and return.
  5. Send. Otherwise create the day's alert, add it to the list to send, and record a fired event.
src/dataplatform/obs/ops.py:333How a finished run's records become signals.
def run_signals(run: RunLogSnapshot, targets: list[FreshnessTarget]) -> list[Signal]:
    """A finished run's signals (pushed as it ends): every dataset it ran, failing or not, so recoveries resolve.
    Detectors and checks come only from the step rows (the one step format); details stay with the owner, and the
    alert carries the owner's screen."""
    make = _maker(targets)
    out = []
    for d in run.datasets:
        name = f"{d.source}.{d.dataset}"
        out.append(make("run_failed", name, name, d.status == "failed", d.run_id, d.message))
    for s in run.steps:
        kind, name = kind_of(s.step), f"{s.source}.{s.dataset}"
        see = f" (see {SCREEN[owner_of(s.step)]})" if kind else ""
        if kind == DETECT:
            out.append(make("detector", f"{name}/{s.step.split('.', 1)[1]}", name, s.status != "succeeded",
                            s.run_id, s.message + see))
        elif kind in ("quality", "recon"):
            out.append(make("check_failed", f"{name}/{s.step}", name, s.status == "failed", s.run_id,
                            s.message + see))
    return out

Every dataset the run touched produces a run_failed signal, failing or not. The not-failing ones matter: they are how a dataset that failed yesterday resolves today. Detector and check signals come only from step rows in the one format, and each message carries the name of the screen where the owner keeps the detail.

src/dataplatform/obs/monitor.py:96The monitor's entry point, called as each run ends.
    def run_finished(self, run_id: str, now: datetime) -> CheckOutput:
        """Pushed by the recorder as a run ends: the run's own datasets, detectors and checks, then lateness (a run
        that loads can end a lateness)."""
        log = self.reader.snapshot(self.domain, self.since, now)
        run = _only(log, run_id)
        targets = fresh.targets_from_store(self.store, self.domain, self.config.tiers)
        names = frozenset(f"{d.source}.{d.dataset}" for d in run.datasets)
        out = self._evaluate(ops.run_signals(run, targets), now, Scope(RUN_SIGNALS, names))
        late = self._lateness(log, now)
        return CheckOutput(out.sent + late.sent, out.events + late.events, late.freshness)

The monitor reads the run log, keeps only this run's records, and evaluates them inside a scope naming this run's datasets and the three run signal kinds. It then runs a lateness check, because a run that loaded data can end a lateness. The lateness check also stores the next due time, and wake does nothing until that time has come.

Delivery

Sending is behind a port with one method, send. Two adapters exist. The in-memory one keeps the alerts in a list and is what every run and test on this site uses. The other, in src/dataplatform/adapters/databricks_notify.py, maps each channel to one of the workspace's notification destinations and posts one payload shape. A channel with no destination bound is refused with an error, never dropped silently.

How it is tested

tests/dataplatform/test_obs_alerts.py drives the evaluator with timelines. Each case is a list of checks, each an hour and the set of failing subjects, and the expected result is the list of messages sent with their hour, rule, subjects, channel and recipient.

tests/dataplatform/test_obs_alerts.py:53The noise controls as timelines. The first two rows are the core promise.
TIMELINES = [
    ("OPS-06: failing for a day fires once", cfg(), [(h, {"a"}) for h in range(0, 24, 2)],
     [(0, "fresh", ("a",), "page", "erp-rota")]),
    ("OPS-06: recovery resolves once", cfg(), [(0, {"a"}), (2, {"a"}), (4, set()), (6, set())],
     [(0, "fresh", ("a",), "page", "erp-rota"), (4, "fresh:resolved", ("a",), "page", "erp-rota")]),
    ("a second subject the same day joins the day's alert", cfg(), [(0, {"a"}), (2, {"a", "b"})],
     [(0, "fresh", ("a",), "page", "erp-rota")]),
    ("flapping the same day sends no second alert", cfg(), [(0, {"a"}), (1, set()), (2, {"a"})],
     [(0, "fresh", ("a",), "page", "erp-rota"), (1, "fresh:resolved", ("a",), "page", "erp-rota")]),
    ("failing again on a new business day alerts again", cfg(), [(0, {"a"}), (1, set()), (24, {"a"})],
     [(0, "fresh", ("a",), "page", "erp-rota"), (1, "fresh:resolved", ("a",), "page", "erp-rota"),
      (24, "fresh", ("a",), "page", "erp-rota")]),
    ("business day, not UTC day: 23:00 and 01:00 IST are two days", cfg(),
     [(17, {"a"}), (18, set()), (19, {"a"})],  # 23:30 IST day 1, then 01:30 IST day 2
     [(17, "fresh", ("a",), "page", "erp-rota"), (18, "fresh:resolved", ("a",), "page", "erp-rota"),
      (19, "fresh", ("a",), "page", "erp-rota")]),
    ("OPS-06: a snooze holds the alert", cfg(snoozes=[SNOOZE]), [(0, {"a"}), (4, {"a"})], []),
    ("OPS-06: an expired snooze lets a still-failing alert return", cfg(snoozes=[SNOOZE]),
     [(0, {"a"}), (4, {"a"}), (11, {"a"}), (12, {"a"})], [(11, "fresh", ("a",), "page", "erp-rota")]),
    ("a snoozed subject that recovers sends no resolution", cfg(snoozes=[SNOOZE]), [(0, {"a"}), (4, set())], []),
    ("a snooze is per subject: others still alert", cfg(snoozes=[SNOOZE]), [(0, {"a", "b"})],
     [(0, "fresh", ("b",), "page", "erp-rota")]),
    ("a disabled rule sends nothing", cfg(rules=[{"id": "fresh", "signal": "freshness", "enabled": False}]),
     [(0, {"a"})], []),
]

The same file holds eleven configuration refusals, the routing table by tier, and the scope cases. tests/dataplatform/test_obs_notify.py is the contract table for the notifier port, run against both adapters.

Not built yet, or known gaps
  • No code produces the heartbeat signal. The rule is configuration only.
  • Alert state lives in memory on the monitor. Restarting the process forgets which subjects were already notified.
  • Sending through a real workspace destination is not done. The transport is injected in tests.
  • The platform's own job failure notification, meant to be set on every deployed job, is written as a snippet but not merged into the bundle generator.
  • A dataset with no freshness target has no owners on its signals, so its alerts go to unrouted. Nothing warns about that at load time.