Home › Key concepts › Freshness and SLAs

Key concept · Walk (intermediate) · the same on every platform

Freshness and SLAs

Promise how up to date data will be, then measure it.

What it is

Freshness is the time since the data was last updated. An SLA (service level agreement) is the promise, such as 'sales are less than 2 hours old by 9 a.m.'. You measure it and alert before users notice.

Why it matters

Users trust data that is on time. A late table is often worse than a wrong one, because nobody knows.

An example

An alert fires at 08:30 because the orders table is 5 hours old, before the 9 a.m. meeting.

The common mistake

Promising 'real time' without saying what that means or measuring it.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Orchestrationrun the steps in order, on time, and retry the failuresMWAA (Airflow) / Step FunctionsData Factory / Fabric pipelinesCloud Composer (Airflow)Lakeflow JobsAirflow or Dagster
Monitoringlogs, metrics and traces, so you see trouble firstCloudWatch + X-RayAzure Monitor + App InsightsCloud Logging + TraceSystem tables + MLflow tracingOpenTelemetry + Grafana

Related ideas

Where it shows up

Part of the Walk level on these paths: Data Engineering (running data).