Monitoring and alerting
System tables, REST API and CLI, pipeline event logs, Lakehouse alerts and Jobs monitoring.
This section tests whether you pick the right place to look: system tables for account-wide history, the pipeline event log for one pipeline's health, the Jobs UI and API for live run status, and alerts for being told when something crosses a line. The biggest way people lose marks is choosing a tool that can see the data but is aimed at the wrong scope or the wrong moment, such as a per-run screen when the question is about trends across the whole account.
Walk the guide, line by line
Each row is one line of the official guide, in its own words on the left. Read across: plain words, what a question looks like, the answer, and the trap.
| Guide line | In plain words | A question looks like | The answer | The trap |
|---|---|---|---|---|
| 4.1 Use Databricks system tables (billing, compute, access, lakeflow) for cost analysis, auditing, and workload monitoring. | System tables are read-only tables in the system catalog that Databricks fills for you. The billing schema holds usage and list prices, compute holds cluster and warehouse history, access holds audit logs and lineage, and lakeflow holds job and pipeline run history. You query them with ordinary SQL. | A finance lead wants last month's spend broken down by job across every workspace in the account, or a security officer wants to know who changed a permission. The deciding fact is that the question spans the whole account over weeks or months. | Query system.billing.usage and join it to system.billing.list_prices for cost, system.access.audit for who did what, system.compute for cluster and warehouse history, and system.lakeflow for job and pipeline runs. The free retention is 395 days by default, which is why system tables suit trend questions. Your user needs USE CATALOG on system plus USE SCHEMA and SELECT on the schema. Try it: On Free Edition, open the catalog explorer and see which system schemas your workspace shows. If a schema is missing, note that an admin has to grant access. | Opening each job's page in the Jobs UI works for one job but cannot add up cost across workspaces or look back months. The UI keeps run history for 60 days, while the lakeflow system tables keep 395 by default. Runner-up wins when: The Jobs UI or API wins when the question is about one run that is happening right now, because system tables are not a live view of a single run. |
| 4.2 Use Databricks REST APIs / CLI and SDK for monitoring and analyzing jobs and pipelines. | Everything you can click in the Jobs and pipelines screens is also available as a REST API, as Databricks CLI commands, and as the Databricks SDK for Python and other languages. Use these when you want a script, a scheduled report or another tool to read run information. | An operations team wants a small script that runs every hour, reads the current state of a handful of jobs and pipelines, and posts a summary to a chat tool. The deciding fact is that a program, not a person, is reading the status. | Call the Jobs API for runs and the Pipelines API for updates, through the SDK or the CLI. The SDK handles authentication and paging for you, and the CLI is convenient in shell scripts and CI steps. Use a service principal as the identity so the script does not depend on one person's account. Try it: Install the Databricks CLI, run databricks jobs list against your Free Edition workspace, and read the same list through the Python SDK. | Screen-scraping the UI or asking someone to export results by hand does not scale and breaks when the page changes. The tempting alternative is system tables, which are good for history but are not the control-plane API for current job and pipeline state. Runner-up wins when: System tables win when the script needs months of history or cost, because the API returns recent runs for individual jobs and does not give you billing. |
| 4.3 Use Pipeline event logs to monitor pipeline health and data quality. | Every Lakeflow Spark Declarative Pipelines pipeline writes an event log, which is a Delta table of structured events: updates, flow progress, row counts and data quality results. You read it with SQL to see whether a pipeline is healthy and how many records passed or failed each expectation. | A pipeline has expectations on three tables and the team wants to know how many rows failed each expectation in the latest update. The deciding fact is that the numbers live inside the pipeline's own events. | Query the event log with event_log and the pipeline ID, or the published table if you set one in the pipeline settings. Data quality results are in flow_progress events, inside the details JSON under data_quality, with passed and failed record counts per expectation and a separate dropped record count. Create a view over the log before granting access to others. Try it: Run a small pipeline with one expectation, then run SELECT * FROM event_log with its pipeline ID and filter event_type to flow_progress. | Looking in the driver log or Spark UI shows errors but not per-expectation counts. The cluster log is for crashes, while the event log is the structured record of pipeline behaviour. Runner-up wins when: The pipeline_events system table (Beta) wins when you need to compare many pipelines across a region in one query instead of one pipeline. |
| 4.4 Use Databricks Lakehouse alerts to monitor governed business metrics, data quality and anomaly signals, usage and cost, SQL warehouse/query health, audit and security events, AI agent quality, and Lakeflow Job branching. | Databricks alerts run a SQL query on a schedule and notify people or a webhook when a condition on the result is met. The statuses are OK, TRIGGERED and ERROR. An alert can also run as a task inside a Lakeflow Job so later tasks branch on the result. The guide's name Lakehouse alerts is used here; the docs simply call them alerts. | A retailer wants the on-call analyst told whenever the daily order count falls below a threshold, or wants a job to skip an expensive step when a data check fails. The deciding fact is that the signal is a number you can compute with a query. | Write the query that returns the metric, set the alert condition and schedule, and add a notification destination. Use the alert as a job task when downstream tasks must branch on it. The same pattern covers data quality, anomalies, cost and usage queries over system tables, and audit events. Try it: Create an alert on a query that counts rows in a small table, set it to trigger above a tiny threshold, and watch the status change. | Job failure notifications only fire when a run fails. A job can finish successfully while the data is wrong, so a failure email never tells you the order count dropped. Runner-up wins when: Job notifications win when the thing to catch is a run failing or running past its expected time, because that signal is not a query result. |
| 4.5 Use Lakeflow Jobs UI and Jobs API to monitor job status and performance metrics. | The Jobs UI shows runs in a matrix, graph, timeline and list, with status, duration, error code and per-task details. The Jobs API gives the same run information to programs. Both are meant for looking at jobs while they run or soon after. | A run finished slower than usual this morning and the engineer wants to see which task took the extra time. The deciding fact is that it is one recent run of one job. | Open the run in the Jobs UI and use the timeline view to compare task durations, then open the slow task for its logs and metrics. Set an expected completion time on the job so slow runs are flagged. The run outcome is decided from the leaf tasks, so a run can show succeeded with failures. Try it: Run a two-task job twice and compare the two runs in the matrix and timeline views. | Starting with system tables works but is slower for a single fresh run, and it reports timing rows rather than the task logs you need. Another trap is reading the job status alone, since a successful run can still be slow. Runner-up wins when: System tables win when you need to compare this job's task durations against the last six months, because run history in the UI is kept for 60 days. |
Mind map: Which monitoring surface do I open
People assume all monitoring tools see the same data, but each one is aimed at a different scope and time span.
| What you observe | What it points to | What to do (and when the runner-up wins) |
|---|---|---|
| Question spans many workspaces or months (cost, audit, run history) | System tables in billing, access, compute or lakeflow | Write SQL; the runner-up wins if the question is only about one run happening now, where the Jobs UI is faster |
| Question is about one pipeline's row counts or expectation results | That pipeline's event log | Query event_log; if you must compare many pipelines at once, the pipeline_events system table (Beta) wins |
| Question is about one recent run of one job | Jobs UI run page or Jobs API | Use timeline and task logs; if you need more than 60 days of history, switch to lakeflow system tables |
| A program must read status on a schedule | REST API, CLI or SDK | Use a service principal; if the script needs billing or months of history, query system tables instead |
Mind map: How should I be told about a problem
People assume a job failure notification covers every problem, but a successful run can still produce bad or missing data.
| What you observe | What it points to | What to do (and when the runner-up wins) |
|---|---|---|
| The run itself fails or runs too long | Job notifications and expected completion time | Configure them on the job |
| Run succeeds but a business number is wrong | A query-based alert | Alert on the metric; the runner-up job notification would stay silent |
| Later tasks must skip when a check fails | Alert as a job task | Branch on its result |
| Rows fail expectations inside a pipeline | Event log query behind an alert | Alert on the failed record count from the event log |