Home › Learn › Databricks compute and cost › Logs and visibility
Databricks compute and cost · data engineering
Why serverless feels opaque: what Databricks logs, how to extract it programmatically and the cheapest way to read it
What you lose and what you keep on serverless, every source of run, query, billing, audit and lineage information with retention, how to extract it by SQL and REST API, and the cheapest way to do it.
Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.
Why
Why serverless feels opaque
Classic compute runs on machines in your cloud account, so you can look inside them. Serverless runs in Databricks' account, so you get no machines to inspect. Databricks says Spark logs are not available for serverless notebooks and jobs (you only get client-side application logs), and the Spark UI is not available (the query profile replaces it). There is no host access and no way to install a monitoring agent.
| What you want to see | Classic compute | Serverless |
|---|---|---|
| Spark UI (stages, tasks, skew) | Yes | No. Use the query profile and query insights. |
| Driver and executor logs | Yes | No Spark logs. Client-side application logs only. |
| Machine CPU and memory over time | Yes, including system.compute.node_timeline for all-purpose and jobs compute | No machines to measure |
| Cluster start, resize and stop events | Yes | Not applicable |
| Cost split by machine tag | Yes, tags reach the cloud bill | No. Use budget policies and job or warehouse ids. |
| Cost per run, in DBUs | Yes | Yes. job_run_id is filled in for serverless jobs. |
| Per-query detail | Yes | Yes. Query history covers SQL warehouses and serverless notebooks and jobs. |
So serverless is not blind. You trade machine-level detail for query-level and run-level detail.
What exists
What information Databricks keeps, by source
| Source | What it holds | Retention | Covers serverless? |
|---|---|---|---|
system.billing.usage | Every billable record: DBUs by SKU, workspace, job run, cluster or warehouse id, and tags | 365 days | Yes |
system.billing.list_prices | List price history per SKU (not your discount) | Indefinite | Yes |
system.compute.clusters, node_types | Cluster configuration history and the machine types available | Check the docs | Classic only |
system.compute.node_timeline | Per-minute utilisation of all-purpose and jobs compute | The docs I found disagree: 90 or 395 days | No |
system.query.history | Each query: who, when, duration, rows, bytes, status, source | The docs I found disagree: 180 or 365 days | Yes, for warehouses and serverless notebooks and jobs |
system.lakeflow.* | Job and pipeline definitions, run and task timelines, pipeline updates | Check the docs | Yes |
system.access.audit | Who did what, in which workspace, when | 365 days | Yes |
system.access.table_lineage, column_lineage | Each read or write of a Unity Catalog table, with column detail | 365 days | For work on governed compute |
| Pipeline event log | Update progress, data quality results, flow metrics, as a table in your storage | Yours | Yes |
| Query profile and job run page | Per-query operator detail and selected metrics. A UI, with the query history API behind it. | Per run | Yes. It replaces the Spark UI on serverless. |
| Spark UI, driver and executor logs | Stages, tasks, executors, stack traces | While the cluster exists, or in delivered logs | No |
An account admin can set a longer retention for system tables (in Beta at the time of the docs I read, with 395 days free). Billing and audit tables are the ones to copy out before they age off.
Extract
How to get it out programmatically
| Route | Use it for | Needs a warehouse? |
|---|---|---|
| SQL on system tables (SQL connector or SDK) | Billing, compute, lineage, audit and run timelines | Yes. The smallest one works. |
Query history API (GET /api/2.0/sql/history/queries) | Queries on warehouses and serverless, with filters by time, user and status. It works without Unity Catalog. | No, it is a control-plane call |
Jobs API (/api/2.1/jobs/runs/list and run details) | Run start and end, duration, task outcomes | No |
Clusters API events (/api/2.0/clusters/events) | Start, resize and terminate on classic clusters | No |
| Audit log delivery | Workspace audit events delivered as JSON or into system tables | No for JSON files. Yes to read the table. |
Rules that keep it cheap and correct: pull only new rows using a time watermark, re-read the last few days because billing rows can be corrected late, and upsert by key.
A sketch (not run against a workspace, so test it first)
# 1) system table, incremental, smallest serverless warehouse
from databricks import sql
conn = sql.connect(server_hostname=HOST, http_path=WAREHOUSE_PATH, access_token=TOKEN)
cur = conn.cursor()
cur.execute("""
SELECT * FROM system.billing.usage
WHERE usage_date >= date_sub(current_date(), 3) -- re-read recent days
""")
rows = cur.fetchall() # upsert into your store by record_id
# 2) query history, no warehouse needed
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import QueryFilter, TimeRange
w = WorkspaceClient()
queries = w.query_history.list(
filter_by=QueryFilter(query_start_time_range=TimeRange(start_time_ms=START_MS, end_time_ms=END_MS)),
include_metrics=True)
# 3) job runs since the last watermark
for run in w.jobs.list_runs(start_time_from=START_MS, expand_tasks=False):
pass # store run_id, start, end, state, job_id
Method and field names in the SDK change between versions, so check them against the version you install.
Cheapest
The cheapest way to read it
| Approach | Cost | Verdict |
|---|---|---|
| Call the REST APIs on a schedule with a watermark | No warehouse. Your own small script, and API rate limits to respect. | Pursue for query history, runs and cluster events |
| One daily incremental SQL pull of the system tables on a 2X-Small serverless warehouse | A 3-minute run is about 0.2 DBU (4 DBU per hour × 0.05 hour), so roughly $0.14 a day, about $4 a month at an illustrative $0.70 per DBU | Pursue for billing, lineage and audit |
| Deliver audit logs as files to your own storage | Storage and requests only | Pursue where you need raw audit data outside Databricks |
| Keep the copy in the store you already run for analysis | No Databricks compute at query time | Pursue. It is the design from the system tables page. |
| Dashboards or agents querying system tables directly | Compute on every query, on large tables | Drop |
| Third-party monitoring that polls constantly | That vendor's fees, plus warehouse time for polling | Drop unless you already use it |
Best setup. Two small scheduled jobs: one that calls the APIs, and one that runs the daily system-table pull. Both write to your own store. Everything you ask later runs there, not on Databricks.
More on why reading system tables costs money, and on copying them: system tables.
Checked
What was checked, and what was not
- Serverless limitations (no Spark UI, no Spark logs on serverless notebooks and jobs, query profile instead)
- System tables (schemas, retention, serverless fields)
- Query history API, plus community and third-party descriptions of the Jobs and Clusters APIs
- A monitoring vendor's summary of what serverless exposes
Not confirmed: the retention of node_timeline and query.history (sources disagree), the details of cluster log delivery to your storage, API rate limits, whether every endpoint needs no warehouse, the SDK method signatures, and the exact system table list for your region. Check each in your workspace.
FAQ
Questions people ask
Why is serverless compute opaque?
It runs in Databricks' account, so there are no machines to inspect. Databricks says Spark logs and the Spark UI are not available for serverless notebooks and jobs. You use the query profile, query history and system tables instead.
Can I see cost per job on serverless?
Yes. The billing usage system table fills in the job run id for serverless jobs, so you can total DBUs per job or run.
What is the cheapest way to read the logs?
Call the REST APIs on a schedule, which need no warehouse, and run one small daily incremental pull of the system tables on the smallest serverless warehouse. Keep the copy in your own store.
Is there an equivalent of reserved instances for serverless?
Not for machines, because they are Databricks' own. A commit discount on DBUs applies to serverless. On classic compute you can also reserve the VMs.
Next
Where to go next
Related on this site: Tagging best practices · Biggest cost and performance levers · Cost and FinOps for the platform · Databricks cost calculator
All Databricks compute and cost pages · All data engineering · Databricks on this site