Home › Learn › Databricks compute and cost › Logs and visibility

Databricks compute and cost · data engineering

Why serverless feels opaque: what Databricks logs, how to extract it programmatically and the cheapest way to read it

What you lose and what you keep on serverless, every source of run, query, billing, audit and lineage information with retention, how to extract it by SQL and REST API, and the cheapest way to do it.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

Why

Why serverless feels opaque

Classic compute runs on machines in your cloud account, so you can look inside them. Serverless runs in Databricks' account, so you get no machines to inspect. Databricks says Spark logs are not available for serverless notebooks and jobs (you only get client-side application logs), and the Spark UI is not available (the query profile replaces it). There is no host access and no way to install a monitoring agent.

What you want to seeClassic computeServerless
Spark UI (stages, tasks, skew)YesNo. Use the query profile and query insights.
Driver and executor logsYesNo Spark logs. Client-side application logs only.
Machine CPU and memory over timeYes, including system.compute.node_timeline for all-purpose and jobs computeNo machines to measure
Cluster start, resize and stop eventsYesNot applicable
Cost split by machine tagYes, tags reach the cloud billNo. Use budget policies and job or warehouse ids.
Cost per run, in DBUsYesYes. job_run_id is filled in for serverless jobs.
Per-query detailYesYes. Query history covers SQL warehouses and serverless notebooks and jobs.

So serverless is not blind. You trade machine-level detail for query-level and run-level detail.

What exists

What information Databricks keeps, by source

SourceWhat it holdsRetentionCovers serverless?
system.billing.usageEvery billable record: DBUs by SKU, workspace, job run, cluster or warehouse id, and tags365 daysYes
system.billing.list_pricesList price history per SKU (not your discount)IndefiniteYes
system.compute.clusters, node_typesCluster configuration history and the machine types availableCheck the docsClassic only
system.compute.node_timelinePer-minute utilisation of all-purpose and jobs computeThe docs I found disagree: 90 or 395 daysNo
system.query.historyEach query: who, when, duration, rows, bytes, status, sourceThe docs I found disagree: 180 or 365 daysYes, for warehouses and serverless notebooks and jobs
system.lakeflow.*Job and pipeline definitions, run and task timelines, pipeline updatesCheck the docsYes
system.access.auditWho did what, in which workspace, when365 daysYes
system.access.table_lineage, column_lineageEach read or write of a Unity Catalog table, with column detail365 daysFor work on governed compute
Pipeline event logUpdate progress, data quality results, flow metrics, as a table in your storageYoursYes
Query profile and job run pagePer-query operator detail and selected metrics. A UI, with the query history API behind it.Per runYes. It replaces the Spark UI on serverless.
Spark UI, driver and executor logsStages, tasks, executors, stack tracesWhile the cluster exists, or in delivered logsNo

An account admin can set a longer retention for system tables (in Beta at the time of the docs I read, with 395 days free). Billing and audit tables are the ones to copy out before they age off.

Extract

How to get it out programmatically

RouteUse it forNeeds a warehouse?
SQL on system tables (SQL connector or SDK)Billing, compute, lineage, audit and run timelinesYes. The smallest one works.
Query history API (GET /api/2.0/sql/history/queries)Queries on warehouses and serverless, with filters by time, user and status. It works without Unity Catalog.No, it is a control-plane call
Jobs API (/api/2.1/jobs/runs/list and run details)Run start and end, duration, task outcomesNo
Clusters API events (/api/2.0/clusters/events)Start, resize and terminate on classic clustersNo
Audit log deliveryWorkspace audit events delivered as JSON or into system tablesNo for JSON files. Yes to read the table.

Rules that keep it cheap and correct: pull only new rows using a time watermark, re-read the last few days because billing rows can be corrected late, and upsert by key.

A sketch (not run against a workspace, so test it first)

# 1) system table, incremental, smallest serverless warehouse
from databricks import sql
conn = sql.connect(server_hostname=HOST, http_path=WAREHOUSE_PATH, access_token=TOKEN)
cur = conn.cursor()
cur.execute("""
  SELECT * FROM system.billing.usage
  WHERE usage_date >= date_sub(current_date(), 3)   -- re-read recent days
""")
rows = cur.fetchall()          # upsert into your store by record_id

# 2) query history, no warehouse needed
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.sql import QueryFilter, TimeRange
w = WorkspaceClient()
queries = w.query_history.list(
    filter_by=QueryFilter(query_start_time_range=TimeRange(start_time_ms=START_MS, end_time_ms=END_MS)),
    include_metrics=True)

# 3) job runs since the last watermark
for run in w.jobs.list_runs(start_time_from=START_MS, expand_tasks=False):
    pass                         # store run_id, start, end, state, job_id

Method and field names in the SDK change between versions, so check them against the version you install.

Cheapest

The cheapest way to read it

ApproachCostVerdict
Call the REST APIs on a schedule with a watermarkNo warehouse. Your own small script, and API rate limits to respect.Pursue for query history, runs and cluster events
One daily incremental SQL pull of the system tables on a 2X-Small serverless warehouseA 3-minute run is about 0.2 DBU (4 DBU per hour × 0.05 hour), so roughly $0.14 a day, about $4 a month at an illustrative $0.70 per DBUPursue for billing, lineage and audit
Deliver audit logs as files to your own storageStorage and requests onlyPursue where you need raw audit data outside Databricks
Keep the copy in the store you already run for analysisNo Databricks compute at query timePursue. It is the design from the system tables page.
Dashboards or agents querying system tables directlyCompute on every query, on large tablesDrop
Third-party monitoring that polls constantlyThat vendor's fees, plus warehouse time for pollingDrop unless you already use it

Best setup. Two small scheduled jobs: one that calls the APIs, and one that runs the daily system-table pull. Both write to your own store. Everything you ask later runs there, not on Databricks.

More on why reading system tables costs money, and on copying them: system tables.

Checked

What was checked, and what was not

Not confirmed: the retention of node_timeline and query.history (sources disagree), the details of cluster log delivery to your storage, API rate limits, whether every endpoint needs no warehouse, the SDK method signatures, and the exact system table list for your region. Check each in your workspace.

FAQ

Questions people ask

Why is serverless compute opaque?

It runs in Databricks' account, so there are no machines to inspect. Databricks says Spark logs and the Spark UI are not available for serverless notebooks and jobs. You use the query profile, query history and system tables instead.

Can I see cost per job on serverless?

Yes. The billing usage system table fills in the job run id for serverless jobs, so you can total DBUs per job or run.

What is the cheapest way to read the logs?

Call the REST APIs on a schedule, which need no warehouse, and run one small daily incremental pull of the system tables on the smallest serverless warehouse. Keep the copy in your own store.

Is there an equivalent of reserved instances for serverless?

Not for machines, because they are Databricks' own. A commit discount on DBUs applies to serverless. On classic compute you can also reserve the VMs.

Next

Where to go next

Related on this site: Tagging best practices · Biggest cost and performance levers · Cost and FinOps for the platform · Databricks cost calculator

All Databricks compute and cost pages · All data engineering · Databricks on this site