Home › Learn › Databricks compute and cost › System tables

Databricks compute and cost · data engineering

Databricks system tables: contents, why reading them costs money, copying them cheaply, and what is hidden

What the system schemas hold, why querying them always costs compute, the cheapest incremental copy, whether Lakebase or UniForm would help, and the information Databricks does not show.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

System tables

System tables in detail

System tables are read-only tables in a catalog called system, inside Unity Catalog. Databricks fills them with operational data about your account: what was used, what it cost, who did what, what ran. You read them with ordinary SQL.

SchemaKey tablesWhat it answers
system.billingusage, list_pricesHow many DBUs, by which SKU, workspace, cluster, warehouse, job, user tags. And the list price per DBU.
system.computeclusters, node_types, node_timeline, warehouses, warehouse_eventsCluster configuration history, per-minute node CPU and memory use, warehouse scale events.
system.queryhistoryEvery SQL query: who, how long, rows read, bytes, whether it came from the result cache, how much came from the disk cache.
system.lakeflowjobs, job_run_timeline, job_task_run_timeline, pipelines, pipeline_update_timelineJob definitions, every run, task durations, pipeline updates. Join to billing for cost per job.
system.accessaudit, table_lineage, column_lineage, workspaces_latestWho did what and when, lineage between tables, workspace list.
system.serving, system.mlflowendpoint usage, served entities, experimentsModel serving traffic and tracking.
system.storagepredictive optimization historyWhich tables were auto-optimized and at what cost.

The most useful query shape: cost per job

Join usage to list prices on SKU and date, then multiply:

SELECT u.usage_metadata.job_id, SUM(u.usage_quantity * p.pricing.default) AS list_cost FROM system.billing.usage u JOIN system.billing.list_prices p ON u.sku_name = p.sku_name AND u.usage_start_time >= p.price_start_time AND (p.price_end_time IS NULL OR u.usage_start_time < p.price_end_time) WHERE u.usage_date >= current_date() - 30 GROUP BY 1

The result is a list-price estimate, not your invoice (see the hidden-information list).

System tables

Why querying system tables always costs money

Reading a system table is a query, and a query needs compute. There is no free metadata endpoint. So even the act of asking "what did I spend?" makes a small spend.

It is a small cost if you design for it. Done well, the whole cost-tracking system costs a few dollars a month. Done carelessly (a wide dashboard on a big warehouse refreshing every few minutes) it becomes a visible line item of its own.

MitigationHowVerdict
Roll up once, query the rollupOne scheduled job, on the smallest compute, reads only yesterday's rows (filter on usage_date) and appends to your own small table. Dashboards read that table.Pursue. The single biggest saving.
Always filter on dateUse usage_date or the time column, and select only the columns you needPursue
Use a tiny warehouse with short auto-stop2X-Small serverless, 1 to 5 minute auto-stop for admin queriesPursue
Use the free viewsThe account console usage page and budgets need no compute of yours. Billable usage log delivery can also write raw usage files to your own storage.Pursue for finance's basic needs
Dashboard that refreshes every few minutesBilling data is hours old, so frequent refresh buys nothingDrop. Refresh daily.
Copy before retention endsRollup table doubles as your long-term historyPursue

System tables

Copying system tables cheaply, and is Lakebase the answer?

Short answer. Lakebase does not remove the cost of reading system tables. Reading them needs compute, and it still needs compute to copy them. Lakebase can only make later reads cheaper. For a cost dashboard, a small Delta table plus a tiny serverless warehouse is usually cheaper and simpler.

Freshness has a ceiling you cannot beat. Billing usage reaches the system table with a delay of hours. Copying it every minute only copies the same rows many times. Match the copy schedule to how fast each source table updates.

Source tableRough update speedSensible copy schedule
billing.usageHours behindEvery 1 to 4 hours, or daily
compute.warehouse_events, query.historyMinutes behindEvery 15 to 60 minutes, only if someone watches live
access.auditMinutes to an hourHourly
compute.node_timelineLarge and detailedDaily, and keep only what you analyse
system.* tablesbig, managed by Databricks Small scheduled jobreads only rows newer thanthe saved watermark Your Delta rollup tablessmall, in your catalog,kept past retention 2X-Small warehouseshort auto-stop,result cache on MERGE Lakebase (optional)synced table, Postgres reads App or APIneeds millisecond reads
The cost you cannot avoid is the one small job that reads the big source tables. Everything after that reads small tables you own.
OptionCost to copyCost to queryVerdict
Incremental job into your own Delta table, read with a 2X-Small serverless warehouseMinutes of tiny compute per runSeconds of a tiny warehouse. Result cache helps.Pursue. Cheapest and cleanest.
Same, plus a synced table into LakebaseSame job, plus Lakebase sync and Lakebase storagePostgres compute (can scale down), no warehouse neededPursue only if an app needs very fast, frequent reads. Otherwise Drop.
Copy everything every few minutes for "freshness"High, many repeated scansSame as aboveDrop. Source is not that fresh anyway.
Query system tables directly from every dashboardNoneBig scans on every refreshDrop
Use the account console usage page and billable usage log delivery for financeNo computeNo computePursue for the basic monthly view

One trap when copying billing data. system.billing.usage can restate earlier rows. A row has a record_type of original, retraction or restatement, and a record_id. If you only append, corrections double count. Copy with a MERGE on record_id, or sum with the record types applied. Also, re-read the last day or two on each run to catch late rows.

System tables

Why are system tables not Delta with UniForm, so reading is free?

Format is not what makes a query cost money. Compute is. Even a perfectly open Delta or Iceberg table needs some engine to read it, and whichever engine you use burns CPU somewhere. UniForm makes one Delta table readable as Iceberg or Hudi too, so other engines can read it. That helps with which engine you use, not with whether you pay for one.

Why Databricks delivers system tables as managed, read-only shared data (I am reasoning from how the product works, not quoting Databricks):

What would actually make reading cost nothing is an engine you already pay for. Options that exist today: copy to your own Delta table and read it with something cheap, deliver usage logs to your own bucket and read them with DuckDB or Athena (Athena charges per scan, DuckDB on a laptop is free), or look at the account console.

System tables

What Databricks does not show you (gaps worth knowing)

This list is my assessment of the real gaps people hit, not an official statement. Some may change, and a few have partial answers.

What is missingWhy it hurtsPartial workaround
Serverless machine details: node count, type, CPU and memory use, scale-up and scale-down decisionsYou cannot prove or disprove over-scaling. You can only watch DBU per run.Track DBU per run over time. Compare against an equivalent classic run once.
The split inside a serverless DBU: Databricks charge versus the machine costYou cannot judge if the premium is fair against classicRun the same workload both ways and compare total invoices.
Cost per query and per userBilling stops at the warehouse or cluster. Shared compute cannot be split exactly.Estimate using query history shares. Prefer one warehouse per BU.
Actual contract prices in system tableslist_prices shows list, not your negotiated rate, so computed dollars are approximateApply your discount factor in the rollup, and reconcile to the invoice monthly.
Cloud-side costs (VMs, disks, network, storage)Not in system tables at allExport the cloud cost report and join on tags or cluster id.
Fine time resolution in billingUsage records are roughly hourly, so a scale-up inside the hour is blendedJoin with node timeline or warehouse events for minute-level detail on classic.
Why Photon fell back for a queryHard to know why a query is slower than expectedOpen the query profile or Spark UI and look at which operators are not Photon.
Data freshness and retention limitsNot real-time, and old data disappearsCopy daily into your own table.

What is available and often forgotten: whether a query hit the result cache and how much of its read came from the disk cache are both columns in system.query.history, so cache effectiveness can be measured.

FAQ

Questions people ask

Why does querying system tables cost money?

A query needs a running warehouse or cluster, the tables are large, and you cannot index or reorganise them. Roll up once into your own small table and query that.

Does copying them to Lakebase solve it?

No. The copy still reads the system tables with compute. Lakebase only makes later reads cheaper, and a small Delta table with a tiny serverless warehouse is usually cheaper and simpler.

Would UniForm make system tables free to read?

No. Format is not what costs money. Any reader needs compute somewhere. UniForm would only let you choose a different engine.

Next

Where to go next

Related on this site: Tagging best practices · UniForm · Cost and FinOps for the platform

All Databricks compute and cost pages · All data engineering · Databricks on this site