Home › Learn › Databricks compute and cost › System tables
Databricks compute and cost · data engineering
Databricks system tables: contents, why reading them costs money, copying them cheaply, and what is hidden
What the system schemas hold, why querying them always costs compute, the cheapest incremental copy, whether Lakebase or UniForm would help, and the information Databricks does not show.
Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.
System tables
System tables in detail
System tables are read-only tables in a catalog called system, inside Unity Catalog. Databricks fills them with operational data about your account: what was used, what it cost, who did what, what ran. You read them with ordinary SQL.
- The account admin enables each schema, and you grant access like any other table.
- They are owned and maintained by Databricks. You cannot edit them, add columns or fix a wrong row.
- The billing tables cover the whole account. Many operational tables cover the workspaces of one region and metastore.
- Retention is limited, typically around a year, so copy anything you want to keep longer.
- Data arrives with a delay (minutes for some, hours for billing). Do not use them for real-time alerts.
| Schema | Key tables | What it answers |
|---|---|---|
system.billing | usage, list_prices | How many DBUs, by which SKU, workspace, cluster, warehouse, job, user tags. And the list price per DBU. |
system.compute | clusters, node_types, node_timeline, warehouses, warehouse_events | Cluster configuration history, per-minute node CPU and memory use, warehouse scale events. |
system.query | history | Every SQL query: who, how long, rows read, bytes, whether it came from the result cache, how much came from the disk cache. |
system.lakeflow | jobs, job_run_timeline, job_task_run_timeline, pipelines, pipeline_update_timeline | Job definitions, every run, task durations, pipeline updates. Join to billing for cost per job. |
system.access | audit, table_lineage, column_lineage, workspaces_latest | Who did what and when, lineage between tables, workspace list. |
system.serving, system.mlflow | endpoint usage, served entities, experiments | Model serving traffic and tracking. |
system.storage | predictive optimization history | Which tables were auto-optimized and at what cost. |
The most useful query shape: cost per job
Join usage to list prices on SKU and date, then multiply:
SELECT u.usage_metadata.job_id, SUM(u.usage_quantity * p.pricing.default) AS list_cost FROM system.billing.usage u JOIN system.billing.list_prices p ON u.sku_name = p.sku_name AND u.usage_start_time >= p.price_start_time AND (p.price_end_time IS NULL OR u.usage_start_time < p.price_end_time) WHERE u.usage_date >= current_date() - 30 GROUP BY 1
The result is a list-price estimate, not your invoice (see the hidden-information list).
System tables
Why querying system tables always costs money
Reading a system table is a query, and a query needs compute. There is no free metadata endpoint. So even the act of asking "what did I spend?" makes a small spend.
- It needs a running warehouse or cluster. A serverless SQL warehouse burns DBUs while it is up. A dashboard on a schedule keeps it awake.
- The tables are large. Audit logs, node timelines and query history can have billions of rows. An unfiltered scan reads a lot of data.
- They are managed by Databricks, and shared to you. You cannot add your own indexes or physically reorganise them. You can only filter by date and columns.
- Every repeat costs again. The same dashboard opened by ten people runs ten times, unless the result cache catches it.
It is a small cost if you design for it. Done well, the whole cost-tracking system costs a few dollars a month. Done carelessly (a wide dashboard on a big warehouse refreshing every few minutes) it becomes a visible line item of its own.
| Mitigation | How | Verdict |
|---|---|---|
| Roll up once, query the rollup | One scheduled job, on the smallest compute, reads only yesterday's rows (filter on usage_date) and appends to your own small table. Dashboards read that table. | Pursue. The single biggest saving. |
| Always filter on date | Use usage_date or the time column, and select only the columns you need | Pursue |
| Use a tiny warehouse with short auto-stop | 2X-Small serverless, 1 to 5 minute auto-stop for admin queries | Pursue |
| Use the free views | The account console usage page and budgets need no compute of yours. Billable usage log delivery can also write raw usage files to your own storage. | Pursue for finance's basic needs |
| Dashboard that refreshes every few minutes | Billing data is hours old, so frequent refresh buys nothing | Drop. Refresh daily. |
| Copy before retention ends | Rollup table doubles as your long-term history | Pursue |
System tables
Copying system tables cheaply, and is Lakebase the answer?
Short answer. Lakebase does not remove the cost of reading system tables. Reading them needs compute, and it still needs compute to copy them. Lakebase can only make later reads cheaper. For a cost dashboard, a small Delta table plus a tiny serverless warehouse is usually cheaper and simpler.
Freshness has a ceiling you cannot beat. Billing usage reaches the system table with a delay of hours. Copying it every minute only copies the same rows many times. Match the copy schedule to how fast each source table updates.
| Source table | Rough update speed | Sensible copy schedule |
|---|---|---|
billing.usage | Hours behind | Every 1 to 4 hours, or daily |
compute.warehouse_events, query.history | Minutes behind | Every 15 to 60 minutes, only if someone watches live |
access.audit | Minutes to an hour | Hourly |
compute.node_timeline | Large and detailed | Daily, and keep only what you analyse |
| Option | Cost to copy | Cost to query | Verdict |
|---|---|---|---|
| Incremental job into your own Delta table, read with a 2X-Small serverless warehouse | Minutes of tiny compute per run | Seconds of a tiny warehouse. Result cache helps. | Pursue. Cheapest and cleanest. |
| Same, plus a synced table into Lakebase | Same job, plus Lakebase sync and Lakebase storage | Postgres compute (can scale down), no warehouse needed | Pursue only if an app needs very fast, frequent reads. Otherwise Drop. |
| Copy everything every few minutes for "freshness" | High, many repeated scans | Same as above | Drop. Source is not that fresh anyway. |
| Query system tables directly from every dashboard | None | Big scans on every refresh | Drop |
| Use the account console usage page and billable usage log delivery for finance | No compute | No compute | Pursue for the basic monthly view |
One trap when copying billing data. system.billing.usage can restate earlier rows. A row has a record_type of original, retraction or restatement, and a record_id. If you only append, corrections double count. Copy with a MERGE on record_id, or sum with the record types applied. Also, re-read the last day or two on each run to catch late rows.
System tables
Why are system tables not Delta with UniForm, so reading is free?
Format is not what makes a query cost money. Compute is. Even a perfectly open Delta or Iceberg table needs some engine to read it, and whichever engine you use burns CPU somewhere. UniForm makes one Delta table readable as Iceberg or Hudi too, so other engines can read it. That helps with which engine you use, not with whether you pay for one.
Why Databricks delivers system tables as managed, read-only shared data (I am reasoning from how the product works, not quoting Databricks):
- The data lives in Databricks' storage, not yours. That keeps storage cost, schema upgrades and access control in one place. If it were in your bucket, you would pay for storage, and they would need write access to every customer's storage.
- One central pipeline serves everyone. Databricks can add columns and fix data for all customers without touching customer buckets.
- Governance. Reads go through Unity Catalog and the sharing layer, so permission checks and audit are consistent.
- Read-only guarantee. A customer cannot corrupt their own audit or billing history.
What would actually make reading cost nothing is an engine you already pay for. Options that exist today: copy to your own Delta table and read it with something cheap, deliver usage logs to your own bucket and read them with DuckDB or Athena (Athena charges per scan, DuckDB on a laptop is free), or look at the account console.
System tables
What Databricks does not show you (gaps worth knowing)
This list is my assessment of the real gaps people hit, not an official statement. Some may change, and a few have partial answers.
| What is missing | Why it hurts | Partial workaround |
|---|---|---|
| Serverless machine details: node count, type, CPU and memory use, scale-up and scale-down decisions | You cannot prove or disprove over-scaling. You can only watch DBU per run. | Track DBU per run over time. Compare against an equivalent classic run once. |
| The split inside a serverless DBU: Databricks charge versus the machine cost | You cannot judge if the premium is fair against classic | Run the same workload both ways and compare total invoices. |
| Cost per query and per user | Billing stops at the warehouse or cluster. Shared compute cannot be split exactly. | Estimate using query history shares. Prefer one warehouse per BU. |
| Actual contract prices in system tables | list_prices shows list, not your negotiated rate, so computed dollars are approximate | Apply your discount factor in the rollup, and reconcile to the invoice monthly. |
| Cloud-side costs (VMs, disks, network, storage) | Not in system tables at all | Export the cloud cost report and join on tags or cluster id. |
| Fine time resolution in billing | Usage records are roughly hourly, so a scale-up inside the hour is blended | Join with node timeline or warehouse events for minute-level detail on classic. |
| Why Photon fell back for a query | Hard to know why a query is slower than expected | Open the query profile or Spark UI and look at which operators are not Photon. |
| Data freshness and retention limits | Not real-time, and old data disappears | Copy daily into your own table. |
What is available and often forgotten: whether a query hit the result cache and how much of its read came from the disk cache are both columns in system.query.history, so cache effectiveness can be measured.
FAQ
Questions people ask
Why does querying system tables cost money?
A query needs a running warehouse or cluster, the tables are large, and you cannot index or reorganise them. Roll up once into your own small table and query that.
Does copying them to Lakebase solve it?
No. The copy still reads the system tables with compute. Lakebase only makes later reads cheaper, and a small Delta table with a tiny serverless warehouse is usually cheaper and simpler.
Would UniForm make system tables free to read?
No. Format is not what costs money. Any reader needs compute somewhere. UniForm would only let you choose a different engine.
Next
Where to go next
Related on this site: Tagging best practices · UniForm · Cost and FinOps for the platform
All Databricks compute and cost pages · All data engineering · Databricks on this site