Home › Learn › Databricks compute and cost › Backup architecture

Databricks compute and cost · data engineering

Backup and recovery architecture for Databricks: the safe, clean design on any cloud

Time travel is not a backup. Three protections for three disasters: fast undo, an immutable backup in a separate account, and a rebuildable second region, with the design rules and a failure-to-tool table.

Cluster anatomyWarehouses and serverlessChoosing computeCompute settingsCost by workloadLogs and visibilityPhoton and cachesHow billing worksDividing the billSystem tablesSharing and regionsBackup architectureCost-saving hacks

Prices, defaults, limits and preview status change often, and dollar figures here are illustrations. Check the current Databricks documentation and price list before quoting a number.

Backup and recovery

Best architecture for backup and recovery

Start with the surprise: Delta time travel is not a backup. It lives in the same storage as the table, and VACUUM deletes old files by design (history is kept for a limited time, often a week). Time travel is a fast undo. A backup is a separate copy that an attacker or a bad script cannot touch.

PRODUCTION (region A, prod cloud account) Workspace + Unity Catalog metastorejobs, warehouses, permissions Storage with versioning and soft deleteDelta tables. Time travel and RESTORE for quick undo.UNDROP for dropped managed tables (short window) Git repos + Terraform / Asset Bundlescode, jobs, workspaces, grants as text BACKUP ACCOUNT (separate credentials) Write-only role from productionObject lock / immutability (cannot delete)Incremental DEEP CLONE of critical tables DR REGION B (secondary) Second workspace and metastore, built from TerraformData copied by deep clone, tested restoreCompute off until a disaster, then started clone clone What protects against what Bad write or bad merge: RESTORE to a past version. Minutes. Table dropped: UNDROP, or restore from storage versioning. Files deleted or VACUUM too early: restore from backup account. Ransomware, stolen admin key: immutable backup account. Region outage: DR region.
Three separate protections for three separate disasters: fast undo in place, an immutable copy in another account, and a rebuildable second region. Code and configuration live as text in Git, so the control side can be recreated.
FailureToolTypical recovery speedVerdict
Bad write, bad MERGE, wrong job runDelta time travel and RESTORE TABLEMinutesPursue as first line
Table dropped by mistakeUNDROP TABLE (managed tables, limited window) or object versioningMinutes to hoursPursue
Files lost, vacuumed early or corruptedIncremental deep clone in a separate locationHoursPursue
Malicious delete, ransomware, compromised adminBackup in a separate account with immutability and no delete rights for productionHoursPursue for critical data
Region outageSecondary region with replicated data and Terraform-built workspaceHours (warm) to a day (cold)Pursue only for what the business needs. It is expensive.
Workspace or job configuration lostTerraform, Asset Bundles, GitHoursPursue
Copying the raw bucket between regions with plain storage replication, for Delta tablesRaw replication can arrive out of order, so a table may point at files that have not arrived yetRisk of a broken tableDrop for Delta. Use deep clone, or validate after replication.

The design rules

  1. Decide RPO and RTO per data set. RPO is how much data you can afford to lose, RTO is how long you can be down. Most tables do not need DR. A few do.
  2. Everything that can be code is code. Workspaces, catalogs, grants, jobs, warehouses and policies in Terraform or Asset Bundles, kept in Git.
  3. Backups live in a different account with different credentials. Production may write, but never delete. Turn on immutability.
  4. Use Delta-aware copies. Deep clone is incremental, copies data and log together, and is consistent.
  5. Unity Catalog metadata is in the control plane. Back up its definitions (catalogs, schemas, grants, external locations) through Terraform or API exports, because the data copy alone does not recreate permissions.
  6. Practise the restore. A backup you have never restored is a hope. Schedule a restore drill and time it.
  7. Mind the bill. Cross-region copies cost transfer fees (see cross-region cost) and storage. Back up gold and critical tables, not every scratch table.

FAQ

Questions people ask

Is Delta time travel a backup?

No. It lives in the same storage and VACUUM deletes old files by design. It is a fast undo.

Is copying the bucket to another region safe for Delta tables?

Not by itself. Files can arrive out of order, so a table's log may point at files that have not arrived. Use deep clone or validate after replication.

Next

Where to go next

Related on this site: Backup and recovery on Azure · Cheapest way to move data in and out

All Databricks compute and cost pages · All data engineering · Databricks on this site