CI/CD for Databricks with Asset Bundles
On Monday morning the nightly orders job in production fails. Someone changed its cluster last week. Who? From what? Nobody knows, because the job was built by clicking through the workspace UI, and the UI has no pull request.
Most teams start this way. The code lives in Git, but the job that runs the code, its schedule, its cluster and its parameters live only in the workspace. Building the same job in a test workspace means clicking it together again, slightly differently.
Databricks Asset Bundles fix this. This post explains them in plain words, then shows a simple pipeline that tests and deploys them for you.
The idea: pack once, carry everywhere
A bundle is a project folder that describes your Databricks resources as code. Resources are the things you would otherwise create by hand: jobs, pipelines, and a few other kinds.
You pack the suitcase once. It holds your code and a YAML file that says which jobs to create, on what compute, on what schedule. Then you carry the same suitcase to three hotels: a dev workspace, a test workspace and a production workspace. Each hotel is called a target. The contents don't change between hotels. Only a few settings do, such as which workspace to use and which catalog to write to.
Because the whole thing is text in Git, every change to a job is a reviewed pull request. You can see who changed the cluster, and you can roll it back.
One folder
You can start a new bundle with databricks bundle init, which asks a few questions and creates a project from a template. Use the current Databricks CLI for this; the old Python package called databricks-cli does not have bundles. Most bundles end up looking like the picture:
databricks.ymlat the top. It names the bundle, lists the variables and defines the targets.resources/holds one YAML file per job or pipeline.databricks.ymlincludes them, so the main file stays short.src/holds the notebooks and Python files the jobs run.tests/holds the unit tests for your transformation code.
Here is a short sketch, not a complete file. The hosts, names and cluster settings are made up for the example.
# databricks.yml (a sketch)
bundle:
name: sales_pipelines
include:
- resources/*.yml
variables:
catalog:
description: The catalog this bundle writes to
default: sales_dev
targets:
dev:
mode: development
default: true
workspace:
host: https://dev-workspace.example.com
test:
workspace:
host: https://test-workspace.example.com
variables:
catalog: sales_test
prod:
mode: production
workspace:
host: https://prod-workspace.example.com
run_as:
service_principal_name: <application id of the prod service principal>
variables:
catalog: sales_prod
And one job in resources/daily_orders.yml, also a sketch:
resources:
jobs:
daily_orders:
name: daily_orders
schedule:
quartz_cron_expression: "0 0 2 * * ?"
timezone_id: Asia/Kolkata
job_clusters:
- job_cluster_key: main
new_cluster:
policy_id: ${var.job_policy_id}
spark_version: 15.4.x-scala2.12
num_workers: 2
tasks:
- task_key: load_orders
job_cluster_key: main
notebook_task:
notebook_path: ../src/load_orders.py
base_parameters:
catalog: ${var.catalog}
Two things are worth noticing. ${var.catalog} means the same job writes to sales_dev, sales_test or sales_prod depending on the target. And the job runs on job compute with a cluster policy, a set of rules that limits what a cluster may look like, written into the definition. That is exactly where the cheapest fixes from our Databricks cost leaks note belong: once they are in the bundle, every deploy keeps them, and nobody can quietly switch the job to an always-on interactive cluster. (A job_policy_id variable would also need declaring under variables; the sketch leaves it out.)
Four commands
You only need a handful of commands, all run from the bundle folder.
databricks bundle validatereads the YAML, fills in the variables for the chosen target and reports mistakes. It changes nothing.databricks bundle deployuploads the code and creates or updates the jobs and pipelines in the target's workspace. Run it again and it updates what is there, rather than making a second copy.databricks bundle run daily_ordersstarts one job or pipeline from the bundle and waits for it to finish.databricks bundle destroyremoves everything that bundle deployed to a target. Handy for cleaning up a dev copy; think twice before pointing it at prod.
You pick the target with -t, as in databricks bundle deploy -t test. With no target, the one marked default: true is used, which in the sketch is dev.
Dev mode: everyone gets their own copy
The mode: development setting on the dev target is what makes a shared dev workspace bearable.
When Priya deploys in development mode, every job gets her name in front, like [dev priya] daily_orders, and the files go into her own folder in the workspace. Arjun's deploy makes [dev arjun] daily_orders. They can both change the same job on the same day without breaking each other. Schedules on these copies are paused, so twenty developer copies don't all run at 2 a.m.
The prod target uses mode: production. No prefix, schedules run as written, and the bundle applies a few extra safety checks before it will deploy. Pair it with run_as, so the jobs run as a service principal and not as whoever happened to deploy.
The pipeline
A service principal is an identity for a machine rather than a person. Your CI system logs in as one. It has exactly the rights it needs in each workspace, and it doesn't leave the company when someone resigns.
Here is the shape, using GitHub Actions as the example. Azure DevOps or GitLab work the same way.
- On every pull request, install the Databricks CLI, run
databricks bundle validate -t test, then run the unit tests. If either fails, the pull request can't merge. - On merge to main, log in as the test service principal and run
databricks bundle deploy -t test. - Run a small integration test with
databricks bundle run -t test daily_orders, against a small, fixed sample of data, and check the output table. - Wait for approval. In GitHub Actions, a protected environment can require a named person to approve before the next job starts.
- Deploy to prod as the prod service principal with
databricks bundle deploy -t prod. Same commit, same bundle.
The service principal's credentials live in the CI system's secret store, never in the repository. The Databricks CLI reads them from environment variables, such as the workspace host plus the service principal's client ID and secret, so nothing sensitive ever appears in the YAML.
Tests: two kinds, both small
Unit tests check your transformation code. Keep the logic in plain Python functions in src/ that take a DataFrame and return a DataFrame. Then a test can build three rows, call the function and check the answer, using pytest and a local Spark session. These run on the CI machine in a minute and need no workspace. A test might say: an order with a refund of ₹40,000 must show net revenue of zero.
The integration run checks that everything fits together in a real workspace: the job deploys, the cluster starts, the permissions work and the table gets written. Keep it to one or two jobs on a small sample. It is a smoke test, not a full reload.
If you use a metadata-driven ingestion framework, the same idea applies twice: the engine gets unit tests, and the config rows get validated in the pull request. Our metadata-driven ingestion post shows that setup, and a bundle is a natural way to deploy the engine job.
Mistakes we see
- Clicking jobs together in the prod UI. Even one hand-made job means prod no longer matches Git. Make prod read-only for people, and let only the service principal deploy.
- A personal token in CI. It works until that person goes on leave or leaves the company, and meanwhile every deploy runs with their rights. Use a service principal per environment.
- One target for everything. Without separate targets, "test" means "prod with fingers crossed". Use at least dev and prod; test is worth adding as soon as more than one person deploys.
- No tests before deploy.
validatecatches broken YAML, not broken logic. A five-line unit test catches the bug that would otherwise reach the Monday report. - Secrets in the YAML. Anything in
databricks.ymlis in Git history forever. Keep passwords and keys in a Databricks secret scope or your cloud's key vault, and refer to them by name.
Questions to check your own setup
- Could you rebuild every production job in a new workspace from Git alone?
- Who, or what, ran the last deploy to prod?
- Can two developers change the same job on the same day without breaking each other?
- Does a pull request run validate and unit tests before anyone can merge it?
- Is there a password, token or key anywhere in your repository?
- Do your jobs run on job compute with a cluster policy, written in the bundle?
If prod still has jobs that exist only in the UI, start by moving one of them into a bundle. Once every change is a reviewed pull request, a broken job on Monday morning becomes a question you can answer.
Go deeper: CI/CD, the idea behind it
Questions or disagreements: basant@cleandataengineering.ai