Databricks Data Engineer Professional
Section 8 · 10%Debugging and Deploying

Debugging and deploying

Spark UI, logs and query profiles, job repair and parameter overrides, Declarative Automation Bundles and Git folders CI/CD.

This section tests whether you read the right diagnostic first and then fix the problem with the smallest safe change, and whether you know how code gets from a repository into a workspace. The biggest way people lose marks is rerunning or redeploying everything when a targeted repair or a single configuration change would do, or reading a driver log when the evidence is in the query profile.

PracticeS8 practice bank →

6 items, practice or timed exam mode.

Walk the guide, line by line

Each row is one line of the official guide, in its own words on the left. Read across: plain words, what a question looks like, the answer, and the trap.

Guide lineIn plain wordsA question looks likeThe answerThe trap
8.1 Identify pertinent diagnostic information using Spark UI, cluster logs, system tables, and query profiles to troubleshoot errors.Each tool shows a different layer. The Spark UI shows jobs, stages and tasks for Spark work. Cluster logs show driver and executor output, including stack traces. System tables show history across runs. The query profile shows each operator of a SQL query with time and rows.A SQL warehouse query got slower and the engineer needs to see which step of the query takes the time. The deciding fact is that the work ran on a SQL warehouse as one query.Open the query profile from the query history, look at the top operators and the rows and time per operator, and see where the time goes, such as a scan or a shuffle. For a notebook job on a cluster, use the Spark UI stages and task times, and the driver log for exceptions. Use system tables to see whether this is new or has happened before.
Try it: Run any query on a serverless warehouse, open it from query history and click See query profile.
Reading the driver log for a slow but successful query shows nothing useful, because there is no error. The Spark UI is also the wrong first stop for a SQL warehouse query, where the query profile is the operator-level view.
Runner-up wins when: The Spark UI wins when the work runs on a cluster and you need task-level evidence such as one task far slower than the rest.
8.2 Analyze the errors and remediate the failed job runs with job repairs and parameter overrides.Job repair reruns only the tasks that failed or were canceled, plus the tasks that depend on them. Tasks that already succeeded are kept. In the repair dialog you can override job parameters for the rerun.A four-task job failed on the third task because a date parameter was wrong, and the first two tasks took two hours. The deciding fact is that the earlier tasks succeeded and only a parameter must change.Use Repair run, enter the corrected parameter value in the dialog, and let Databricks rerun the failed task and its downstream tasks only. Repair uses the current job and task settings, so a fixed notebook path or cluster setting also applies. Each repaired task starts from the beginning, so write tasks so a rerun does not duplicate data.
Try it: Make a two-task job fail on the second task, fix it, and use Repair run to see only that task and its dependents rerun.
Starting a brand new run with Run now repeats the two hours of successful work. Repair also applies only to jobs with at least two tasks.
Runner-up wins when: Run now wins for a single-task job, where repair is not available, or when you want a fresh run rather than a continuation of the same run.
8.3 Deploy Databricks resources using Declarative Automation Bundles (formerly Databricks Asset Bundles).Declarative Automation Bundles describe jobs, pipelines and other resources as YAML files in your repository, with a databricks.yml file at the root. The Databricks CLI validates, deploys and runs them. The docs use this name; it was formerly Databricks Asset Bundles.A team has a job and a pipeline defined by hand in the development workspace and must deploy the same set to production from a pipeline run on each merge. The deciding fact is that the same definition must land in two environments.Put the resources in a bundle with one target per environment, then run databricks bundle validate and databricks bundle deploy with the target name in CI. Targets override workspace, variables, permissions and run-as per environment. Only one target is marked as the default.
Try it: Run databricks bundle init, edit the job, then validate and deploy to your Free Edition workspace.
Exporting a job as JSON and recreating it by hand in production drifts over time and is not reviewed in a pull request. Bundles keep the definition in source control.
Runner-up wins when: A Git folder wins when you only need notebooks and code files updated in a workspace and do not need to create or change jobs and pipelines.
8.4 Configure Git-based CI/CD workflows using Databricks Git folders (formerly Repos) to deploy notebooks and code.A Git folder links a workspace folder to a Git repository. The recommended pattern is that developers work in their own folder on a branch, and a production Git folder follows a deployment branch that only automation updates.A team wants the production notebooks to match the main branch after each merged pull request, with nobody editing code by hand in production. The deciding fact is that production must follow a branch automatically.Create a production Git folder outside user folders, track the deployment branch, and let an automation identity update it when a pull request merges. A service principal with a Git credential and Can Run on that folder can call the Repos API or SDK from the CI system. Limit edit rights on the folder to admins and the service principal.
Try it: Clone a small repository into a Git folder, make a branch, push a change, and pull it into the folder.
Letting developers pull branches into the production folder by hand allows unreviewed changes and drift. The page says production folders should be updated only by automation.
Runner-up wins when: Bundles win when the change includes job or pipeline settings, because a Git folder syncs files and does not deploy resource definitions.

Mind map: Which diagnostic do I open first

People assume the driver log is the place for every problem, but logs show errors and most slowness comes with no error at all.

What you observeWhat it points toWhat to do (and when the runner-up wins)
A job throws an exceptionDriver or executor log and the failed task outputRead the stack trace; if one task is far slower than peers without an error, use the Spark UI stages tab instead
A SQL warehouse query is slowQuery profileFind the top operator; the runner-up Spark UI wins for cluster notebook work
Is this failure new or a repeatSystem tables for run historyQuery lakeflow run timelines over weeks
Slow stage on a clusterSpark UI stages and tasksCompare task durations to spot skew

Mind map: Fix a failed run or ship code

People assume redeploying or rerunning everything is the safe move, but the smallest targeted action is safer and cheaper.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Later task failed, earlier tasks succeededRepair runOverride the wrong parameter in the dialog; for a single-task job use Run now
Same job and pipeline must reach several workspacesBundle with targetsValidate and deploy in CI
Only notebooks and code files must follow a branchProduction Git folder updated by automationUpdate it on merge; if jobs change too, use a bundle
Fix lives in a notebook path or cluster settingRepair uses current settingsEdit the job, then repair