Databricks Data Engineer Professional
Section 1 · 23%Developing Code for Data Processing using Python and SQL

Developing code

Bundles, libraries, UDFs, Lakeflow Pipelines and Auto Loader, Jobs control flow, AUTO CDC, streaming semantics and tests. The biggest section: 23%.

This section tests whether you can choose the right building block for code that runs on Databricks: how a project is packaged, how its libraries are installed, which kind of function to write, and how data pipelines and jobs are defined and run. The biggest way people lose marks is picking the tool they know best instead of the one that fits the one constraint in the question, such as who may run a function, how fresh the data must be, or whether the source only ever appends.

PracticeS1 practice bank →

14 items, practice or timed exam mode.

Walk the guide, line by line

Each row is one line of the official guide, in its own words on the left. Read across: plain words, what a question looks like, the answer, and the trap.

Guide lineIn plain wordsA question looks likeThe answerThe trap
1.1 Implement scalable Python project structures for Declarative Automation Bundles (formerly Databricks Asset Bundles / DABs) to support modular development and CI/CD integration.A Declarative Automation Bundle (the new name for Databricks Asset Bundles) is a folder with a databricks.yml file that describes your jobs, pipelines and targets as code. A scalable Python project keeps the real logic in an installable package, keeps notebooks or entry scripts thin, and lets the bundle build and deploy that package the same way in every environment.A team has business logic copied between five notebooks and wants unit tests, a pull-request build, and the same code promoted from dev to prod. The deciding fact is that the code must be reusable and deployable by an automated pipeline.Move the logic into a Python package with a pyproject.toml, declare it in the bundle's artifacts section with type whl and a build command, and reference the built wheel under the task's libraries. Run databricks bundle validate and databricks bundle deploy with a target in the CI pipeline. Define dev and prod as targets in databricks.yml so the same code deploys with different settings.
Try it: Run databricks bundle init with the default-python template, then validate and deploy to a dev target and read the generated databricks.yml.
Deploying notebooks that import each other through workspace paths works, but it cannot be versioned as a unit, unit tested outside the workspace, or promoted as one artifact, so it fails the CI/CD requirement.
Runner-up wins when: Notebook tasks deployed by the bundle win when the work is short, exploratory glue with no reuse and no tests are needed.
1.2 Apply troubleshooting techniques to dependency conflicts and installation failures for external libraries (PyPI, local wheels, source archives) across serverless, pipeline, and bundle-deployed environments.When a library will not install or breaks something, you find where the dependency is declared and which environment is running it. On serverless compute you set dependencies in the environment (a base environment version plus a list of packages written like a requirements.txt). Wheels and Python projects can come from workspace files or Unity Catalog volumes.A job on serverless compute fails to install a private wheel that another team updated, or a package pulls in PySpark and the session stops. The deciding fact is which environment declares the library and what changed in it.Read the pip logs and the Installed tab to see the real conflict. Do not install PySpark or any library that depends on it on serverless, because that stops the session; remove it and reset the environment. If you changed a custom package, bump its version so jobs pick up the new file. Serverless does not guarantee the CPU type (x86_64 or aarch64), so prefer pure-Python wheels or include both variants. Check the docs for pipeline-specific dependency rules; this walkthrough does not cover them.
Try it: In a serverless notebook, open the Environment pane, add one PyPI package, apply it, and read the pip logs.
Pinning every package to an old version and retrying hides the cause. It also does nothing when the real fault is a stale cached wheel with the same version number or a package that drags in PySpark.
Runner-up wins when: Installing with a notebook pip command wins for a quick one-off check in an interactive notebook, but it is not repeatable in a deployed job.
1.3 Develop User-Defined Functions (UDFs) using Pandas, Python, and SQL, including Unity Catalog functions for the given constraints.A user-defined function (UDF) is your own code that runs inside a query. A plain Python UDF handles one row at a time. A pandas UDF is vectorized: it receives whole batches as pandas Series, moved with Apache Arrow, so it is usually much faster. A SQL UDF is written in SQL and is the simplest. A Unity Catalog function is registered in a catalog and schema, so it is governed with permissions and reusable by name from anywhere.A function must be shared by many teams and run by name in SQL, with access controlled by grants. Or a function does a heavy numeric operation over millions of rows and speed is the deciding fact.For sharing and governance, create the function with CREATE FUNCTION in Unity Catalog; callers need EXECUTE on the function plus USAGE on its schema and catalog. For speed over many rows, use a pandas UDF with the pandas_udf decorator and type hints, because it works on batches. Use an iterator pandas UDF when setup such as loading a model should happen once. Prefer a SQL UDF or built-in functions when the logic can be written in SQL.
Try it: Create a SQL function in your own schema, grant EXECUTE to another user or group in a test, and call it from a query.
A notebook-scoped PySpark UDF is not governed or shareable by name, and a row-at-a-time Python UDF is slow on large data. Using one when the stem says shared or fast fails the constraint.
Runner-up wins when: A Unity Catalog Python UDF wins over a pandas UDF when the question stresses governance and sharing more than raw speed.
1.4 Create production-ready data pipelines for streaming data using Lakeflow Pipelines and Auto Loader.Lakeflow Pipelines is the managed engine where you declare tables and it handles ordering, retries and checkpoints. Auto Loader is the file source called cloudFiles: it finds new files in cloud storage or a volume and processes each one once. Together they give a production streaming ingest with very little code.Files land continuously in cloud storage and must be loaded to a table once each, even after restarts, with schema changes handled. The deciding fact is that the source is arriving files.Read the folder with Auto Loader inside a pipeline and write to a streaming table. Auto Loader keeps its list of seen files in the checkpoint, which gives exactly-once file processing. Inside a pipeline, the pipeline manages the schema and checkpoint locations for you. Directory listing is the default detection mode; file notification mode is recommended for most workloads because it can cut listing cost.
Try it: Create a pipeline over a volume folder, drop two JSON files in at different times, and watch each one load once.
A scheduled batch read of the whole folder reprocesses old files or needs your own bookkeeping of what was loaded, which is the work Auto Loader already does.
Runner-up wins when: Reading a Kafka topic directly wins when the source is a message bus rather than files.
1.5 Create and automate ETL workloads using Lakeflow Jobs via the UI, API, and CLI.Lakeflow Jobs runs and schedules your work as a set of tasks with dependencies. You can build a job in the UI, call the Jobs REST API, or use the Databricks CLI. The same job definition can also live in a bundle so it is deployed as code.A job exists in the UI and the team wants the same job created by a script in each environment, with a run triggered afterwards. The deciding fact is that it must be repeatable without clicking.Describe the job as JSON or in a bundle, create it with databricks jobs create --json (inline or @file), and trigger it with databricks jobs run-now. The jobs reset command overwrites all settings of a job, so use update for a partial change. The submit command makes a one-time run that is not saved as a job and does not retry on failure.
Try it: Create a one-task job in the UI, then read its JSON view and recreate it with the CLI.
Using jobs submit for a daily load looks quick, but those runs are not saved as a job, do not appear in the UI as a job and do not retry, so scheduling and monitoring are lost.
Runner-up wins when: The UI wins for a one-off job you will build once and never repeat.
1.6 Choose between a streaming table and a materialized view for a given latency, cost, and refresh requirement.A streaming table processes only new rows from an append-only source and keeps what it already wrote. A materialized view is a stored query result that the pipeline keeps up to date, either by updating only what changed or by recomputing, and it is defined by its query so it can handle updates and deletes in its inputs.A table must be the exact result of a query, with grouping across history, over data whose rows get corrected. Or a table must grow cheaply with new events and never reread old ones.Choose a streaming table for append-only sources, early bronze ingestion and simple transformations, where each update reads only new input. Choose a materialized view for gold tables with complex aggregations, or when the source has updates, such as an AUTO CDC target. Incremental refresh of a materialized view needs serverless compute; on classic compute it is fully recomputed on each update. Views refresh at a set interval and update in seconds to minutes, so they do not suit very low latency.
Try it: In a pipeline, define one streaming table and one materialized view over it, then insert a row and compare the two updates.
Using a streaming table over a source that changes rows breaks, because streaming tables expect appends only. Using a materialized view for plain event ingestion can pay for recomputation you do not need.
Runner-up wins when: A materialized view wins for a simple append feed too, when the team wants the result always defined by its query rather than by what was streamed.
1.7 Implement CDC pipelines using AUTO CDC APIs (formerly APPLY CHANGES) in Lakeflow Pipelines, including SCD Type 1 and Type 2 via stored_as_scd_type.CDC means change data capture: a feed of inserts, updates and deletes from a source system. The AUTO CDC feature in Lakeflow Pipelines applies that feed to a target table for you. Setting stored_as_scd_type to 1 keeps only the latest row per key, and setting it to 2 keeps every version as history.A feed of customer changes arrives with a key, an operation column and a sequence number, and the business wants to see what an address was last March. The deciding facts are whether history is needed and which column orders the changes.Use create_auto_cdc_flow in Python (the docs name; it replaces apply_changes and has the same syntax) with keys, sequence_by, apply_as_deletes for the delete marker, and stored_as_scd_type set to 2. Type 2 adds __START_AT and __END_AT columns, and the current row has __END_AT null. Late rows are placed by sequence_by, not by arrival order.
Try it: Write a tiny pipeline with a five-row change feed, run it with type 1, then switch to type 2 and compare the target tables.
Choosing type 1 because it is simpler: it overwrites the old value, so last March's address is gone. Another trap is hand-writing MERGE logic and ordering by arrival time, which breaks when changes arrive out of order.
Runner-up wins when: Type 1 wins when the business only needs the current state of each key and wants the smallest table.
1.8 Choose between Spark Structured Streaming and Apache Spark™ Declarative Pipelines for scalable ETL given operational constraints.Spark Structured Streaming is the Spark API where you write and run each streaming query yourself. Spark Declarative Pipelines (Lakeflow Pipelines on Databricks) is a layer on top where you declare tables and the engine works out order, parallelism and retries.A small team must keep dozens of dependent streaming tables running and does not want to manage run order or retries by hand. Or the opposite: a team needs a custom per-batch routine that the declarative model cannot express.When many tables depend on each other and the team wants less operational work, choose pipelines: the docs say it analyzes dependencies, orders and parallelizes runs, and retries transient failures at task, flow and pipeline level. Raw Structured Streaming fits when you need direct control over the query itself.Assuming raw Structured Streaming is always the more professional choice. It means you own orchestration, retries and checkpoint locations, which costs a small team real effort.
Runner-up wins when: Structured Streaming wins when the logic needs low-level control, such as a custom stateful operator, that declarative flows do not offer.
1.9 Create Lakeflow Jobs that use control-flow operators such as If/Else conditions and For Each loops.Lakeflow Jobs can branch and loop. An If/else condition task compares two values and sends the run down a true path or a false path. A For each task runs one nested task once per item in a list.A job must run a cleanup step only when an upstream task reports more than zero bad records, or must run the same notebook once for each region in a list.Use an If/else condition task with the expression built from a task value, such as the bad record count, and make the cleanup depend on the true outcome. For lists, use a For each task whose inputs come from a task value or job parameter; the current item is {{input}}, and the Concurrency field sets how many iterations run at once (the default is 1).
Try it: Build a two-task job: one task sets a task value, an If/else reads it, and a For each loops over a three-item array.
Using run-if dependencies for a value test. Those react to upstream success or failure, not to a number. Also, the equals and not-equals operators compare as text, so 12.0 == 12 is false, while greater-than and less-than compare numbers.
Runner-up wins when: A run-if dependency wins when the decider is whether an upstream task failed, not what value it produced.
1.10 Choose appropriate compute and configuration for environments and dependencies — including serverless compute (serverless environments, dependency management, performance mode), high-memory notebook tasks, and auto-optimization settings (e.g., disallowing retries).Serverless compute needs no cluster setup, but you still choose an environment version and dependencies, a performance mode, a memory size, and whether serverless auto-optimization may add retries.A job task must not run twice, or a notebook task runs out of memory in the notebook process, or the owner wants a cheaper run that can wait a few minutes to start.For work that must execute at most once, switch off serverless auto-optimization in the task's retry policy, because it is on by default and can add retries. Standard performance mode costs less and suits jobs that tolerate a 4 to 6 minute start. High memory (32 GB instead of 16 GB, Public Preview, notebook tasks only) enlarges the notebook process, not the Spark session.Setting retries to zero and assuming that is enough. Auto-optimization is a separate switch that may add retries. Another trap is using high memory for a Spark executor problem, since it does not change Spark memory.
Runner-up wins when: Standard performance mode wins when cost matters most and a slower start is acceptable.
1.11 Develop unit and integration tests using assertDataFrameEqual, assertSchemaEqual, DataFrame.transform, and testing frameworks to ensure code correctness.A unit test checks one function on a tiny made-up input. PySpark ships assertDataFrameEqual and assertSchemaEqual in pyspark.testing to compare results, and DataFrame.transform lets you chain functions that take a DataFrame and return one, which makes them easy to test alone.A team has a cleaning step buried in a long notebook and wants fast tests that run in any CI pipeline without touching production data.Move the logic into a function from DataFrame to DataFrame, apply it with transform, and test it with a small in-memory input and an expected DataFrame compared by assertDataFrameEqual, plus assertSchemaEqual for column names and types. Run the tests with a framework such as pytest. Check the docs for the runtime version that includes these helpers.
Try it: Write one pytest test with two input rows, an expected DataFrame, and assertDataFrameEqual in a Databricks notebook.
Comparing collect() lists or only row counts. Counts pass when values are wrong, and list comparison breaks on row order or float rounding that the assert helpers handle.
Runner-up wins when: An integration test against a real table wins when the risk is in how systems connect, not in the function logic.
1.12 Apply Structured Streaming stateful-processing semantics, including watermarks, output modes, foreachBatch, and checkpoints for fault-tolerant exactly-once state recovery.Stateful streaming keeps memory between batches, for example running counts per window. A watermark says how late data may arrive before old state is dropped. The output mode says what is written each batch. The checkpoint stores progress and state so a restart resumes where it stopped. foreachBatch lets you run your own code on each micro-batch.A windowed aggregation must emit each window once, or a foreachBatch writer must not duplicate rows when a failed batch is replayed.Append mode with a watermark emits a window only after the watermark passes it, so output is delayed by the lateness threshold. Update mode emits as results change, and complete mode keeps all state and rewrites the target every trigger. foreachBatch alone gives at-least-once writes; for Delta, set txnAppId and bind txnVersion to batchId so a replayed batch is skipped. Each query needs its own checkpoint.Believing the checkpoint alone makes a foreachBatch write exactly once. A batch can be replayed after a failure, so the sink write must be idempotent. Also, changing stateful operations or state schema needs a new checkpoint.
Runner-up wins when: Update mode wins when downstream readers want early results that are later corrected, and the sink can take updates.

Mind map: Streaming table or materialized view

Streaming tables are not simply faster materialized views; they make a different promise about the source.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Source only ever appends new events and old rows never changeStreaming tableIngest with a streaming table so each update reads only new rows
Source rows are updated or deleted, or the result needs grouping over all historyMaterialized viewDefine the table by its query so changes flow through
Materialized view on classic computeFull recompute on every updateUse serverless for incremental refresh; a streaming table wins when the data is append-only and cost matters
Target of AUTO CDC feeding a downstream tableUpdates, not appendsDownstream must be a materialized view, because a streaming table cannot read updates

Mind map: Which kind of function

A UDF is not just a UDF; where it is registered and how it receives data decide governance and speed.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Many teams must call one function by name under grantsUnity Catalog functionCREATE FUNCTION and grant EXECUTE
Millions of rows and the logic is numeric PythonPandas UDFUse pandas_udf so work happens on batches through Arrow
Logic fits a SQL expressionSQL UDF or built-insSkip Python entirely; no separate Python environment
Setup such as model loading is costlyIterator pandas UDFLoad once per task, then loop over batches; a plain pandas UDF wins when there is no setup

Mind map: Where does a stream write go wrong

The wrong assumption is that a checkpoint makes every write exactly once.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Window result appears only after a delayAppend mode with a watermarkExpected; use update mode if early, correctable results are acceptable
Duplicate rows after a failed foreachBatch restartAt-least-once replay of a batchSet txnAppId and txnVersion equal to batchId on Delta writes; use your own idempotency for other sinks
Error after changing an aggregationState schema no longer matches the checkpointStart a new checkpoint, and change txnAppId with it
Records far past the watermarkState dropped; handling is not guaranteedWiden the watermark if late data matters more than state size

Mind map: Choosing the Lakeflow Jobs control and compute setting

The wrong assumption is that one retry setting controls every retry on serverless.

What you observeWhat it points toWhat to do (and when the runner-up wins)
Run a step only when a value exceeds a numberIf/else condition task on a task valueMake the step depend on the true outcome
Run a step only when something upstream failedRun-if dependencyUse run-if, because no value test is needed
Same notebook for each item in a listFor each taskPass the list as inputs and set Concurrency; the default is 1
Task must run at most onceServerless auto-optimization may add retriesUncheck auto-optimization in the retry policy
Cheaper run, slow start is fineStandard performance modeUse it; performance optimized is the default when speed matters