Home › Learn › Databricks interview prep › Declarative and semantic
Databricks interview prep · data engineering
What declarative means, and a semantic layer example
Declarative versus imperative in plain words, Spark Declarative Pipelines versus Databricks Lakeflow, and a worked Unity Catalog metric view.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
Declarative is the same idea as SQL, applied to a whole pipeline of tables.
The idea
What declarative means
Imperative: you write the steps. Read this file, filter, write it there; read that table, merge it in; run the next job. You decide order, retries, checkpoints, when to run and what to do on failure.
Declarative: you write what the results should be. Table orders_clean is this query over orders_raw; table daily_revenue is this query over orders_clean. The engine works out the order, runs things in dependency order, processes incrementally, retries and keeps checkpoints.
Think of directions: imperative is turn-by-turn instructions to a driver; declarative is giving the address and letting the GPS choose and re-route. SQL is already declarative for one query; declarative pipelines apply it to many tables.
| Gain | Cost |
|---|---|
| The engine builds the dependency graph from your definitions | Less control over exact execution steps |
| Built-in incremental processing and checkpoints | You work inside the framework's rules |
| Retries and recovery handled | Debugging happens at pipeline level |
| Less code, easier to read | Very custom side effects do not fit |
Parts
The building blocks (the same in both)
| Concept | Meaning |
|---|---|
| Streaming table | Updated incrementally from an append-style source; each record processed once |
| Materialized view | A table defined by a query, kept current with that query's result |
| Temporary view | An intermediate step inside the pipeline, not published |
| Flow | The query that feeds a table (a table can have several flows) |
| Dataflow graph | The dependency graph built from all definitions |
| Pipeline | The set of definitions that run and refresh together |
The difference
Apache Spark Declarative Pipelines versus Databricks
Databricks contributed the core of Delta Live Tables to Apache Spark as Spark Declarative Pipelines (announced in 2025, in the Spark 4.x line; I believe 4.1). The Databricks product is Lakeflow Spark Declarative Pipelines (earlier Delta Live Tables). Roughly: open source is the language and basic engine; Databricks is the operated product. Check the current release notes for how much has moved upstream.
| Area | Open-source Spark | Databricks Lakeflow |
|---|---|---|
| What it is | Authoring model and execution framework inside Spark | The full managed product |
| Where it runs | Any Spark cluster you run | Serverless or classic Databricks compute |
| Operations | You run the cluster, retries, upgrades | Platform handles lifecycle, scaling, retries, upgrades |
| Scheduling | Bring your own (Airflow, cron) | Lakeflow Jobs; triggered or continuous |
| Governance | Whatever catalog you configure | Unity Catalog: lineage, access control, auditing |
| Data quality | Basic | Expectations: warn, drop or fail, with metrics |
| CDC / SCD | Write the merge yourself, unless added upstream | AUTO CDC / APPLY CHANGES for SCD1 and SCD2 with ordering |
| Materialized view refresh | Limited | Incremental refresh of only what changed (serverless) |
| Engine | Open-source Spark | Photon and Databricks optimizations |
| Ingestion | Spark sources | Auto Loader and Lakeflow Connect |
| Monitoring | Spark UI and logs | Pipeline UI, event log tables, lineage graph |
| Portability | High | Tied to Databricks features |
| Cost | Your infrastructure | DBUs for pipeline compute |
from pyspark import pipelines as dp # module name in open source; check your runtime
@dp.table
def orders_clean():
return spark.readStream.table("orders_raw").where("amount > 0")
@dp.materialized_view
def daily_revenue():
return spark.read.table("orders_clean").groupBy("order_date").sum("amount")You do not write checkpoint paths, run order, write mode, triggers or retry loops. Anything that uses expectations, AUTO CDC, Unity Catalog options or Auto Loader will not run on plain Spark.
Choose
When declarative is right
| Use declarative when | Prefer imperative when |
|---|---|
| Standard medallion ETL | Custom logic and side effects such as calling APIs |
| You want incremental processing without managing stream state | You need exact control of write and merge behaviour |
| Many interdependent tables | One-off scripts |
| Quality rules belong in the pipeline | Runtime constraints rule the framework out |
Full treatment, costs and harness patterns: Declarative or not.
Semantic layer
Example: a Unity Catalog metric view
A metric view defines business measures once, so every dashboard, Genie space and notebook agrees on what revenue means.
CREATE VIEW main.sales.order_metrics
WITH METRICS LANGUAGE YAML AS $$
version: 0.1
source: main.sales.orders
joins:
- name: customer
source: main.sales.customers
on: source.customer_id = customer.id
dimensions:
- name: order_month
expr: DATE_TRUNC('MONTH', order_date)
- name: segment
expr: customer.segment
measures:
- name: total_revenue
expr: SUM(amount)
- name: avg_order_value
expr: SUM(amount) / COUNT(DISTINCT order_id)
$$;
SELECT segment, MEASURE(total_revenue)
FROM main.sales.order_metrics
GROUP BY segment;The point to make: avg_order_value is a ratio and cannot be averaged across rows. The measure is re-evaluated at whatever grain the query groups by, which plain views get wrong. It is governed by Unity Catalog, so row filters and lineage apply. Check the current YAML version and syntax in the docs. Full course: Metric views, taught from zero.
Practice
Questions to rehearse
- What does declarative mean, versus workflow tasks?
- Streaming table versus materialized view?
- What did Databricks contribute to open-source Spark and what stayed proprietary?
- What are expectations and what three actions can you take?
- How does AUTO CDC handle out-of-order records?
- Metric view versus a plain SQL view?
FAQ
Questions people ask
Is Lakeflow the same as Delta Live Tables?
It is the renamed and extended successor. The core idea and much of the code are the same.
Why use a metric view?
So a measure like average order value is defined once and stays correct at any grouping.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site