Home › Learn › Databricks interview prep › Declarative and semantic

Databricks interview prep · data engineering

What declarative means, and a semantic layer example

Declarative versus imperative in plain words, Spark Declarative Pipelines versus Databricks Lakeflow, and a worked Unity Catalog metric view.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Imperative: you write the steps,order, retries, checkpointsDeclarative: you write what eachtable should beThe engine builds the graph andruns it in order

Declarative is the same idea as SQL, applied to a whole pipeline of tables.

The idea

What declarative means

Imperative: you write the steps. Read this file, filter, write it there; read that table, merge it in; run the next job. You decide order, retries, checkpoints, when to run and what to do on failure.

Declarative: you write what the results should be. Table orders_clean is this query over orders_raw; table daily_revenue is this query over orders_clean. The engine works out the order, runs things in dependency order, processes incrementally, retries and keeps checkpoints.

Think of directions: imperative is turn-by-turn instructions to a driver; declarative is giving the address and letting the GPS choose and re-route. SQL is already declarative for one query; declarative pipelines apply it to many tables.

GainCost
The engine builds the dependency graph from your definitionsLess control over exact execution steps
Built-in incremental processing and checkpointsYou work inside the framework's rules
Retries and recovery handledDebugging happens at pipeline level
Less code, easier to readVery custom side effects do not fit

Parts

The building blocks (the same in both)

ConceptMeaning
Streaming tableUpdated incrementally from an append-style source; each record processed once
Materialized viewA table defined by a query, kept current with that query's result
Temporary viewAn intermediate step inside the pipeline, not published
FlowThe query that feeds a table (a table can have several flows)
Dataflow graphThe dependency graph built from all definitions
PipelineThe set of definitions that run and refresh together

The difference

Apache Spark Declarative Pipelines versus Databricks

Databricks contributed the core of Delta Live Tables to Apache Spark as Spark Declarative Pipelines (announced in 2025, in the Spark 4.x line; I believe 4.1). The Databricks product is Lakeflow Spark Declarative Pipelines (earlier Delta Live Tables). Roughly: open source is the language and basic engine; Databricks is the operated product. Check the current release notes for how much has moved upstream.

AreaOpen-source SparkDatabricks Lakeflow
What it isAuthoring model and execution framework inside SparkThe full managed product
Where it runsAny Spark cluster you runServerless or classic Databricks compute
OperationsYou run the cluster, retries, upgradesPlatform handles lifecycle, scaling, retries, upgrades
SchedulingBring your own (Airflow, cron)Lakeflow Jobs; triggered or continuous
GovernanceWhatever catalog you configureUnity Catalog: lineage, access control, auditing
Data qualityBasicExpectations: warn, drop or fail, with metrics
CDC / SCDWrite the merge yourself, unless added upstreamAUTO CDC / APPLY CHANGES for SCD1 and SCD2 with ordering
Materialized view refreshLimitedIncremental refresh of only what changed (serverless)
EngineOpen-source SparkPhoton and Databricks optimizations
IngestionSpark sourcesAuto Loader and Lakeflow Connect
MonitoringSpark UI and logsPipeline UI, event log tables, lineage graph
PortabilityHighTied to Databricks features
CostYour infrastructureDBUs for pipeline compute
from pyspark import pipelines as dp   # module name in open source; check your runtime

@dp.table
def orders_clean():
    return spark.readStream.table("orders_raw").where("amount > 0")

@dp.materialized_view
def daily_revenue():
    return spark.read.table("orders_clean").groupBy("order_date").sum("amount")

You do not write checkpoint paths, run order, write mode, triggers or retry loops. Anything that uses expectations, AUTO CDC, Unity Catalog options or Auto Loader will not run on plain Spark.

Choose

When declarative is right

Use declarative whenPrefer imperative when
Standard medallion ETLCustom logic and side effects such as calling APIs
You want incremental processing without managing stream stateYou need exact control of write and merge behaviour
Many interdependent tablesOne-off scripts
Quality rules belong in the pipelineRuntime constraints rule the framework out

Full treatment, costs and harness patterns: Declarative or not.

Semantic layer

Example: a Unity Catalog metric view

A metric view defines business measures once, so every dashboard, Genie space and notebook agrees on what revenue means.

CREATE VIEW main.sales.order_metrics
WITH METRICS LANGUAGE YAML AS $$
version: 0.1
source: main.sales.orders
joins:
  - name: customer
    source: main.sales.customers
    on: source.customer_id = customer.id
dimensions:
  - name: order_month
    expr: DATE_TRUNC('MONTH', order_date)
  - name: segment
    expr: customer.segment
measures:
  - name: total_revenue
    expr: SUM(amount)
  - name: avg_order_value
    expr: SUM(amount) / COUNT(DISTINCT order_id)
$$;

SELECT segment, MEASURE(total_revenue)
FROM main.sales.order_metrics
GROUP BY segment;

The point to make: avg_order_value is a ratio and cannot be averaged across rows. The measure is re-evaluated at whatever grain the query groups by, which plain views get wrong. It is governed by Unity Catalog, so row filters and lineage apply. Check the current YAML version and syntax in the docs. Full course: Metric views, taught from zero.

Practice

Questions to rehearse

  1. What does declarative mean, versus workflow tasks?
  2. Streaming table versus materialized view?
  3. What did Databricks contribute to open-source Spark and what stayed proprietary?
  4. What are expectations and what three actions can you take?
  5. How does AUTO CDC handle out-of-order records?
  6. Metric view versus a plain SQL view?
In an interview, say: Declarative means I state what each table is and the engine decides order, increments and retries. Open-source Spark gives the language and basic engine; Databricks adds governance, expectations, AUTO CDC, incremental refresh and managed compute.

FAQ

Questions people ask

Is Lakeflow the same as Delta Live Tables?

It is the renamed and extended successor. The core idea and much of the code are the same.

Why use a metric view?

So a measure like average order value is defined once and stays correct at any grouping.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site