Home › Data engineering › Declarative pipelines

Data engineering · guide

Declarative or not: Spark Declarative Pipelines, open source and on Databricks

Spark now has a declarative way to build pipelines, in open source and, with more features, on Databricks. It removes a lot of plumbing. It does not remove the hard business parts of a data platform, and on classic Databricks compute it costs more. This page says where it pays and where it does not.

In one paragraph. Declarative pipelines let you say what each table is and let the engine work out order, what is new and how to refresh. Open-source Apache Spark Declarative Pipelines (Spark 4.1 and later) gives the engine, the graph, checkpoints and dry-run; Databricks Lakeflow adds snapshot change capture, data quality expectations, an event log, continuous mode and serverless. Use it for bronze to silver and for simple aggregates. Keep a harness for gold facts: changed windows, keys as of the event, completeness and restatements. On serverless it costs the same as a job; on classic compute 1.3 to 2.4 times as much.

Lesson first

In plain words

A recipe can be a list of steps (chop, fry, stir, plate) or a description of the dish (a curry made of fried onions and tomatoes). A good cook given the description works out the steps. Declarative pipelines are the second kind: you describe each table, the engine does the steps.

you write the steps (imperative)read new rows since my bookmarktransformmerge into the targetsave the bookmarkrun table B after table Ayou declare the tables (declarative)table B = transform(table A)table C = summary of table Bthe engine works out:order, what is new, checkpoints,retries, full or incremental refresh

Imperative: you own every step. Declarative: you own the definitions, the engine owns the steps.

WordWhat it means here
Imperative pipelineYou write each step: read, transform, write, remember where you stopped, run the next table. A Spark job or notebook.
Declarative pipelineYou declare what each table is (a transformation of other tables); the engine decides order, what is new and how to refresh.
Apache Spark Declarative Pipelines (SDP)The open-source declarative engine in Apache Spark from version 4.1, donated by Databricks; Python module pyspark.pipelines and the spark-pipelines command.
Lakeflow Spark Declarative PipelinesDatabricks' product built on SDP (formerly Delta Live Tables), with production features on top.
Streaming tableA table fed by one or more streaming flows; each run reads only what is new, using checkpoints.
Materialized viewA query stored as a table, refreshed by the engine; one batch flow.
AUTO CDCApplies change records (insert, update, delete, out of order) to a table as SCD Type 1 or Type 2.
ExpectationA data quality rule on a table: keep, drop or fail rows that break it.

1. Two products

Open-source SDP vs Databricks Lakeflow

Databricks Lakeflow Spark Declarative PipelinesApache Spark Declarative Pipelines(open source, Spark 4.1 and later)tables, materialized views, views,the graph, dry-run, checkpointsAUTO CDC from snapshotsdata quality expectationsqueryable event logupdate flows, foreachBatch sinkscontinuous modeincremental refresh, serverless, catalog

Lakeflow is the open-source engine plus production features that exist only on Databricks.

FeatureApache Spark Declarative Pipelines (open source)Lakeflow Spark Declarative Pipelines (Databricks)
Streaming tables, materialized views, temporary viewsYesYes
Dependency graph from the code; parallel runsYesYes
dry-run: syntax, missing tables or columns, cycles, without dataYesYes (validate)
Checkpoints, refresh and full refresh per tableYesYes
SinksAppend-only, Python onlyPlus update flows and foreachBatch sinks
AUTO CDC from change records (SCD 1 and 2)Checked in Spark's source (13 Nov 2026): none in 4.1, SCD Type 1 only in 4.2, Type 1 and 2 in 4.3. Needs a target with row-level updates (Iceberg or similar). Databricks' comparison page still says Lakeflow onlyYes
AUTO CDC from full snapshotsNoYes
Data quality expectationsNoYes
Queryable event logNoYes
Continuous modeNoYes
Incremental refresh of materialized viewsNot stated in the guideYes, when possible
Where it runsAny Spark 4.x: a laptop, a cluster, other clouds' Spark servicesDatabricks only, on its runtime and serverless
Code portability-Databricks says transformation code stays portable to other SDP runtimes; Lakeflow-only features do not move

2. Cost

Do declarative and non-declarative cost the same?

On serverless, yes. On classic compute, no: Databricks charges a higher rate per unit of compute (DBU) for declarative pipelines, by edition. Rates below are per DBU, AWS, Premium tier, from our cost calculator's price file (checked 9 Nov 2026; check current prices before deciding).

ComputePer DBUMachinesCompared with jobs
Jobs, classic compute$0.15Plus the cloud machinesBaseline
Declarative pipelines Core, classic$0.20Plus the cloud machines1.3x jobs
Declarative pipelines Pro, classic (needed for AUTO CDC)$0.25Plus the cloud machines1.7x jobs
Declarative pipelines Advanced, classic (needed for expectations)$0.36Plus the cloud machines2.4x jobs
Serverless jobs and serverless pipelines$0.35Machines included; same rate for bothSame for declarative and not
Open-source SDP on any SparkNo licence chargeYou pay only for the Spark you run it on-
Worked example. A pipeline that uses 10 DBUs an hour for one hour a day, on classic compute: as a job $1.50 a day, as declarative Core $2.00, Pro $2.50, Advanced $3.60, plus the same machine cost each way. On serverless: $3.50 a day either way, machines included. The edition is per pipeline: pay for Advanced only where expectations are used.

3. The honest list

Where declarative helps, and where it does not

NeedWhyDeclarative
Many tables with dependenciesThe graph comes from the code; no hand-kept run orderHelps
Streaming or frequent incremental loadsCheckpoints and what-is-new handled by the engineHelps
Catching broken pipelines before they rundry-run finds missing columns and cycles with no dataHelps
SCD Type 1 or 2 from a change feedAUTO CDC handles order, duplicates and deletesHelps (Lakeflow; open source if confirmed)
Data quality rules attached to tablesExpectations with keep, drop or failHelps on Lakeflow only
Simple aggregates and summariesMaterialized views refreshed by the engineHelps
Logic that needs control of how it writesReplace only changed windows, merge by a key with custom rules, write to several tables in one stepDoes not help: you fight the engine
Business bookkeeping around a loadCompleteness per window, restatement records, correction plans, a row ledger, a run log in your own metadata storeDoes not help: not its job
Keys looked up as of an event, with late membersSurrogate keys, unknown members, re-keyingDoes not help
Seeing every intermediate stepDeclarative hides steps by design; the event log (Lakeflow) shows flows, not your step outputsPartly
Local tests on a laptop with another engine (DuckDB)SDP code needs Spark 4.xDoes not help; keep logic in plain functions
Cost on classic computeDeclarative rates are 1.3 to 2.4 times the jobs rateCosts more; same on serverless

4. Bronze to silver

Bronze to silver: the best fit

Silver is mostly the same steps for every table, often streaming, with history from change records: exactly what a declarative engine is good at. The cleaning stages themselves stay yours, as plain functions the engine calls.

Silver needDeclarative featureVerdict
Read new bronze rows incrementallyStreaming table over bronze, with checkpointsPursue (either version)
SCD Type 1 and 2 from change records, out of order, deletesAUTO CDCPursue with care: Lakeflow now; open source once tested on Iceberg
SCD Type 2 from full daily copiesAUTO CDC from snapshotsLakeflow only; own code elsewhere
Quality rules: quarantine or flag by severityExpectations (keep, drop, fail)Lakeflow only; a quarantine table with reasons is richer, keep your own stage
Standard cleaning stages (contract check, types, time zones, personal data, provenance)Plain DataFrame functions called inside a flowKeep your stages; let the engine run them
Mass-delete guard, source-versioned history, void a bad batch and rebuildNoKeep your own
Files landing in storageDatabricks file ingestion inside LakeflowBind per platform

SCD Type 2 in depth, AUTO CDC against your own MERGE: SCD Type 2: AUTO CDC or own PySpark.

5. Silver to gold

Silver to gold: help with the edges only

Gold's hard parts are business bookkeeping, which a declarative engine does not model. Use it for the graph and the aggregates; keep a harness for the facts. Why and how: Silver to gold.

Gold needDeclarative featureVerdict
Dimensions, facts and aggregates in dependency orderThe graphPursue with care: a job running the harness in order does this too
Simple aggregates and summariesMaterialized views (incremental on Lakeflow)Pursue with care: optional
Facts rebuilt by changed window with a lookbackNo: a materialized view is one batch flow over the whole tableKeep a harness
Surrogate keys as of the event, unknown members, re-keyingNoKeep a harness
Completeness from every owed feed, restatements, correction plans, row ledgerNoKeep a harness
Accumulating snapshots updated by keyAUTO CDC Type 1 on the fact grain can do itPursue with care

6. Together

Declarative and a harness, together

  1. Keep every transformation a plain, pure function that takes DataFrames and returns one, so it runs inside SDP, inside a harness, or in a test on a laptop.
  2. Let the declarative engine own the graph, checkpoints and simple refreshes.
  3. Let the harness own what the engine does not: windows, keys as of the event, completeness, restatements, the row ledger and the run log.
  4. On Databricks, bind Lakeflow's expectations and snapshot CDC where the team wants them; keep the rules declared in your own metadata so another platform can bind them differently.
  5. Run dry-run in continuous integration for every pipeline, declarative or not.

6b. Restrictions

What declarative restricts

RestrictionWhat happensWhat to do
The pipeline owns its tablesNo other job may write to a streaming table. A fix to one bad batch becomes a full refresh, or a correction fed through the change feed.Plan recovery around it
Removed or renamed definitions leave tables behindWe saw it twice: after a redeploy the new pipeline could not write the old tables, and renamed definitions left orphans.Drop them in teardown
Creation-time properties need a new tableFormat features set at creation are kept even by a full refresh.Decide the format before the first run
Source must be append-onlyA streaming read breaks if source rows are updated or deleted (unless told to skip them).Read bronze, not a rewritten table
Full refresh replays only what the source still holdsIf bronze has aged out, history cannot be rebuilt.Keep a baseline batch before data ages out
Logic or schema changes may need a full refreshCheckpoints belong to the query.Treat flow changes as releases
AUTO CDC's fixed shapeOne key set, an orderable sequence column, SCD Type 1 or 2 only, fixed __START_AT and __END_AT columns.Other history kinds stay in your own code
Format on DatabricksPipeline tables are Delta; to be read as Iceberg they need Iceberg v3 compatibility (UniForm), not native Iceberg.Datasets that must be native Iceberg stay on a job

6c. Tested

What we tested (13 and 14 Nov 2026)

A tiny change feed of order lines (insert, update, a late change with an older sequence, a delete) through a declarative flow that calls one of our own stage functions, with AUTO CDC writing SCD Type 2, as two triggered updates on Databricks Free Edition (serverless), then the same on a laptop with open-source Spark.

CheckResult
Our own stage function inside the flowPassed: dealer codes cleaned by our hygiene rule
SCD Type 2 versions in sequence orderPassed: versions 10, 20, 30
A late change with an older sequencePassed: kept as history, never current
A deletePassed: the key's version is closed
An update to a measurePassed: a new version opened
Readable as IcebergOnly with Iceberg v3 compatibility; v2 failed (needs column mapping, and deletion vectors off)
The same 8-case SCD Type 2 contract our own engine passesPassed: 8 of 8, compared by meaning (closed versus flagged deletes)
Our whole silver stage chain over real tyre bronze, six datasetsPassed: equal to our job engine's silver on every dataset (two SCD2, one SCD1, three append), 0 differing rows
Streaming read of Iceberg bronzePassed: a declarative pipeline stream-reads Unity Catalog managed Iceberg
Bad rows planted in bronzeQuarantined with their rule names; the test also exposed a gap in our own engine (contract value rules were not run), now fixed in both
Open-source SDP, same test on a laptopBlocked: SCD2 AUTO CDC needs Spark 4.3, and no Iceberg or Delta release runs on 4.3 yet

7. Decisions

Decisions with verdicts

DecisionChoiceVerdict
Bronze to silver on DatabricksLakeflow streaming tables and AUTO CDC around your own cleaning stagesPursue
Bronze to silver portable across cloudsOpen-source SDP with AUTO CDC on Iceberg, once testedPursue with care
Silver to goldA harness for facts and dimensions; SDP only for some aggregates, as an optionPursue
Data quality rulesDeclared once in your metadata; bound to expectations on Lakeflow, to your own stage elsewherePursue
Classic compute with Advanced edition for every pipelinePaying 2.4 times the jobs rate for features most tables do not useDrop: pick the edition per pipeline, or use serverless
All logic written in SDP decoratorsTies tests to Spark 4.x and the engineDrop: decorators wrap plain functions

8. Not settled yet

What is not settled yet

QuestionWhere it stands
Is AUTO CDC open source?Yes, newly: we checked Spark's source on 13 Nov 2026. Spark 4.1 has none, 4.2 has SCD Type 1 only, 4.3 has Type 1 and 2. Databricks' comparison page has not caught up.
Can open-source SCD Type 2 run today?Not on an open table format. We tried on a laptop: Iceberg's newest runtime (1.12) is built for Spark 4.0 and 4.1 and fails to load on 4.3, and Delta's newest (4.4.1) supports Spark up to 4.2. Until a format supports Spark 4.3, open-source declarative SCD Type 2 waits; on Databricks, Lakeflow does it today.
Does open-source SDP refresh materialized views incrementally?Its guide does not say. Test before relying on it for large aggregates.
Expectations in open source?Not in the guide today. Keep quality rules in your own stage.

FAQ

Questions people ask

Are DLT, Lakeflow Declarative Pipelines and Spark Declarative Pipelines the same thing?

One family. DLT (Delta Live Tables) is Databricks' original product; it was renamed Lakeflow Declarative Pipelines in 2025, and its core engine was given to Apache Spark as Spark Declarative Pipelines (Spark 4.1 and later). Lakeflow is that open-source engine plus Databricks-only features such as AUTO CDC from snapshots, expectations, the event log and serverless. Old code says import dlt, new code says from pyspark import pipelines as dp.

Should I use dlt-meta or write my own metadata-driven layer?

If you already have a metadata store, a validating loader and tested transformation functions, write the small layer yourself: ours is about 350 lines that turn each dataset's plan into flows and AUTO CDC settings. dlt-meta brings its own onboarding files and metadata tables, which would be a second source of truth beside yours; it suits a team starting from nothing. Use it as a feature checklist either way.

Why use a streaming table for a batch load?

Streaming describes how a streaming table reads, not how long it runs. In triggered mode each run reads only what is new since its checkpoint, writes it and stops: incremental batch, with no cluster left running. AUTO CDC can only write into a streaming table, because SCD history needs a target that remembers which changes it has applied.

Are jobs and declarative pipelines both serverless?

On Databricks both can run on serverless or classic compute; on serverless they cost the same rate per DBU with machines included, on classic declarative pipelines cost more by edition. Open-source Spark Declarative Pipelines is not a serverless product: it runs on whatever Spark you give it.

Can AUTO CDC be used for SCD Type 2 in batch loads?

Yes. AUTO CDC reads a stream of change records, but a pipeline run in triggered mode processes the changes available and stops, which is a batch run; for full daily copies Lakeflow has AUTO CDC from snapshots. The reasons not to use it are cost (the Pro edition on classic compute), portability and history kinds it does not cover, not batch.

What is the difference between Apache Spark Declarative Pipelines and Databricks Lakeflow Declarative Pipelines?

Apache Spark Declarative Pipelines is the open-source engine in Spark 4.1 and later: streaming tables, materialized views, the dependency graph, dry-run and checkpoints. Databricks Lakeflow is built on it and adds AUTO CDC from snapshots, data quality expectations, a queryable event log, update flows, continuous mode, incremental refresh and serverless compute.

Do declarative pipelines cost more than jobs on Databricks?

On classic compute, yes: in our price file (AWS, Premium, checked November 2026) jobs are $0.15 per DBU and declarative pipelines $0.20 (Core), $0.25 (Pro, needed for AUTO CDC) or $0.36 (Advanced, needed for expectations), plus machines in both cases. On serverless both are $0.35 per DBU with machines included. Open-source SDP has no licence charge.

When should I use declarative pipelines?

For many dependent tables, streaming and incremental loads, SCD from change feeds and simple aggregates. Not for logic that must control how it writes, such as replacing only changed windows, or for business bookkeeping like completeness, restatements and surrogate keys looked up as of an event.

Do declarative pipelines replace a gold harness?

No. They can run the graph and refresh aggregates, but they do not rebuild facts by changed window, look up surrogate keys as of the event, track completeness from every owed feed or record restatements. Keep a harness for those and keep logic in plain functions both can call.

Sources and related

Sources and where it connects

Apache Spark Declarative Pipelines programming guide · Databricks: Spark Declarative Pipelines vs Lakeflow · Databricks: Lakeflow pipeline concepts · Databricks: Lakeflow Declarative Pipelines pricing

Silver to gold · Gold data modelling · SCD Type 2 on Databricks · Hop 2: bronze to silver · All data engineering