Home › Learn › Databricks interview prep › dbt and ODCS
Databricks interview prep · data engineering
dbt and the Open Data Contract Standard
What problems dbt solves and where it stops, how it compares with declarative pipelines, and what the Open Data Contract Standard solves.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
dbt is the T in ELT: tested, documented, version-controlled SQL models.
dbt
What problems it solves
dbt handles the transform step. Data is already loaded; dbt takes SELECT statements, works out their order, runs them on your engine, and adds tests, docs and version control. It does not extract, load or process data itself.
| Problem before dbt | What dbt does |
|---|---|
| Scattered scripts and procedures nobody can find | Every transformation is a SQL file in a repo, one model per file |
| Execution order hand-wired in a scheduler | {{ ref('orders') }} makes the dependency graph, so order is inferred |
| Hard-coded table names differ between dev and prod | Targets switch schema and database; same code everywhere |
| No review or history for SQL | Git: pull requests, blame, rollback |
| Quality checked by hand | Built-in tests (unique, not null, accepted values, relationships) plus custom SQL tests |
| Nobody knows what a table means | Docs next to code and a generated site with a lineage graph |
| Copy-pasted SQL | Macros (Jinja) and packages |
| Boilerplate to create, insert, merge, drop | You write only the SELECT; a materialization makes it a view, table, incremental model or snapshot |
| Full reloads of big tables | Incremental models (append, merge, insert overwrite) |
| Tracking changing dimensions | Snapshots implement SCD type 2 |
| No CI for data code | Run and test only changed models and dependents on a pull request |
| Metrics defined differently in each dashboard | Semantic layer (MetricFlow) defines them once |
Does not solve: ingestion, streaming, non-SQL transformation (Python models exist in some adapters), orchestration of everything else, performance tuning, platform governance, or a bad data design.
On Databricks
dbt-databricks adapter
- Runs SQL on a SQL warehouse or cluster; models become Delta tables and views in Unity Catalog with three-level names.
- Incremental strategies include merge, append, insert_overwrite and replace_where.
- Clustering, partitioning and table properties are set in model config; liquid clustering and materialized views or streaming tables as materializations are supported in recent adapter versions (verify).
- Run it as a task in Databricks Jobs.
Compare
dbt versus Lakeflow Declarative Pipelines
| dbt | Declarative pipelines | |
|---|---|---|
| Language | SQL with Jinja (Python models in some cases) | SQL and Python |
| Strength | Engineering workflow for SQL models, tests, docs, portable across warehouses | Streaming, incremental processing, expectations, CDC, managed compute |
| Streaming | Not its focus | First-class |
| Portability | Many engines | Databricks (core upstreamed to Spark) |
| Testing | Rich test framework and CI practice | Expectations inside the pipeline |
| Typical fit | Analytics engineering, gold models, multi-warehouse teams | Ingestion-heavy medallion pipelines, streaming, CDC |
Many teams use pipelines for ingestion and bronze to silver, and dbt for gold. Others standardize on one. Choose by team skills, streaming needs and portability.
ODCS
Open Data Contract Standard
ODCS is an open, vendor-neutral YAML specification for a data contract: an agreement between the team that produces a dataset and the teams that consume it. It is maintained by the Bitol project under the Linux Foundation and grew out of PayPal's template. Check the current spec version before quoting section names.
Most data breakage is communication failure between producers and consumers, not technology failure.
| Without a contract | What goes wrong |
|---|---|
| A source renames a column or changes a type | Downstream pipelines and dashboards break; you hear from angry users |
| A consumer assumes a column is never null | Silent wrong numbers |
| No owner named | Incidents take days to route |
| Freshness lives in someone's head | A daily table arrives at noon |
| Column meaning unclear (currency, tax) | Different teams compute different answers |
| Sensitive fields not marked | Privacy mistakes |
| Each consumer writes its own checks | Duplicate, inconsistent rules |
| Each platform has its own metadata format | Contracts cannot move between tools |
| Section | States |
|---|---|
| Fundamentals | Name, version, status, domain, description, purpose |
| Schema | Tables, columns, types, required, keys, descriptions, classification (such as PII), business terms |
| Data quality | Uniqueness, completeness, ranges, row counts, custom SQL checks, with severity |
| Service levels | Freshness, latency, availability, retention |
| Team and roles | Owner, steward, contacts, access |
| Servers | Where the data lives |
| Support and pricing | Channels, cost |
| Custom properties | Extensions |
apiVersion: v3.0.0
kind: DataContract
id: orders-contract
name: Orders
version: 1.2.0
status: active
domain: sales
schema:
- name: orders
properties:
- name: order_id
logicalType: string
required: true
unique: true
- name: customer_email
logicalType: string
classification: PII
slaProperties:
- property: freshness
value: 24
unit: h| Contract element | Where enforced on Databricks |
|---|---|
| Schema | Delta schema and constraints, Unity Catalog tables |
| Quality rules | Pipeline expectations, dbt tests, generated checks |
| Ownership, descriptions, tags | Unity Catalog comments and governed tags |
| Access | Grants, row filters, column masks |
| Freshness and SLAs | Jobs monitoring, data quality monitoring, alerts |
| Change management | CI in Git comparing contract versions |
It does not enforce anything by itself; value comes from CI checks and runtime validation. It also does not fix organizational problems. dbt model contracts are narrower: they enforce column names and types at build time. Start with widely used, high-value datasets. See also data contracts.
Practice
Questions to rehearse
- What does ref() do and why is it more than a table name?
- Table, view, incremental and ephemeral models: when each?
- What can go wrong in an incremental model? (Late data, missing unique key causing duplicates, schema changes.)
- How do snapshots do SCD2?
- How would you run dbt in CI so a pull request builds only what changed?
- dbt versus Lakeflow Declarative Pipelines: how do you choose?
- What breaking changes would you detect in a data contract automatically?
- What happens when a producer violates a contract at 2am?
FAQ
Questions people ask
Does dbt process the data?
No. It sends SQL to your engine, which does the work.
Is ODCS a tool?
No. It is a file format; tools read it to generate and run checks.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site