Home › Learn › Databricks interview prep › Question bank
Databricks interview prep · data engineering
Hard problems and the question bank
The hardest Databricks data engineering scenarios with strong answers, a question bank by topic, and ten sentences to have ready for an interview.
Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.
A shape that works for almost every hard question.
Overview
Major challenges on Databricks and how to overcome them
| Challenge | Why it hurts | How to overcome it |
|---|---|---|
| Small files and bad layout | Slow scans, high request cost | Liquid clustering, compaction, Predictive Optimization, fix the writer |
| Data skew | One task holds up a stage | Drop junk keys, AQE, isolate or salt hot keys |
| Shuffle volume and spill | Slow, memory failures | Filter early, broadcast, aggregate first, right-size |
| Cost overruns | Idle and mis-typed compute | Policies, job or serverless compute, incremental loads, tags and budgets |
| Concurrent write conflicts | Failed commits | Disjoint data per writer, predicates in MERGE, row-level concurrency, retries |
| Streaming correctness | Duplicates, late data, state growth | Checkpoints, idempotent sinks, watermarks, bounded state |
| Governance at scale | Leaks, unclear ownership | Unity Catalog, row filters and masks, tags, system tables, contracts |
| Environment drift | Prod differs from dev | Asset Bundles, targets, service principals |
| Schema change | Downstream breakage | Column mapping, mergeSchema, expectations, versioned views, contracts |
| Driver memory | Whole application dies | No collect, small broadcasts, compact files, bigger driver last |
| Retrieval quality in AI features | Wrong answers | Evaluation set, chunking, hybrid search, tenant filters |
Overview
Advanced features to know
- Unity Catalog: governance, lineage, system tables, volumes, Delta Sharing, Lakehouse Federation.
- Delta: deletion vectors, liquid clustering, change data feed, column mapping, type widening, UniForm.
- Ingestion and pipelines: Auto Loader, Lakeflow Declarative Pipelines, Lakeflow Connect, AUTO CDC, Lakeflow Jobs.
- Engine: AQE, Photon, serverless, Predictive Optimization, materialized views.
- Semantic and AI: metric views, Genie, Vector Search, AI functions, Model Serving.
- Delivery: Asset Bundles, Databricks Connect.
- Details and status are on the AQE, Photon and PO page; verify what is generally available.
Scenarios
The hardest problems and strong answers
| Problem | Strong answer |
|---|---|
| Exactly-once streaming to Delta with late and duplicate data | Checkpoints plus an idempotent sink. Deduplicate on an event ID inside the watermark. Use foreachBatch with MERGE on the event ID so a replayed batch gives the same result. State the late-data decision: data later than the watermark is dropped, so decide whether to keep a side channel. |
| ConcurrentAppendException in parallel writers | Make writers touch disjoint data, put the partition predicate in the MERGE condition, use deletion vectors with row-level concurrency, retry with backoff. |
| CDC with out-of-order events and deletes (SCD2) | AUTO CDC with a sequence column. If hand-written: deduplicate each batch to the latest per key first, because MERGE fails when several source rows match one target row, and guard with the sequence number. |
| Right to erasure in a huge table with time travel | DELETE with deletion vectors, REORG … APPLY (PURGE), VACUUM past retention; cover clones, backups, derived tables and the change data feed. |
| 20 TB table, every query scans everything | Check stats coverage, then clustering keys against real filters from query history; move to liquid clustering, fix file sizes, add materialized views for top aggregations; verify with files pruned in the query profile. |
| Join skewed by a NULL or default key | Filter NULLs out and union back; for a real hot key, isolate it and broadcast or salt only that key. |
| Multi-tenant platform | Catalogs or schemas per tenant or domain, row filters or ABAC where tenants share tables, cluster policies with mandatory tags, system tables for chargeback, service principals for production jobs. |
| Schema evolution without breaking consumers | Additive changes with mergeSchema, column mapping for renames, Auto Loader rescued column, expectations to quarantine bad records, versioned views as the consumer interface. |
| Migrating to Unity Catalog | Inventory, upgrade managed and external tables, replace mounts with external locations and volumes, repoint jobs, convert ACLs to grants. |
| A job got slow | Start from evidence: stage time, task spread (skew), spill, input size, plan changes. Most causes are file layout, skew, shuffle volume, a bad join or a UDF. |
Rehearse
Question bank
Delta internals
- What is in _delta_log and what is a checkpoint? (JSON commits; a Parquet checkpoint every 10 commits so the log need not be replayed from the start.)
- How does Delta give ACID on object storage? (Atomic create of the next commit file plus optimistic concurrency.)
- How does time travel interact with VACUUM and log retention?
- Deletion vectors: benefit and cost?
Performance
- A MERGE into a 5 TB table takes three hours. Diagnose and fix.
- Name AQE's three main features.
- When does a broadcast join fail?
- Why do only the first 32 columns get statistics by default, and what follows for table design?
Streaming and ingestion
- Auto Loader versus COPY INTO versus readStream on files.
- What happens if you change a streaming query's logic and restart on the same checkpoint?
- What does a watermark drop and what does it keep in state?
Governance
- Managed versus external tables: what happens on DROP?
- Dedicated versus standard compute for security.
- How do you give an auditor lineage and access logs? (System tables, audit logs.)
Architecture
- Where do you enforce quality in the medallion layers and where not?
- Design a pipeline that can be rebuilt from scratch. (Immutable bronze, idempotent transforms, deterministic keys.)
- Serverless versus classic for jobs: cost, control, startup.
Remember
Ten sentences to have ready
- OPTIMIZE creates new files, VACUUM removes old ones, and neither changes your data.
- VACUUM shortens time travel and can break long-running readers.
- Liquid clustering replaces partitioning and Z-order for new tables.
- Predictive Optimization removes the maintenance job on managed tables.
- Row filters and masks are SQL functions attached to the table and enforced in the plan.
- Data skipping needs statistics and a layout that benefits from them.
- Skew shows as one slow task, not a slow stage average.
- A stream that does not need to be on all day should use availableNow.
- Idempotent writes plus checkpoints give exactly-once, not the engine alone.
- Measure first (Spark UI, query profile, system tables), then change one thing.
Related
Learn the base ideas first
Data engineering fundamentals · Declarative pipelines · SCD2: AUTO CDC or own PySpark · Streaming on AWS and Databricks · Spark performance: 12 problems · Certifications
FAQ
Questions people ask
How should I answer a question I am unsure about?
Say what you know, name what you would check in the docs or the query profile, and say how you would test it. That is a senior answer.
Which topics come up most for senior roles?
Delta maintenance and layout, skew and joins, streaming correctness, cost control, and governance.
Next
Where to go next
All Databricks interview topics · All data engineering · Databricks on this site