Home › Learn › Databricks interview prep › Question bank

Databricks interview prep · data engineering

Hard problems and the question bank

The hardest Databricks data engineering scenarios with strong answers, a question bank by topic, and ten sentences to have ready for an interview.

Maintenance commandsRow-level securityCompute choiceJoinsSkew and shuffleAQE, Photon, POCost and performance leversDeclarative and semanticAsset BundlesLakeflow ConnectUniFormVector Searchdbt and ODCSEgress and ingressMaintenance in picturesClustering and compactionMaintenance strategySlow writesBackup and recoveryTaggingQuestion bank

Defaults, limits and preview status change often. Check the current Databricks docs before quoting a number.

Say the idea in onesentenceName the trade-offSay how you would measureitGive the failure you haveseen

A shape that works for almost every hard question.

Overview

Major challenges on Databricks and how to overcome them

ChallengeWhy it hurtsHow to overcome it
Small files and bad layoutSlow scans, high request costLiquid clustering, compaction, Predictive Optimization, fix the writer
Data skewOne task holds up a stageDrop junk keys, AQE, isolate or salt hot keys
Shuffle volume and spillSlow, memory failuresFilter early, broadcast, aggregate first, right-size
Cost overrunsIdle and mis-typed computePolicies, job or serverless compute, incremental loads, tags and budgets
Concurrent write conflictsFailed commitsDisjoint data per writer, predicates in MERGE, row-level concurrency, retries
Streaming correctnessDuplicates, late data, state growthCheckpoints, idempotent sinks, watermarks, bounded state
Governance at scaleLeaks, unclear ownershipUnity Catalog, row filters and masks, tags, system tables, contracts
Environment driftProd differs from devAsset Bundles, targets, service principals
Schema changeDownstream breakageColumn mapping, mergeSchema, expectations, versioned views, contracts
Driver memoryWhole application diesNo collect, small broadcasts, compact files, bigger driver last
Retrieval quality in AI featuresWrong answersEvaluation set, chunking, hybrid search, tenant filters

Overview

Advanced features to know

Scenarios

The hardest problems and strong answers

ProblemStrong answer
Exactly-once streaming to Delta with late and duplicate dataCheckpoints plus an idempotent sink. Deduplicate on an event ID inside the watermark. Use foreachBatch with MERGE on the event ID so a replayed batch gives the same result. State the late-data decision: data later than the watermark is dropped, so decide whether to keep a side channel.
ConcurrentAppendException in parallel writersMake writers touch disjoint data, put the partition predicate in the MERGE condition, use deletion vectors with row-level concurrency, retry with backoff.
CDC with out-of-order events and deletes (SCD2)AUTO CDC with a sequence column. If hand-written: deduplicate each batch to the latest per key first, because MERGE fails when several source rows match one target row, and guard with the sequence number.
Right to erasure in a huge table with time travelDELETE with deletion vectors, REORG … APPLY (PURGE), VACUUM past retention; cover clones, backups, derived tables and the change data feed.
20 TB table, every query scans everythingCheck stats coverage, then clustering keys against real filters from query history; move to liquid clustering, fix file sizes, add materialized views for top aggregations; verify with files pruned in the query profile.
Join skewed by a NULL or default keyFilter NULLs out and union back; for a real hot key, isolate it and broadcast or salt only that key.
Multi-tenant platformCatalogs or schemas per tenant or domain, row filters or ABAC where tenants share tables, cluster policies with mandatory tags, system tables for chargeback, service principals for production jobs.
Schema evolution without breaking consumersAdditive changes with mergeSchema, column mapping for renames, Auto Loader rescued column, expectations to quarantine bad records, versioned views as the consumer interface.
Migrating to Unity CatalogInventory, upgrade managed and external tables, replace mounts with external locations and volumes, repoint jobs, convert ACLs to grants.
A job got slowStart from evidence: stage time, task spread (skew), spill, input size, plan changes. Most causes are file layout, skew, shuffle volume, a bad join or a UDF.

Rehearse

Question bank

Delta internals

  1. What is in _delta_log and what is a checkpoint? (JSON commits; a Parquet checkpoint every 10 commits so the log need not be replayed from the start.)
  2. How does Delta give ACID on object storage? (Atomic create of the next commit file plus optimistic concurrency.)
  3. How does time travel interact with VACUUM and log retention?
  4. Deletion vectors: benefit and cost?

Performance

  1. A MERGE into a 5 TB table takes three hours. Diagnose and fix.
  2. Name AQE's three main features.
  3. When does a broadcast join fail?
  4. Why do only the first 32 columns get statistics by default, and what follows for table design?

Streaming and ingestion

  1. Auto Loader versus COPY INTO versus readStream on files.
  2. What happens if you change a streaming query's logic and restart on the same checkpoint?
  3. What does a watermark drop and what does it keep in state?

Governance

  1. Managed versus external tables: what happens on DROP?
  2. Dedicated versus standard compute for security.
  3. How do you give an auditor lineage and access logs? (System tables, audit logs.)

Architecture

  1. Where do you enforce quality in the medallion layers and where not?
  2. Design a pipeline that can be rebuilt from scratch. (Immutable bronze, idempotent transforms, deterministic keys.)
  3. Serverless versus classic for jobs: cost, control, startup.

Remember

Ten sentences to have ready

  1. OPTIMIZE creates new files, VACUUM removes old ones, and neither changes your data.
  2. VACUUM shortens time travel and can break long-running readers.
  3. Liquid clustering replaces partitioning and Z-order for new tables.
  4. Predictive Optimization removes the maintenance job on managed tables.
  5. Row filters and masks are SQL functions attached to the table and enforced in the plan.
  6. Data skipping needs statistics and a layout that benefits from them.
  7. Skew shows as one slow task, not a slow stage average.
  8. A stream that does not need to be on all day should use availableNow.
  9. Idempotent writes plus checkpoints give exactly-once, not the engine alone.
  10. Measure first (Spark UI, query profile, system tables), then change one thing.

Related

Learn the base ideas first

Data engineering fundamentals · Declarative pipelines · SCD2: AUTO CDC or own PySpark · Streaming on AWS and Databricks · Spark performance: 12 problems · Certifications

FAQ

Questions people ask

How should I answer a question I am unsure about?

Say what you know, name what you would check in the docs or the query profile, and say how you would test it. That is a senior answer.

Which topics come up most for senior roles?

Delta maintenance and layout, skew and joins, streaming correctness, cost control, and governance.

Next

Where to go next

All Databricks interview topics · All data engineering · Databricks on this site