Home › Paths › Data Engineering › Databricks
Learning path · Databricks
Data Engineering on Databricks
You move, clean and shape data so other people can trust the numbers.
Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.
Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.
Every topic below is a concept (the job) with the Databricks product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.
Beginner · 8 weeks
Crawl: You can build one thing end to end, with help.
Land a dataset, clean it in layers, and show it on a dashboard.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Databricks you use |
|---|---|---|
| SQL, really well | joins, group by, window functions; read a query plan once | Databricks SQLsql engine |
| Files and storage | CSV, JSON, Parquet; why columnar files are faster and cheaper | Unity Catalog volumes on cloud storageobject storage |
| Your first pipeline | extract, load, transform; raw, clean and serving layers | Lakeflow Connect / Auto Loaderingestion Databricks jobs (Spark)batch compute |
| Schedule it | run daily, retry on failure, alert when it breaks | Lakeflow Jobsorchestration |
| Show it | one dashboard a real person would use | AI/BI dashboardsdashboards |
| Git from day one | every change in a pull request | GitHub + Git foldersgit and pull requests |
Build on GitHub
Ecom simple data platform
Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.
Built on Databricks already.
You are ready to move on when
- You can explain where every number on your dashboard came from.
- Your pipeline re-runs without creating duplicates.
- Someone else can run your project from the README alone.
Intermediate · 3 months
Walk: You can build it so it keeps working, without help.
Build pipelines that keep working: tested, modelled, governed and streaming where it matters.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Databricks you use |
|---|---|---|
| Lakehouse tables | open table formats, upserts, time travel, partitioning | Delta Lakeopen table format Databricks SQL warehousewarehouse |
| Modelling | dimensional models (facts and dimensions), slowly changing dimensions | Lakeflow Declarative Pipelines or dbtsql modelling |
| Quality and contracts | checks at every layer; a contract between producer and consumer | Pipeline expectationsdata quality |
| Change data and streams | capture changes from apps; late and duplicate events | Lakeflow Connect / Auto Loaderingestion Structured Streamingstreaming |
| Catalog and access | owners, grants, lineage, personal data | Unity Catalogcatalog and access Unity Catalog lineagelineage |
| Infrastructure as code and CI | the platform in files; tests in the pipeline | Declarative Automation Bundles (Asset Bundles)infrastructure as code GitHub Actions + bundlesci/cd |
Build on GitHub
CDC to streaming lakehouse
Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.
Data contracts in CI
A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.
Ontology and knowledge graph
One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.
Built on Open source today. Porting it to Databricks is a good first contribution.
You are ready to move on when
- A bad row is stopped by a check, not found by a user.
- A schema change in the source fails a test before it breaks a dashboard.
- You can rebuild the whole platform from code.
Advanced · 6 months +
Run: You can design it, prove it and lead others through it.
Design a platform, prove its cost and correctness, and make data ready for AI.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Databricks you use |
|---|---|---|
| Cost engineering | what each job costs; skew, small files and shuffle; cut the bill | Billing system tablescost Databricks jobs (Spark)batch compute |
| Semantic layer | metrics defined once, used by dashboards and AI alike | Metric viewssemantic layer |
| Data for AI | documents to chunks to vectors; a governed path to the model | Vector Searchvector search Foundation Model APIsmodel api |
| Platform specs | what must exist, what must be true, what it must survive | Unity Catalogcatalog and access Pipeline expectationsdata quality |
| Lead the design | write the decision record; review others' pipelines | GitHub + Git foldersgit and pull requests |
Build on GitHub
Spark cost agent
Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.
Built on Open source today. Porting it to Databricks is a good first contribution.
Clean Data Engineering specs
Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.
Built on Databricks already.
Conversation UI on data
Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.
Built on Databricks already.
AskSherlock
An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.
Built on Open source today. Porting it to Databricks is a good first contribution.
You are ready to move on when
- You can show a cost before and after, and why it changed.
- A business user gets the same answer from the dashboard and from the AI assistant.
- You have reviewed and improved someone else's pipeline.
Optional
Certifications on Databricks
Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.
- Databricks Data Engineer Associate, then Professional
Free prep we have written: Certifications.
Learn together
Do it with others
Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.