Home › Paths › Data Engineering › Databricks

Learning path · Databricks

Data Engineering on Databricks

You move, clean and shape data so other people can trust the numbers.

Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.

Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.

Every topic below is a concept (the job) with the Databricks product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.

Beginner · 8 weeks

Crawl: You can build one thing end to end, with help.

Land a dataset, clean it in layers, and show it on a dashboard.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Databricks you use
SQL, really welljoins, group by, window functions; read a query plan onceDatabricks SQLsql engine
Files and storageCSV, JSON, Parquet; why columnar files are faster and cheaperUnity Catalog volumes on cloud storageobject storage
Your first pipelineextract, load, transform; raw, clean and serving layersLakeflow Connect / Auto Loaderingestion
Databricks jobs (Spark)batch compute
Schedule itrun daily, retry on failure, alert when it breaksLakeflow Jobsorchestration
Show itone dashboard a real person would useAI/BI dashboardsdashboards
Git from day oneevery change in a pull requestGitHub + Git foldersgit and pull requests

Build on GitHub

Under way · join

Ecom simple data platform

Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.

Built on Databricks already.

You are ready to move on when

Intermediate · 3 months

Walk: You can build it so it keeps working, without help.

Build pipelines that keep working: tested, modelled, governed and streaming where it matters.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Databricks you use
Lakehouse tablesopen table formats, upserts, time travel, partitioningDelta Lakeopen table format
Databricks SQL warehousewarehouse
Modellingdimensional models (facts and dimensions), slowly changing dimensionsLakeflow Declarative Pipelines or dbtsql modelling
Quality and contractschecks at every layer; a contract between producer and consumerPipeline expectationsdata quality
Change data and streamscapture changes from apps; late and duplicate eventsLakeflow Connect / Auto Loaderingestion
Structured Streamingstreaming
Catalog and accessowners, grants, lineage, personal dataUnity Catalogcatalog and access
Unity Catalog lineagelineage
Infrastructure as code and CIthe platform in files; tests in the pipelineDeclarative Automation Bundles (Asset Bundles)infrastructure as code
GitHub Actions + bundlesci/cd

Build on GitHub

Proposed · needs a lead

CDC to streaming lakehouse

Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.

Proposed · needs a lead

Data contracts in CI

A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.

Under way · join

Ontology and knowledge graph

One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.

Built on Open source today. Porting it to Databricks is a good first contribution.

You are ready to move on when

Advanced · 6 months +

Run: You can design it, prove it and lead others through it.

Design a platform, prove its cost and correctness, and make data ready for AI.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Databricks you use
Cost engineeringwhat each job costs; skew, small files and shuffle; cut the billBilling system tablescost
Databricks jobs (Spark)batch compute
Semantic layermetrics defined once, used by dashboards and AI alikeMetric viewssemantic layer
Data for AIdocuments to chunks to vectors; a governed path to the modelVector Searchvector search
Foundation Model APIsmodel api
Platform specswhat must exist, what must be true, what it must surviveUnity Catalogcatalog and access
Pipeline expectationsdata quality
Lead the designwrite the decision record; review others' pipelinesGitHub + Git foldersgit and pull requests

Build on GitHub

Under way · join

Spark cost agent

Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.

Built on Open source today. Porting it to Databricks is a good first contribution.

Needs a lead

Clean Data Engineering specs

Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.

Built on Databricks already.

Under way · join

Conversation UI on data

Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.

Built on Databricks already.

Just starting · join

AskSherlock

An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.

Built on Open source today. Porting it to Databricks is a good first contribution.

You are ready to move on when

Optional

Certifications on Databricks

Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.

Free prep we have written: Certifications.

Learn together

Do it with others

Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.

All projects →   Another path →