Home › Paths › Data Engineering › Azure
Learning path · Azure
Data Engineering on Azure
You move, clean and shape data so other people can trust the numbers.
Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.
Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.
Every topic below is a concept (the job) with the Azure product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.
Beginner · 8 weeks
Crawl: You can build one thing end to end, with help.
Land a dataset, clean it in layers, and show it on a dashboard.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Azure you use |
|---|---|---|
| SQL, really well | joins, group by, window functions; read a query plan once | Fabric SQL endpointsql engine |
| Files and storage | CSV, JSON, Parquet; why columnar files are faster and cheaper | ADLS Gen2 / OneLakeobject storage |
| Your first pipeline | extract, load, transform; raw, clean and serving layers | Data Factory copy / Fabric mirroringingestion Fabric Sparkbatch compute |
| Schedule it | run daily, retry on failure, alert when it breaks | Data Factory / Fabric pipelinesorchestration |
| Show it | one dashboard a real person would use | Power BIdashboards |
| Git from day one | every change in a pull request | GitHub or Azure Reposgit and pull requests |
Build on GitHub
Ecom simple data platform
Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.
Built on Azure already.
You are ready to move on when
- You can explain where every number on your dashboard came from.
- Your pipeline re-runs without creating duplicates.
- Someone else can run your project from the README alone.
Intermediate · 3 months
Walk: You can build it so it keeps working, without help.
Build pipelines that keep working: tested, modelled, governed and streaming where it matters.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Azure you use |
|---|---|---|
| Lakehouse tables | open table formats, upserts, time travel, partitioning | Delta Lake in OneLakeopen table format Fabric Warehousewarehouse |
| Modelling | dimensional models (facts and dimensions), slowly changing dimensions | dbt on Fabricsql modelling |
| Quality and contracts | checks at every layer; a contract between producer and consumer | Purview data qualitydata quality |
| Change data and streams | capture changes from apps; late and duplicate events | Data Factory copy / Fabric mirroringingestion Event Hubs / Fabric Eventstreamstreaming |
| Catalog and access | owners, grants, lineage, personal data | Microsoft Purviewcatalog and access Purview lineagelineage |
| Infrastructure as code and CI | the platform in files; tests in the pipeline | Bicep or Terraforminfrastructure as code GitHub Actions or Azure Pipelinesci/cd |
Build on GitHub
CDC to streaming lakehouse
Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.
Data contracts in CI
A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.
Ontology and knowledge graph
One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.
Built on Open source today. Porting it to Azure is a good first contribution.
You are ready to move on when
- A bad row is stopped by a check, not found by a user.
- A schema change in the source fails a test before it breaks a dashboard.
- You can rebuild the whole platform from code.
Advanced · 6 months +
Run: You can design it, prove it and lead others through it.
Design a platform, prove its cost and correctness, and make data ready for AI.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Azure you use |
|---|---|---|
| Cost engineering | what each job costs; skew, small files and shuffle; cut the bill | Cost Managementcost Fabric Sparkbatch compute |
| Semantic layer | metrics defined once, used by dashboards and AI alike | Power BI semantic modelssemantic layer |
| Data for AI | documents to chunks to vectors; a governed path to the model | Azure AI Searchvector search Azure OpenAI in Microsoft Foundrymodel api |
| Platform specs | what must exist, what must be true, what it must survive | Microsoft Purviewcatalog and access Purview data qualitydata quality |
| Lead the design | write the decision record; review others' pipelines | GitHub or Azure Reposgit and pull requests |
Build on GitHub
Spark cost agent
Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.
Built on Open source today. Porting it to Azure is a good first contribution.
Clean Data Engineering specs
Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.
Built on Databricks today. Porting it to Azure is a good first contribution.
Conversation UI on data
Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.
Built on Databricks, Open source today. Porting it to Azure is a good first contribution.
AskSherlock
An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.
Built on Open source today. Porting it to Azure is a good first contribution.
You are ready to move on when
- You can show a cost before and after, and why it changed.
- A business user gets the same answer from the dashboard and from the AI assistant.
- You have reviewed and improved someone else's pipeline.
Optional
Certifications on Azure
Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.
- Microsoft Fabric Data Engineer Associate
Free prep we have written: Certifications.
Learn together
Do it with others
Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.