DataEngineering Training: Ecom Simple Data Platform
Learn data engineering by designing real projects; project 1 is a small e-commerce lakehouse on Databricks.
Active · join meData EngineeringForward-deployed ready
1 · Purpose
This training track teaches data engineering by designing real project architectures, not by memorising tools. Every project goes from requirements to a tool-free conceptual design, then to a physical design that names the technology. Project 1 builds a small e-commerce lakehouse: raw files become clean, historised tables, a star schema and dashboards. It is for friends and newcomers who want the thinking a working data engineer uses every day.
2 · Business benefits
Job-ready skillsLearners practise the write patterns (append, overwrite, merge, history) that real pipelines use daily.
Lower cloud spend while learningEverything runs on a free serverless edition with small sample data and nothing left running.
Better design habitsSeparating what a system must do from how well it must do it makes designs easier to review and cost.
Fewer production surprisesThirteen war-story pages show common failures, their cause and their fix before they happen at work.
3 · Status and how to join
Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one.
Good first tasks this week:
- Build one consumption view on the daily sales mart.
- Build the first dashboard: KPI tiles, a line chart and a bar chart.
- Add a freshness check to the data-quality notebook.
- Write a war-story page about a pipeline failure you have seen.
4 · Requirements
What it must do
- Read raw daily files without ever changing them.
- Build clean tables, one notebook per table, each with the right write pattern: append, overwrite, merge or full history.
- Add standard audit columns to every clean table.
- Build a star schema with a sales fact and customer, product, category and date dimensions.
- Build consumption views and one or two dashboards.
- Run the whole pipeline as one scheduled workflow, incrementally, using high-water marks.
- Record row counts and a run id per step in an audit table.
How well it must do it
- Cost: serverless compute and small data; nothing left running.
- Reliability: every notebook is safe to re-run without duplicates.
- Security: governed catalog with per-schema grants; no mounts and no account keys.
- Data quality: schema enforcement, null and duplicate checks, and exactly one current row per key in history tables.
- Maintainability: one notebook per table and a small config that drives table-specific behaviour.
5 · Architecture, very high level
6 · Tech stack, per platform
| Component | Databricks | Azure | AWS | Open stack |
|---|---|---|---|---|
| Sample data generator | PySpark notebook ✓ | — | — | — |
| Raw zone | Unity Catalog Volume with Parquet ✓ | ADLS Gen2 + Unity Catalog external location | Amazon S3 | MinIO or local disk with Parquet |
| Raw ingest | PySpark notebook, metadata and watermark driven ✓ | Azure Data Factory + Azure Databricks | AWS Glue | — |
| Clean layer | Delta tables with MERGE and SCD2 ✓ | Azure Databricks + Delta Lake | Databricks on AWS or Glue with Delta | Apache Spark + Delta Lake |
| Business layer | Delta tables ✓ | Azure Databricks + Delta Lake | Databricks on AWS | Apache Spark + Delta Lake |
| Serving and dashboards | Databricks SQL views and dashboards | Azure SQL Database / Power BI | Amazon QuickSight | Apache Superset |
| Orchestration and audit | Databricks Workflows / Asset Bundle ✓ | Azure Data Factory | Step Functions / MWAA | Apache Airflow |
| Governance | Unity Catalog ✓ | Unity Catalog | Unity Catalog or Lake Formation | — |
7 · What needs to be done
Training scaffold 3 of 3 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Explainers for the eight core technologies | done | M | technical writing |
| Project template and live portal page | done | S | documentation |
| Diagram-first theory pages, including Delta internals | done | M | data engineering |
Project 1 design 2 of 2 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Architecture: conceptual and physical | done | S | architecture |
| Design: write pattern per table, audit columns, star schema | done | M | data modelling |
Project 1 build 4 of 6 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Setup, sample data and raw ingest notebooks | done | M | PySpark |
| Clean customers (history), orders, products and order items | done | M | PySpark / Delta |
| Daily sales and customer lifetime value marts | done | M | Spark SQL |
| Light data-quality notebook | done | S | data quality |
| Consumption layer of views and features | todo | M | Spark SQL |
| Two dashboards on the business layer | todo | S | BI |
Next projects 0 of 1 done
| Task | Status | Size | Skill |
|---|---|---|---|
| Project 2 using Data Factory and Azure SQL Database | todo | L | Azure data engineering |
8 · Roles
| Role | How many | Hours / week | What they do |
|---|---|---|---|
| Learner | 4 | 3-5 | Builds the project notebooks and answers the level-up questions. |
| Mentor | 1 | 2-3 | Reviews designs and code and adds new projects one at a time. |
| Contributor | 1 | 1-2 | Writes new war-story pages or theory pages. |
9 · What you'll learn
- When to append, overwrite, merge or keep full history for a table.
- How to make an append load safe to re-run.
- How to join a fact to the right version of a historised dimension.
- Views versus materialised views: freshness against speed.
- Conceptual architecture first, tools second.
Level-up questions
- What breaks at ten times the data volume?
- Which dashboard tiles justify a materialised view?
- How would you replay one bad day without rebuilding everything?
- Where should data-quality checks stop the pipeline instead of just warning?