DataEngineering Training: Ecom Simple Data Platform

Learn data engineering by designing real projects; project 1 is a small e-commerce lakehouse on Databricks.

Active · join meData EngineeringForward-deployed ready

9 of 12 tasks done

1 · Purpose

This training track teaches data engineering by designing real project architectures, not by memorising tools. Every project goes from requirements to a tool-free conceptual design, then to a physical design that names the technology. Project 1 builds a small e-commerce lakehouse: raw files become clean, historised tables, a star schema and dashboards. It is for friends and newcomers who want the thinking a working data engineer uses every day.

For: People new to data engineering, and analysts or developers moving into it.

2 · Business benefits

Job-ready skillsLearners practise the write patterns (append, overwrite, merge, history) that real pipelines use daily.
Lower cloud spend while learningEverything runs on a free serverless edition with small sample data and nothing left running.
Better design habitsSeparating what a system must do from how well it must do it makes designs easier to review and cost.
Fewer production surprisesThirteen war-story pages show common failures, their cause and their fix before they happen at work.

3 · Status and how to join

Active · join me. Under way: pick a task below, or tell me what you'd like to learn and I'll suggest one. Lead: Basant Choudhary.

Good first tasks this week:

  • Build one consumption view on the daily sales mart.
  • Build the first dashboard: KPI tiles, a line chart and a bar chart.
  • Add a freshness check to the data-quality notebook.
  • Write a war-story page about a pipeline failure you have seen.

4 · Requirements

What it must do

How well it must do it

5 · Architecture, very high level

Sample data generatorCreates synthetic raw files,including a second customerRaw zoneKeeps every file received,as-is, for audit and replay.Raw ingestLoads only new files into rawtables, driven by metadata andClean layerCleans, conforms and historiseseach table with the right writeBusiness layerStar schema and marts such asdaily sales and customerServing and dashboardsViews and dashboards foranalysts.Orchestration and auditRuns the steps in order andlogs every run.GovernanceCatalogs, secures and separatesdev, test and prod.daily filesnew files onlyraw rowsclean, historised facts and mart
Components and what each is responsible for, with no tool names: the tools come in the next section.

Also: Orchestration and audit → Raw ingest: run in order · Governance → Clean layer: grants and lineage

6 · Tech stack, per platform

ComponentDatabricksAzureAWSOpen stack
Sample data generatorPySpark notebook ✓———
Raw zoneUnity Catalog Volume with Parquet ✓ADLS Gen2 + Unity Catalog external locationAmazon S3MinIO or local disk with Parquet
Raw ingestPySpark notebook, metadata and watermark driven ✓Azure Data Factory + Azure DatabricksAWS Glue—
Clean layerDelta tables with MERGE and SCD2 ✓Azure Databricks + Delta LakeDatabricks on AWS or Glue with DeltaApache Spark + Delta Lake
Business layerDelta tables ✓Azure Databricks + Delta LakeDatabricks on AWSApache Spark + Delta Lake
Serving and dashboardsDatabricks SQL views and dashboardsAzure SQL Database / Power BIAmazon QuickSightApache Superset
Orchestration and auditDatabricks Workflows / Asset Bundle ✓Azure Data FactoryStep Functions / MWAAApache Airflow
GovernanceUnity Catalog ✓Unity CatalogUnity Catalog or Lake Formation—

Bold with ✓ = built and running today. Everything else is the planned or possible binding: the components stay the same, only the tools change.

7 · What needs to be done

Training scaffold 3 of 3 done

TaskStatusSizeSkill
Explainers for the eight core technologiesdoneMtechnical writing
Project template and live portal pagedoneSdocumentation
Diagram-first theory pages, including Delta internalsdoneMdata engineering

Project 1 design 2 of 2 done

TaskStatusSizeSkill
Architecture: conceptual and physicaldoneSarchitecture
Design: write pattern per table, audit columns, star schemadoneMdata modelling

Project 1 build 4 of 6 done

TaskStatusSizeSkill
Setup, sample data and raw ingest notebooksdoneMPySpark
Clean customers (history), orders, products and order itemsdoneMPySpark / Delta
Daily sales and customer lifetime value martsdoneMSpark SQL
Light data-quality notebookdoneSdata quality
Consumption layer of views and featurestodoMSpark SQL
Two dashboards on the business layertodoSBI

Next projects 0 of 1 done

TaskStatusSizeSkill
Project 2 using Data Factory and Azure SQL DatabasetodoLAzure data engineering

Size: S = an evening or two, M = about a week part-time, L = a few weeks.

8 · Roles

RoleHow manyHours / weekWhat they do
Learner43-5Builds the project notebooks and answers the level-up questions.
Mentor12-3Reviews designs and code and adds new projects one at a time.
Contributor11-2Writes new war-story pages or theory pages.

9 · What you'll learn

Level-up questions

10 · Links