Home › Paths › Data Engineering

Learning path

Data Engineering

You move, clean and shape data so other people can trust the numbers.

Why this path

Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.

How to use this page: 1. learn the key concepts, which are the same everywhere; 2. pick your platform; 3. work through Crawl, Walk and Run.

Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.

Step 1 · the same everywhere

Key concepts

Learn these ideas first. They are the same on every cloud and with every tool; only the product names change. Each one is a short page: what it is, why it matters, an example and the common mistake.

Step 2

Where do you work?

Pick your platform and you get one page with that platform's products in every step.

Beginner · 8 weeks

Crawl: You can build one thing end to end, with help.

Land a dataset, clean it in layers, and show it on a dashboard.

Key ideas at this level, the same on every platform:

Open this level on: AWS · Azure · Google Cloud · Databricks · Open source

Learn: the concept, and the product on each platform

LearnAWSAzureGoogle CloudDatabricksOpen source
SQL, really welljoins, group by, window functions; read a query plan onceAmazon AthenaFabric SQL endpointBigQueryDatabricks SQLDuckDB
Files and storageCSV, JSON, Parquet; why columnar files are faster and cheaperAmazon S3ADLS Gen2 / OneLakeCloud StorageUnity Catalog volumes on cloud storageLocal disk or MinIO
Your first pipelineextract, load, transform; raw, clean and serving layersAWS DMS / AppFlow
AWS Glue / EMR (Spark)
Data Factory copy / Fabric mirroring
Fabric Spark
Datastream
Dataproc (Spark)
Lakeflow Connect / Auto Loader
Databricks jobs (Spark)
dlt or Airbyte
PySpark or Polars
Schedule itrun daily, retry on failure, alert when it breaksMWAA (Airflow) / Step FunctionsData Factory / Fabric pipelinesCloud Composer (Airflow)Lakeflow JobsAirflow or Dagster
Show itone dashboard a real person would useAmazon QuickSightPower BILookerAI/BI dashboardsSuperset or Metabase
Git from day oneevery change in a pull requestGitHubGitHub or Azure ReposGitHubGitHub + Git foldersGitHub

Build on GitHub

Under way · join

Ecom simple data platform

Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.

You are ready to move on when

Intermediate · 3 months

Walk: You can build it so it keeps working, without help.

Build pipelines that keep working: tested, modelled, governed and streaming where it matters.

Key ideas at this level, the same on every platform:

Open this level on: AWS · Azure · Google Cloud · Databricks · Open source

Learn: the concept, and the product on each platform

LearnAWSAzureGoogle CloudDatabricksOpen source
Lakehouse tablesopen table formats, upserts, time travel, partitioningApache Iceberg (S3 Tables)
Amazon Redshift
Delta Lake in OneLake
Fabric Warehouse
Iceberg on BigLake
BigQuery
Delta Lake
Databricks SQL warehouse
Delta Lake or Iceberg
DuckDB or Postgres
Modellingdimensional models (facts and dimensions), slowly changing dimensionsdbt on Redshift or Athenadbt on FabricDataform or dbt on BigQueryLakeflow Declarative Pipelines or dbtdbt with DuckDB
Quality and contractschecks at every layer; a contract between producer and consumerGlue Data QualityPurview data qualityDataplex data qualityPipeline expectationsGreat Expectations or Soda
Change data and streamscapture changes from apps; late and duplicate eventsAWS DMS / AppFlow
Kinesis / Amazon MSK
Data Factory copy / Fabric mirroring
Event Hubs / Fabric Eventstream
Datastream
Pub/Sub + Dataflow
Lakeflow Connect / Auto Loader
Structured Streaming
dlt or Airbyte
Kafka or Redpanda
Catalog and accessowners, grants, lineage, personal dataGlue Data Catalog + Lake Formation
Amazon DataZone lineage
Microsoft Purview
Purview lineage
Dataplex
Dataplex lineage
Unity Catalog
Unity Catalog lineage
OpenMetadata
OpenLineage + Marquez
Infrastructure as code and CIthe platform in files; tests in the pipelineTerraform or AWS CDK
GitHub Actions + CodeDeploy
Bicep or Terraform
GitHub Actions or Azure Pipelines
Terraform
GitHub Actions + Cloud Build
Declarative Automation Bundles (Asset Bundles)
GitHub Actions + bundles
Docker Compose
GitHub Actions

Build on GitHub

Proposed · needs a lead

CDC to streaming lakehouse

Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.

Proposed · needs a lead

Data contracts in CI

A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.

Under way · join

Ontology and knowledge graph

One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.

You are ready to move on when

Advanced · 6 months +

Run: You can design it, prove it and lead others through it.

Design a platform, prove its cost and correctness, and make data ready for AI.

Key ideas at this level, the same on every platform:

Open this level on: AWS · Azure · Google Cloud · Databricks · Open source

Learn: the concept, and the product on each platform

LearnAWSAzureGoogle CloudDatabricksOpen source
Cost engineeringwhat each job costs; skew, small files and shuffle; cut the billCost Explorer + CUR
AWS Glue / EMR (Spark)
Cost Management
Fabric Spark
Billing export to BigQuery
Dataproc (Spark)
Billing system tables
Databricks jobs (Spark)
Query plans and run times
PySpark or Polars
Semantic layermetrics defined once, used by dashboards and AI alikedbt semantic layerPower BI semantic modelsLooker (LookML)Metric viewsCube Core
Data for AIdocuments to chunks to vectors; a governed path to the modelOpenSearch Serverless / Aurora pgvector
Amazon Bedrock
Azure AI Search
Azure OpenAI in Microsoft Foundry
Vertex AI Vector Search / AlloyDB
Gemini on Vertex AI
Vector Search
Foundation Model APIs
pgvector or Qdrant
Ollama, or a free hosted open model
Platform specswhat must exist, what must be true, what it must surviveGlue Data Catalog + Lake Formation
Glue Data Quality
Microsoft Purview
Purview data quality
Dataplex
Dataplex data quality
Unity Catalog
Pipeline expectations
OpenMetadata
Great Expectations or Soda
Lead the designwrite the decision record; review others' pipelinesGitHubGitHub or Azure ReposGitHubGitHub + Git foldersGitHub

Build on GitHub

Under way · join

Spark cost agent

Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.

Needs a lead

Clean Data Engineering specs

Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.

Under way · join

Conversation UI on data

Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.

Just starting · join

AskSherlock

An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.

You are ready to move on when