Home › Paths › Data Engineering
Learning path
Data Engineering
You move, clean and shape data so other people can trust the numbers.
Why this path
Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.
How to use this page: 1. learn the key concepts, which are the same everywhere; 2. pick your platform; 3. work through Crawl, Walk and Run.
Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.
Step 1 · the same everywhere
Key concepts
Learn these ideas first. They are the same on every cloud and with every tool; only the product names change. Each one is a short page: what it is, why it matters, an example and the common mistake.
Moving data
Storing data
Shaping data
Trusting data
Running data
Step 2
Where do you work?
Pick your platform and you get one page with that platform's products in every step.
Beginner · 8 weeks
Crawl: You can build one thing end to end, with help.
Land a dataset, clean it in layers, and show it on a dashboard.
Key ideas at this level, the same on every platform:
Open this level on: AWS · Azure · Google Cloud · Databricks · Open source
Learn: the concept, and the product on each platform
| Learn | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| SQL, really welljoins, group by, window functions; read a query plan once | Amazon Athena | Fabric SQL endpoint | BigQuery | Databricks SQL | DuckDB |
| Files and storageCSV, JSON, Parquet; why columnar files are faster and cheaper | Amazon S3 | ADLS Gen2 / OneLake | Cloud Storage | Unity Catalog volumes on cloud storage | Local disk or MinIO |
| Your first pipelineextract, load, transform; raw, clean and serving layers | AWS DMS / AppFlow AWS Glue / EMR (Spark) | Data Factory copy / Fabric mirroring Fabric Spark | Datastream Dataproc (Spark) | Lakeflow Connect / Auto Loader Databricks jobs (Spark) | dlt or Airbyte PySpark or Polars |
| Schedule itrun daily, retry on failure, alert when it breaks | MWAA (Airflow) / Step Functions | Data Factory / Fabric pipelines | Cloud Composer (Airflow) | Lakeflow Jobs | Airflow or Dagster |
| Show itone dashboard a real person would use | Amazon QuickSight | Power BI | Looker | AI/BI dashboards | Superset or Metabase |
| Git from day oneevery change in a pull request | GitHub | GitHub or Azure Repos | GitHub | GitHub + Git folders | GitHub |
Build on GitHub
Ecom simple data platform
Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.
You are ready to move on when
- You can explain where every number on your dashboard came from.
- Your pipeline re-runs without creating duplicates.
- Someone else can run your project from the README alone.
Intermediate · 3 months
Walk: You can build it so it keeps working, without help.
Build pipelines that keep working: tested, modelled, governed and streaming where it matters.
Key ideas at this level, the same on every platform:
Open this level on: AWS · Azure · Google Cloud · Databricks · Open source
Learn: the concept, and the product on each platform
| Learn | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Lakehouse tablesopen table formats, upserts, time travel, partitioning | Apache Iceberg (S3 Tables) Amazon Redshift | Delta Lake in OneLake Fabric Warehouse | Iceberg on BigLake BigQuery | Delta Lake Databricks SQL warehouse | Delta Lake or Iceberg DuckDB or Postgres |
| Modellingdimensional models (facts and dimensions), slowly changing dimensions | dbt on Redshift or Athena | dbt on Fabric | Dataform or dbt on BigQuery | Lakeflow Declarative Pipelines or dbt | dbt with DuckDB |
| Quality and contractschecks at every layer; a contract between producer and consumer | Glue Data Quality | Purview data quality | Dataplex data quality | Pipeline expectations | Great Expectations or Soda |
| Change data and streamscapture changes from apps; late and duplicate events | AWS DMS / AppFlow Kinesis / Amazon MSK | Data Factory copy / Fabric mirroring Event Hubs / Fabric Eventstream | Datastream Pub/Sub + Dataflow | Lakeflow Connect / Auto Loader Structured Streaming | dlt or Airbyte Kafka or Redpanda |
| Catalog and accessowners, grants, lineage, personal data | Glue Data Catalog + Lake Formation Amazon DataZone lineage | Microsoft Purview Purview lineage | Dataplex Dataplex lineage | Unity Catalog Unity Catalog lineage | OpenMetadata OpenLineage + Marquez |
| Infrastructure as code and CIthe platform in files; tests in the pipeline | Terraform or AWS CDK GitHub Actions + CodeDeploy | Bicep or Terraform GitHub Actions or Azure Pipelines | Terraform GitHub Actions + Cloud Build | Declarative Automation Bundles (Asset Bundles) GitHub Actions + bundles | Docker Compose GitHub Actions |
Build on GitHub
CDC to streaming lakehouse
Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.
Data contracts in CI
A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.
Ontology and knowledge graph
One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.
You are ready to move on when
- A bad row is stopped by a check, not found by a user.
- A schema change in the source fails a test before it breaks a dashboard.
- You can rebuild the whole platform from code.
Advanced · 6 months +
Run: You can design it, prove it and lead others through it.
Design a platform, prove its cost and correctness, and make data ready for AI.
Key ideas at this level, the same on every platform:
Open this level on: AWS · Azure · Google Cloud · Databricks · Open source
Learn: the concept, and the product on each platform
| Learn | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Cost engineeringwhat each job costs; skew, small files and shuffle; cut the bill | Cost Explorer + CUR AWS Glue / EMR (Spark) | Cost Management Fabric Spark | Billing export to BigQuery Dataproc (Spark) | Billing system tables Databricks jobs (Spark) | Query plans and run times PySpark or Polars |
| Semantic layermetrics defined once, used by dashboards and AI alike | dbt semantic layer | Power BI semantic models | Looker (LookML) | Metric views | Cube Core |
| Data for AIdocuments to chunks to vectors; a governed path to the model | OpenSearch Serverless / Aurora pgvector Amazon Bedrock | Azure AI Search Azure OpenAI in Microsoft Foundry | Vertex AI Vector Search / AlloyDB Gemini on Vertex AI | Vector Search Foundation Model APIs | pgvector or Qdrant Ollama, or a free hosted open model |
| Platform specswhat must exist, what must be true, what it must survive | Glue Data Catalog + Lake Formation Glue Data Quality | Microsoft Purview Purview data quality | Dataplex Dataplex data quality | Unity Catalog Pipeline expectations | OpenMetadata Great Expectations or Soda |
| Lead the designwrite the decision record; review others' pipelines | GitHub | GitHub or Azure Repos | GitHub | GitHub + Git folders | GitHub |
Build on GitHub
Spark cost agent
Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.
Clean Data Engineering specs
Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.
Conversation UI on data
Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.
AskSherlock
An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.
You are ready to move on when
- You can show a cost before and after, and why it changed.
- A business user gets the same answer from the dashboard and from the AI assistant.
- You have reviewed and improved someone else's pipeline.