Home › Paths › Data Engineering › AWS
Learning path · AWS
Data Engineering on AWS
You move, clean and shape data so other people can trust the numbers.
Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.
Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.
Every topic below is a concept (the job) with the AWS product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.
Beginner · 8 weeks
Crawl: You can build one thing end to end, with help.
Land a dataset, clean it in layers, and show it on a dashboard.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On AWS you use |
|---|---|---|
| SQL, really well | joins, group by, window functions; read a query plan once | Amazon Athenasql engine |
| Files and storage | CSV, JSON, Parquet; why columnar files are faster and cheaper | Amazon S3object storage |
| Your first pipeline | extract, load, transform; raw, clean and serving layers | AWS DMS / AppFlowingestion AWS Glue / EMR (Spark)batch compute |
| Schedule it | run daily, retry on failure, alert when it breaks | MWAA (Airflow) / Step Functionsorchestration |
| Show it | one dashboard a real person would use | Amazon QuickSightdashboards |
| Git from day one | every change in a pull request | GitHubgit and pull requests |
Build on GitHub
Ecom simple data platform
Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.
Built on Azure, Databricks today. Porting it to AWS is a good first contribution.
You are ready to move on when
- You can explain where every number on your dashboard came from.
- Your pipeline re-runs without creating duplicates.
- Someone else can run your project from the README alone.
Intermediate · 3 months
Walk: You can build it so it keeps working, without help.
Build pipelines that keep working: tested, modelled, governed and streaming where it matters.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On AWS you use |
|---|---|---|
| Lakehouse tables | open table formats, upserts, time travel, partitioning | Apache Iceberg (S3 Tables)open table format Amazon Redshiftwarehouse |
| Modelling | dimensional models (facts and dimensions), slowly changing dimensions | dbt on Redshift or Athenasql modelling |
| Quality and contracts | checks at every layer; a contract between producer and consumer | Glue Data Qualitydata quality |
| Change data and streams | capture changes from apps; late and duplicate events | AWS DMS / AppFlowingestion Kinesis / Amazon MSKstreaming |
| Catalog and access | owners, grants, lineage, personal data | Glue Data Catalog + Lake Formationcatalog and access Amazon DataZone lineagelineage |
| Infrastructure as code and CI | the platform in files; tests in the pipeline | Terraform or AWS CDKinfrastructure as code GitHub Actions + CodeDeployci/cd |
Build on GitHub
CDC to streaming lakehouse
Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.
Data contracts in CI
A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.
Ontology and knowledge graph
One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.
Built on Open source today. Porting it to AWS is a good first contribution.
You are ready to move on when
- A bad row is stopped by a check, not found by a user.
- A schema change in the source fails a test before it breaks a dashboard.
- You can rebuild the whole platform from code.
Advanced · 6 months +
Run: You can design it, prove it and lead others through it.
Design a platform, prove its cost and correctness, and make data ready for AI.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On AWS you use |
|---|---|---|
| Cost engineering | what each job costs; skew, small files and shuffle; cut the bill | Cost Explorer + CURcost AWS Glue / EMR (Spark)batch compute |
| Semantic layer | metrics defined once, used by dashboards and AI alike | dbt semantic layersemantic layer |
| Data for AI | documents to chunks to vectors; a governed path to the model | OpenSearch Serverless / Aurora pgvectorvector search Amazon Bedrockmodel api |
| Platform specs | what must exist, what must be true, what it must survive | Glue Data Catalog + Lake Formationcatalog and access Glue Data Qualitydata quality |
| Lead the design | write the decision record; review others' pipelines | GitHubgit and pull requests |
Build on GitHub
Spark cost agent
Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.
Built on Open source today. Porting it to AWS is a good first contribution.
Clean Data Engineering specs
Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.
Built on Databricks today. Porting it to AWS is a good first contribution.
Conversation UI on data
Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.
Built on Databricks, Open source today. Porting it to AWS is a good first contribution.
AskSherlock
An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.
Built on Open source today. Porting it to AWS is a good first contribution.
You are ready to move on when
- You can show a cost before and after, and why it changed.
- A business user gets the same answer from the dashboard and from the AI assistant.
- You have reviewed and improved someone else's pipeline.
Optional
Certifications on AWS
Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.
- AWS Certified Data Engineer – Associate
Free prep we have written: Certifications.
Learn together
Do it with others
Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.