Home › Paths › Data Engineering › AWS

Learning path · AWS

Data Engineering on AWS

You move, clean and shape data so other people can trust the numbers.

Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.

Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.

Every topic below is a concept (the job) with the AWS product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.

Beginner · 8 weeks

Crawl: You can build one thing end to end, with help.

Land a dataset, clean it in layers, and show it on a dashboard.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn AWS you use
SQL, really welljoins, group by, window functions; read a query plan onceAmazon Athenasql engine
Files and storageCSV, JSON, Parquet; why columnar files are faster and cheaperAmazon S3object storage
Your first pipelineextract, load, transform; raw, clean and serving layersAWS DMS / AppFlowingestion
AWS Glue / EMR (Spark)batch compute
Schedule itrun daily, retry on failure, alert when it breaksMWAA (Airflow) / Step Functionsorchestration
Show itone dashboard a real person would useAmazon QuickSightdashboards
Git from day oneevery change in a pull requestGitHubgit and pull requests

Build on GitHub

Under way · join

Ecom simple data platform

Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.

Built on Azure, Databricks today. Porting it to AWS is a good first contribution.

You are ready to move on when

Intermediate · 3 months

Walk: You can build it so it keeps working, without help.

Build pipelines that keep working: tested, modelled, governed and streaming where it matters.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn AWS you use
Lakehouse tablesopen table formats, upserts, time travel, partitioningApache Iceberg (S3 Tables)open table format
Amazon Redshiftwarehouse
Modellingdimensional models (facts and dimensions), slowly changing dimensionsdbt on Redshift or Athenasql modelling
Quality and contractschecks at every layer; a contract between producer and consumerGlue Data Qualitydata quality
Change data and streamscapture changes from apps; late and duplicate eventsAWS DMS / AppFlowingestion
Kinesis / Amazon MSKstreaming
Catalog and accessowners, grants, lineage, personal dataGlue Data Catalog + Lake Formationcatalog and access
Amazon DataZone lineagelineage
Infrastructure as code and CIthe platform in files; tests in the pipelineTerraform or AWS CDKinfrastructure as code
GitHub Actions + CodeDeployci/cd

Build on GitHub

Proposed · needs a lead

CDC to streaming lakehouse

Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.

Proposed · needs a lead

Data contracts in CI

A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.

Under way · join

Ontology and knowledge graph

One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.

Built on Open source today. Porting it to AWS is a good first contribution.

You are ready to move on when

Advanced · 6 months +

Run: You can design it, prove it and lead others through it.

Design a platform, prove its cost and correctness, and make data ready for AI.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn AWS you use
Cost engineeringwhat each job costs; skew, small files and shuffle; cut the billCost Explorer + CURcost
AWS Glue / EMR (Spark)batch compute
Semantic layermetrics defined once, used by dashboards and AI alikedbt semantic layersemantic layer
Data for AIdocuments to chunks to vectors; a governed path to the modelOpenSearch Serverless / Aurora pgvectorvector search
Amazon Bedrockmodel api
Platform specswhat must exist, what must be true, what it must surviveGlue Data Catalog + Lake Formationcatalog and access
Glue Data Qualitydata quality
Lead the designwrite the decision record; review others' pipelinesGitHubgit and pull requests

Build on GitHub

Under way · join

Spark cost agent

Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.

Built on Open source today. Porting it to AWS is a good first contribution.

Needs a lead

Clean Data Engineering specs

Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.

Built on Databricks today. Porting it to AWS is a good first contribution.

Under way · join

Conversation UI on data

Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.

Built on Databricks, Open source today. Porting it to AWS is a good first contribution.

Just starting · join

AskSherlock

An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.

Built on Open source today. Porting it to AWS is a good first contribution.

You are ready to move on when

Optional

Certifications on AWS

Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.

Free prep we have written: Certifications.

Learn together

Do it with others

Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.

All projects →   Another path →