Home › Paths › Data Engineering › Open source
Learning path · Open source
Data Engineering on Open source
You move, clean and shape data so other people can trust the numbers.
Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.
Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.
Every topic below is a concept (the job) with the Open source product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.
Beginner · 8 weeks
Crawl: You can build one thing end to end, with help.
Land a dataset, clean it in layers, and show it on a dashboard.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Open source you use |
|---|---|---|
| SQL, really well | joins, group by, window functions; read a query plan once | DuckDBsql engine |
| Files and storage | CSV, JSON, Parquet; why columnar files are faster and cheaper | Local disk or MinIOobject storage |
| Your first pipeline | extract, load, transform; raw, clean and serving layers | dlt or Airbyteingestion PySpark or Polarsbatch compute |
| Schedule it | run daily, retry on failure, alert when it breaks | Airflow or Dagsterorchestration |
| Show it | one dashboard a real person would use | Superset or Metabasedashboards |
| Git from day one | every change in a pull request | GitHubgit and pull requests |
Build on GitHub
Ecom simple data platform
Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.
Built on Azure, Databricks today. Porting it to Open source is a good first contribution.
You are ready to move on when
- You can explain where every number on your dashboard came from.
- Your pipeline re-runs without creating duplicates.
- Someone else can run your project from the README alone.
Intermediate · 3 months
Walk: You can build it so it keeps working, without help.
Build pipelines that keep working: tested, modelled, governed and streaming where it matters.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Open source you use |
|---|---|---|
| Lakehouse tables | open table formats, upserts, time travel, partitioning | Delta Lake or Icebergopen table format DuckDB or Postgreswarehouse |
| Modelling | dimensional models (facts and dimensions), slowly changing dimensions | dbt with DuckDBsql modelling |
| Quality and contracts | checks at every layer; a contract between producer and consumer | Great Expectations or Sodadata quality |
| Change data and streams | capture changes from apps; late and duplicate events | dlt or Airbyteingestion Kafka or Redpandastreaming |
| Catalog and access | owners, grants, lineage, personal data | OpenMetadatacatalog and access OpenLineage + Marquezlineage |
| Infrastructure as code and CI | the platform in files; tests in the pipeline | Docker Composeinfrastructure as code GitHub Actionsci/cd |
Build on GitHub
CDC to streaming lakehouse
Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.
Data contracts in CI
A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.
Ontology and knowledge graph
One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.
Built on Open source already.
You are ready to move on when
- A bad row is stopped by a check, not found by a user.
- A schema change in the source fails a test before it breaks a dashboard.
- You can rebuild the whole platform from code.
Advanced · 6 months +
Run: You can design it, prove it and lead others through it.
Design a platform, prove its cost and correctness, and make data ready for AI.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Open source you use |
|---|---|---|
| Cost engineering | what each job costs; skew, small files and shuffle; cut the bill | Query plans and run timescost PySpark or Polarsbatch compute |
| Semantic layer | metrics defined once, used by dashboards and AI alike | Cube Coresemantic layer |
| Data for AI | documents to chunks to vectors; a governed path to the model | pgvector or Qdrantvector search Ollama, or a free hosted open modelmodel api |
| Platform specs | what must exist, what must be true, what it must survive | OpenMetadatacatalog and access Great Expectations or Sodadata quality |
| Lead the design | write the decision record; review others' pipelines | GitHubgit and pull requests |
Build on GitHub
Spark cost agent
Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.
Built on Open source already.
Clean Data Engineering specs
Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.
Built on Databricks today. Porting it to Open source is a good first contribution.
Conversation UI on data
Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.
Built on Open source already.
AskSherlock
An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.
Built on Open source already.
You are ready to move on when
- You can show a cost before and after, and why it changed.
- A business user gets the same answer from the dashboard and from the AI assistant.
- You have reviewed and improved someone else's pipeline.
Learn together
Do it with others
Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.