Home › Paths › Data Engineering › Open source

Learning path · Open source

Data Engineering on Open source

You move, clean and shape data so other people can trust the numbers.

Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.

Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.

Every topic below is a concept (the job) with the Open source product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.

Beginner · 8 weeks

Crawl: You can build one thing end to end, with help.

Land a dataset, clean it in layers, and show it on a dashboard.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Open source you use
SQL, really welljoins, group by, window functions; read a query plan onceDuckDBsql engine
Files and storageCSV, JSON, Parquet; why columnar files are faster and cheaperLocal disk or MinIOobject storage
Your first pipelineextract, load, transform; raw, clean and serving layersdlt or Airbyteingestion
PySpark or Polarsbatch compute
Schedule itrun daily, retry on failure, alert when it breaksAirflow or Dagsterorchestration
Show itone dashboard a real person would useSuperset or Metabasedashboards
Git from day oneevery change in a pull requestGitHubgit and pull requests

Build on GitHub

Under way · join

Ecom simple data platform

Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.

Built on Azure, Databricks today. Porting it to Open source is a good first contribution.

You are ready to move on when

Intermediate · 3 months

Walk: You can build it so it keeps working, without help.

Build pipelines that keep working: tested, modelled, governed and streaming where it matters.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Open source you use
Lakehouse tablesopen table formats, upserts, time travel, partitioningDelta Lake or Icebergopen table format
DuckDB or Postgreswarehouse
Modellingdimensional models (facts and dimensions), slowly changing dimensionsdbt with DuckDBsql modelling
Quality and contractschecks at every layer; a contract between producer and consumerGreat Expectations or Sodadata quality
Change data and streamscapture changes from apps; late and duplicate eventsdlt or Airbyteingestion
Kafka or Redpandastreaming
Catalog and accessowners, grants, lineage, personal dataOpenMetadatacatalog and access
OpenLineage + Marquezlineage
Infrastructure as code and CIthe platform in files; tests in the pipelineDocker Composeinfrastructure as code
GitHub Actionsci/cd

Build on GitHub

Proposed · needs a lead

CDC to streaming lakehouse

Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.

Proposed · needs a lead

Data contracts in CI

A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.

Under way · join

Ontology and knowledge graph

One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.

Built on Open source already.

You are ready to move on when

Advanced · 6 months +

Run: You can design it, prove it and lead others through it.

Design a platform, prove its cost and correctness, and make data ready for AI.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Open source you use
Cost engineeringwhat each job costs; skew, small files and shuffle; cut the billQuery plans and run timescost
PySpark or Polarsbatch compute
Semantic layermetrics defined once, used by dashboards and AI alikeCube Coresemantic layer
Data for AIdocuments to chunks to vectors; a governed path to the modelpgvector or Qdrantvector search
Ollama, or a free hosted open modelmodel api
Platform specswhat must exist, what must be true, what it must surviveOpenMetadatacatalog and access
Great Expectations or Sodadata quality
Lead the designwrite the decision record; review others' pipelinesGitHubgit and pull requests

Build on GitHub

Under way · join

Spark cost agent

Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.

Built on Open source already.

Needs a lead

Clean Data Engineering specs

Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.

Built on Databricks today. Porting it to Open source is a good first contribution.

Under way · join

Conversation UI on data

Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.

Built on Open source already.

Just starting · join

AskSherlock

An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.

Built on Open source already.

You are ready to move on when

Learn together

Do it with others

Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.

All projects →   Another path →