Home › Paths › Data Engineering › Google Cloud
Learning path · Google Cloud
Data Engineering on Google Cloud
You move, clean and shape data so other people can trust the numbers.
Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.
Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.
Every topic below is a concept (the job) with the Google Cloud product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.
Beginner · 8 weeks
Crawl: You can build one thing end to end, with help.
Land a dataset, clean it in layers, and show it on a dashboard.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Google Cloud you use |
|---|---|---|
| SQL, really well | joins, group by, window functions; read a query plan once | BigQuerysql engine |
| Files and storage | CSV, JSON, Parquet; why columnar files are faster and cheaper | Cloud Storageobject storage |
| Your first pipeline | extract, load, transform; raw, clean and serving layers | Datastreamingestion Dataproc (Spark)batch compute |
| Schedule it | run daily, retry on failure, alert when it breaks | Cloud Composer (Airflow)orchestration |
| Show it | one dashboard a real person would use | Lookerdashboards |
| Git from day one | every change in a pull request | GitHubgit and pull requests |
Build on GitHub
Ecom simple data platform
Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.
Built on Azure, Databricks today. Porting it to Google Cloud is a good first contribution.
You are ready to move on when
- You can explain where every number on your dashboard came from.
- Your pipeline re-runs without creating duplicates.
- Someone else can run your project from the README alone.
Intermediate · 3 months
Walk: You can build it so it keeps working, without help.
Build pipelines that keep working: tested, modelled, governed and streaming where it matters.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Google Cloud you use |
|---|---|---|
| Lakehouse tables | open table formats, upserts, time travel, partitioning | Iceberg on BigLakeopen table format BigQuerywarehouse |
| Modelling | dimensional models (facts and dimensions), slowly changing dimensions | Dataform or dbt on BigQuerysql modelling |
| Quality and contracts | checks at every layer; a contract between producer and consumer | Dataplex data qualitydata quality |
| Change data and streams | capture changes from apps; late and duplicate events | Datastreamingestion Pub/Sub + Dataflowstreaming |
| Catalog and access | owners, grants, lineage, personal data | Dataplexcatalog and access Dataplex lineagelineage |
| Infrastructure as code and CI | the platform in files; tests in the pipeline | Terraforminfrastructure as code GitHub Actions + Cloud Buildci/cd |
Build on GitHub
CDC to streaming lakehouse
Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.
Data contracts in CI
A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.
Ontology and knowledge graph
One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.
Built on Open source today. Porting it to Google Cloud is a good first contribution.
You are ready to move on when
- A bad row is stopped by a check, not found by a user.
- A schema change in the source fails a test before it breaks a dashboard.
- You can rebuild the whole platform from code.
Advanced · 6 months +
Run: You can design it, prove it and lead others through it.
Design a platform, prove its cost and correctness, and make data ready for AI.
Key ideas at this level, the same on every platform:
Learn
| Learn | What it means | On Google Cloud you use |
|---|---|---|
| Cost engineering | what each job costs; skew, small files and shuffle; cut the bill | Billing export to BigQuerycost Dataproc (Spark)batch compute |
| Semantic layer | metrics defined once, used by dashboards and AI alike | Looker (LookML)semantic layer |
| Data for AI | documents to chunks to vectors; a governed path to the model | Vertex AI Vector Search / AlloyDBvector search Gemini on Vertex AImodel api |
| Platform specs | what must exist, what must be true, what it must survive | Dataplexcatalog and access Dataplex data qualitydata quality |
| Lead the design | write the decision record; review others' pipelines | GitHubgit and pull requests |
Build on GitHub
Spark cost agent
Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.
Built on Open source today. Porting it to Google Cloud is a good first contribution.
Clean Data Engineering specs
Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.
Built on Databricks today. Porting it to Google Cloud is a good first contribution.
Conversation UI on data
Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.
Built on Databricks, Open source today. Porting it to Google Cloud is a good first contribution.
AskSherlock
An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.
Built on Open source today. Porting it to Google Cloud is a good first contribution.
You are ready to move on when
- You can show a cost before and after, and why it changed.
- A business user gets the same answer from the dashboard and from the AI assistant.
- You have reviewed and improved someone else's pipeline.
Optional
Certifications on Google Cloud
Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.
- Google Cloud Professional Data Engineer
Free prep we have written: Certifications.
Learn together
Do it with others
Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.