Home › Paths › Data Engineering › Google Cloud

Learning path · Google Cloud

Data Engineering on Google Cloud

You move, clean and shape data so other people can trust the numbers.

Free, like everything here: no sign-up, no paywall. Only the cloud you practise on may charge; the open-source path costs nothing.

Every AI system is only as good as the data under it. Data engineers build that floor: pipelines that land data, tables that hold it, checks that stop bad rows, and models that give each metric one meaning.

Every topic below is a concept (the job) with the Google Cloud product that does it. Learn the concept once; on another platform only the product name changes. Read alongside: Data platform architecture, drawn right · Medallion architecture, drawn right · The AI Universe · Certification prep on this site.

Beginner · 8 weeks

Crawl: You can build one thing end to end, with help.

Land a dataset, clean it in layers, and show it on a dashboard.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Google Cloud you use
SQL, really welljoins, group by, window functions; read a query plan onceBigQuerysql engine
Files and storageCSV, JSON, Parquet; why columnar files are faster and cheaperCloud Storageobject storage
Your first pipelineextract, load, transform; raw, clean and serving layersDatastreamingestion
Dataproc (Spark)batch compute
Schedule itrun daily, retry on failure, alert when it breaksCloud Composer (Airflow)orchestration
Show itone dashboard a real person would useLookerdashboards
Git from day oneevery change in a pull requestGitHubgit and pull requests

Build on GitHub

Under way · join

Ecom simple data platform

Orders, customers and products from a shop, landed raw, cleaned in layers and served as a sales dashboard.

Built on Azure, Databricks today. Porting it to Google Cloud is a good first contribution.

You are ready to move on when

Intermediate · 3 months

Walk: You can build it so it keeps working, without help.

Build pipelines that keep working: tested, modelled, governed and streaming where it matters.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Google Cloud you use
Lakehouse tablesopen table formats, upserts, time travel, partitioningIceberg on BigLakeopen table format
BigQuerywarehouse
Modellingdimensional models (facts and dimensions), slowly changing dimensionsDataform or dbt on BigQuerysql modelling
Quality and contractschecks at every layer; a contract between producer and consumerDataplex data qualitydata quality
Change data and streamscapture changes from apps; late and duplicate eventsDatastreamingestion
Pub/Sub + Dataflowstreaming
Catalog and accessowners, grants, lineage, personal dataDataplexcatalog and access
Dataplex lineagelineage
Infrastructure as code and CIthe platform in files; tests in the pipelineTerraforminfrastructure as code
GitHub Actions + Cloud Buildci/cd

Build on GitHub

Proposed · needs a lead

CDC to streaming lakehouse

Capture changes from an app database, stream them into bronze, silver and gold tables, with quality checks and lineage.

Proposed · needs a lead

Data contracts in CI

A producer and a consumer agree a contract (ODCS); a breaking schema change fails the pull request, not the dashboard.

Under way · join

Ontology and knowledge graph

One graph over code, tables, specs and AI systems, so you can ask what breaks if a column changes.

Built on Open source today. Porting it to Google Cloud is a good first contribution.

You are ready to move on when

Advanced · 6 months +

Run: You can design it, prove it and lead others through it.

Design a platform, prove its cost and correctness, and make data ready for AI.

Key ideas at this level, the same on every platform:

Learn

LearnWhat it meansOn Google Cloud you use
Cost engineeringwhat each job costs; skew, small files and shuffle; cut the billBilling export to BigQuerycost
Dataproc (Spark)batch compute
Semantic layermetrics defined once, used by dashboards and AI alikeLooker (LookML)semantic layer
Data for AIdocuments to chunks to vectors; a governed path to the modelVertex AI Vector Search / AlloyDBvector search
Gemini on Vertex AImodel api
Platform specswhat must exist, what must be true, what it must surviveDataplexcatalog and access
Dataplex data qualitydata quality
Lead the designwrite the decision record; review others' pipelinesGitHubgit and pull requests

Build on GitHub

Under way · join

Spark cost agent

Read Spark event logs, find the expensive mistakes (skew, small files, cartesian joins) and explain each in plain words.

Built on Open source today. Porting it to Google Cloud is a good first contribution.

Needs a lead

Clean Data Engineering specs

Write down what a data platform must have, what must be true of it, and the failures it must survive, then test real pipelines against it.

Built on Databricks today. Porting it to Google Cloud is a good first contribution.

Under way · join

Conversation UI on data

Ask questions of your data in plain English, where the model picks metrics and never writes raw SQL.

Built on Databricks, Open source today. Porting it to Google Cloud is a good first contribution.

Just starting · join

AskSherlock

An open-source ask-your-data assistant over a cost warehouse, with every answer traceable to a metric.

Built on Open source today. Porting it to Google Cloud is a good first contribution.

You are ready to move on when

Optional

Certifications on Google Cloud

Projects prove more than certificates, but a certificate can open a door. These fit this path. Exam names and codes change, so check the current one before you book.

Free prep we have written: Certifications.

Learn together

Do it with others

Every project above has a page, a list of open work and a "good first issue" label on GitHub. Pick one at your level, say hello by email, and build it in the open. If a project says needs a lead, that can be you.

All projects →   Another path →