Home › Data Engineering

Stream · free guides, tools and paths

Data Engineering

Everything on this site about building data platforms: the story from sources to dashboards and agents, the architecture guides, tools and simulators, how to deliver data projects, the key concepts, and learning paths for each platform.

Foundations · shared by every stream

Start with the data system, then the platform

For data leaders first, then builders: the whole data system in one picture, then a page for each part with the questions to chase and the decisions to take.

Data systems, end to end

Start here. Upstreams, the data platform and downstreams on one page, then a page per block and per hop with the questions to chase, what breaks in production and the decisions.

Upstreams: know your sources

85 questions to ask about every source, streams included, with the reason for each; APIs, files, joins, and where the answers go.

The data platform, whole

Five hops along the medallion and 24 cross-cutting concerns, with layout, environments and cost pages.

Foundations · data platform

Architecture guides

Forward deployed engineer roadmap

One starting page for the data and AI forward deployed engineer: the role on one picture, four phases, eight tracks, and links into every guide.

Data engineering fundamentals

Six plain-words mind-map guides: performance, cost, processing, security, storage and database families, with the CostSherlock rules that catch each mistake.

Performance fundamentals

Where the time goes in a data job: disk vs memory vs network, tasks, shuffles, layout, compression, waiting, and why gold exists.

Cost fundamentals

Where data platform money goes: idle and oversized compute, wasteful queries, storage growth, refresh frequency, egress, anomalies.

Processing fundamentals

How data jobs run: lazy plans, partitions, shuffles, batch vs streaming, state, idempotency, and when one machine is enough.

Security fundamentals

Identity, secrets, encryption, classification, masking, audit and safe sharing, as a checklist you can walk.

Storage fundamentals

Files plus a ledger: formats, compression, partitioning and clustering, table logs, updates, maintenance and tiers.

Database families

Relational, NoSQL, MPP, open table formats, object storage, graph and vector: what each is built for and how to choose.

Where data engineering time goes

Understanding what the source data means takes more hours than writing code. Two charts of where the time goes, and how to win the non-technical hours back.

Data platform architecture

How we build it: control plane, metadata, three services, streaming lane, agents that only suggest.

Delta Lake vs Iceberg on Databricks

Same Parquet, two contents pages; which Databricks features differ (change data feed, streaming, pipelines, lineage, masks) and which to pick on AWS.

Gold data modelling: grain, keys, facts and dimensions

Kimball done right on a lakehouse: grain first, surrogate vs business keys, SCD2 joins, every kind of fact, the bus matrix, schemas, materialize or pre-aggregate and at what grain, fast Power BI, joining and grain problems.

Silver to gold: the harness that builds facts and dimensions

One harness for every fact: incremental loading and watermarks, windows, completeness and restatements, late data, history load, reloads and bad batches, pure builds tested on a synthetic world, troubleshooting and regression.

Declarative or not: Spark Declarative Pipelines, open source and on Databricks

What declarative pipelines are, open-source SDP vs Lakeflow, what each costs (serverless vs classic), where declarative helps and where not, bronze to silver, silver to gold, and with a harness.

SCD Type 2: AUTO CDC or our own PySpark?

CDC input or full snapshots: when Lakeflow AUTO CDC is right, when to write the MERGE ourselves, clean tested code for both, and the cost.

Gold harness spec: facts and SCD2 dimensions from silver

Requirements, design and low-level design for one harness that builds every gold table as robustly as AUTO CDC: watermarks, triggers, keys, late data, pure functions, record-drop checks.

Iceberg lakehouse on AWS with Databricks: who owns the catalog?

The whole platform with every concern, then AWS-owned (S3 Tables, Glue, Lake Formation) vs Unity Catalog-owned designs: who can write, what each restricts, every concern compared, case studies and the winner.

AWS-native vs Databricks lakehouse, with limitations

S3 Tables, Glue and EMR against Unity Catalog, Delta and UniForm: format, compute, upkeep, pipelines, Power BI, governance, lineage, orchestration, cost, a scorecard and the documented limitations.

Who maintains your tables

Managed, external and AWS-managed Iceberg tables: who runs compaction and VACUUM, why the catalog pointer decides, and the setup to pick on AWS with Databricks.

Metric views, taught from zero

Ten pages: the semantic model idea first, then metric views, dbt and Power BI side by side, limits, parts, grain, row-level security, materialization, production and one example end to end.

Table, view, materialized view or metric view

Which of the four to use for a business number, with worked examples of the grain problem, ten copies of one formula, fan-out joins and speed, a decision chart and a verdict per scenario.

Databricks materialized views: refresh, cost, limits

Where a refresh runs, incremental refresh vs full recompute, refresh policies, what forces a full recompute, cost, a checklist to avoid it, and six ways to see which refresh ran.

Databricks permissions: who can do what

Five layers from identity to storage, the three admin roles, the USE CATALOG + USE SCHEMA + SELECT rule, ownership vs MANAGE, row filters and ABAC, service principals, and a debugging order.

Row vs column storage: file formats and databases

How row and column layouts differ, how a column store rebuilds a row by position, what a query reads, why updates are awkward, and which to use, with a measured demo.

How reads and writes work in Delta Lake and Spark

Eleven picture-first pages measured on a real Delta table: the delta log and Parquet, data skipping, layout, joins, shuffles, the driver, PySpark and the JVM, Photon, query plans, the write paths and why ClickHouse is fast.

Databricks compute and cost, end to end

Thirteen pages with drawings: planes, driver and workers, memory and machine types, SQL warehouses and serverless (who owns it, prod and dev, business units), Photon and every cache, how billing works (DBUs, SKUs, autoscaling), dividing the bill, system tables and what reading them costs, Delta Sharing and cross-region cost, backup and recovery architecture, and cost-saving hacks.

Databricks interview prep

Twenty-one pages for hard data engineering interviews: table maintenance in pictures with real runs, OPTIMIZE and VACUUM, maintenance strategy, slow writes, backup and recovery on Azure, tagging, row-level security, compute choice, joins, skew and salting, AQE and Photon, cost levers, declarative pipelines, Asset Bundles, Lakeflow Connect, UniForm, Vector Search, dbt, ODCS, egress, and a question bank.

Medallion architecture

Bronze, silver and gold, one order through every layer, nine rules, products per platform.

Streaming options at a glance

Every fast-lane and platform-lane option in one table: path, delay, push or pull, Unity Catalog, cost, verdict and why, with links.

Streaming on AWS and Databricks

IoT and ordinary sources, seconds and minutes: roles first, then options per part with pros, cons and a verdict; SCD, DLT, Auto Loader, replays, SLA report, cost.

Databricks real time: real-time mode and Lakehouse//RT

Real-time mode (GA) moves data in milliseconds; Lakehouse//RT (beta) answers queries in milliseconds. Limits, cost against Flink and DynamoDB, and where each fits.

Real-time analytics

From a database change to a number on screen in a second: designs, products and cost.

Test architecture for data

Unit, contract, data and simulation tests, and where each runs.

Test strategy in practice

Seventeen kinds of test on a reference build: when each runs, what blocks a release, what is proven, and how each works.

Shift left: fail fast on data code

Static and offline checks that catch PySpark and SQL problems before they run.

AWS + Databricks, and the ETL end to end

The whole picture with a product in every box, and one ETL followed end to end.

Data Engineering · practice

Tools and simulators

Streaming simulator

IoT and database data through the fast lane and the medallion to a live screen and Power BI; then all three lanes over a simulated day.

Databricks cost calculator

Whole-platform and per-hop cost, streaming, hidden costs, what drives your bill.

Source system questionnaire

The upstream questions as a printable form for source owners.

DataTwin Studio

Watch one batch go through a real Spark and Delta Lake platform, then break it.

Problem Lab

Common data platform problems, reproduced and fixed.

Performance stories

Why jobs are slow, shown on real runs.

Data Engineering · practice

Delivering data projects

The technical anchor

The role that holds a data project's technical shape together.

Discovery

Fail fast for the whole project: the discovery document, beliefs to test, sign-off.

Project review

Review a data project's documents for gaps at its stage, with the checks.

Hard problems by layer

The problems that are still hard at each layer of a data platform.

Data Engineering · build with us

Open specifications, projects and leaders

Spec-driven data platforms

For leaders: describe the platform in specs, and generate and check the rest.

DataTwin, explained

Open spec: a fire drill for a data platform, with pretend sources and known answers.

Clean Data Engineering specs

Open specs for what a data platform must have, be and face.

Data engineering projects to join

Real projects with a page each: why, architecture, tasks to pick up.

Data Engineering · key concepts

Key concepts

Short pages on the ideas every data engineer uses, whatever the platform. The full A to Z covers every stream.

Moving data

Storing data

Shaping data

Trusting data

Running data

Data Engineering · on your platform

Learning path by platform

Crawl, walk and run on the platform you use, with projects. See the whole path.