One starting page for the data and AI forward deployed engineer: the role on one picture, four phases, eight tracks, and links into every guide.
Six plain-words mind-map guides: performance, cost, processing, security, storage and database families, with the CostSherlock rules that catch each mistake.
Where the time goes in a data job: disk vs memory vs network, tasks, shuffles, layout, compression, waiting, and why gold exists.
Where data platform money goes: idle and oversized compute, wasteful queries, storage growth, refresh frequency, egress, anomalies.
How data jobs run: lazy plans, partitions, shuffles, batch vs streaming, state, idempotency, and when one machine is enough.
Identity, secrets, encryption, classification, masking, audit and safe sharing, as a checklist you can walk.
Files plus a ledger: formats, compression, partitioning and clustering, table logs, updates, maintenance and tiers.
Relational, NoSQL, MPP, open table formats, object storage, graph and vector: what each is built for and how to choose.
Understanding what the source data means takes more hours than writing code. Two charts of where the time goes, and how to win the non-technical hours back.
How we build it: control plane, metadata, three services, streaming lane, agents that only suggest.
Same Parquet, two contents pages; which Databricks features differ (change data feed, streaming, pipelines, lineage, masks) and which to pick on AWS.
Kimball done right on a lakehouse: grain first, surrogate vs business keys, SCD2 joins, every kind of fact, the bus matrix, schemas, materialize or pre-aggregate and at what grain, fast Power BI, joining and grain problems.
One harness for every fact: incremental loading and watermarks, windows, completeness and restatements, late data, history load, reloads and bad batches, pure builds tested on a synthetic world, troubleshooting and regression.
What declarative pipelines are, open-source SDP vs Lakeflow, what each costs (serverless vs classic), where declarative helps and where not, bronze to silver, silver to gold, and with a harness.
CDC input or full snapshots: when Lakeflow AUTO CDC is right, when to write the MERGE ourselves, clean tested code for both, and the cost.
Requirements, design and low-level design for one harness that builds every gold table as robustly as AUTO CDC: watermarks, triggers, keys, late data, pure functions, record-drop checks.
The whole platform with every concern, then AWS-owned (S3 Tables, Glue, Lake Formation) vs Unity Catalog-owned designs: who can write, what each restricts, every concern compared, case studies and the winner.
S3 Tables, Glue and EMR against Unity Catalog, Delta and UniForm: format, compute, upkeep, pipelines, Power BI, governance, lineage, orchestration, cost, a scorecard and the documented limitations.
Managed, external and AWS-managed Iceberg tables: who runs compaction and VACUUM, why the catalog pointer decides, and the setup to pick on AWS with Databricks.
Ten pages: the semantic model idea first, then metric views, dbt and Power BI side by side, limits, parts, grain, row-level security, materialization, production and one example end to end.
Which of the four to use for a business number, with worked examples of the grain problem, ten copies of one formula, fan-out joins and speed, a decision chart and a verdict per scenario.
Where a refresh runs, incremental refresh vs full recompute, refresh policies, what forces a full recompute, cost, a checklist to avoid it, and six ways to see which refresh ran.
Five layers from identity to storage, the three admin roles, the USE CATALOG + USE SCHEMA + SELECT rule, ownership vs MANAGE, row filters and ABAC, service principals, and a debugging order.
How row and column layouts differ, how a column store rebuilds a row by position, what a query reads, why updates are awkward, and which to use, with a measured demo.
Eleven picture-first pages measured on a real Delta table: the delta log and Parquet, data skipping, layout, joins, shuffles, the driver, PySpark and the JVM, Photon, query plans, the write paths and why ClickHouse is fast.
Thirteen pages with drawings: planes, driver and workers, memory and machine types, SQL warehouses and serverless (who owns it, prod and dev, business units), Photon and every cache, how billing works (DBUs, SKUs, autoscaling), dividing the bill, system tables and what reading them costs, Delta Sharing and cross-region cost, backup and recovery architecture, and cost-saving hacks.
Twenty-one pages for hard data engineering interviews: table maintenance in pictures with real runs, OPTIMIZE and VACUUM, maintenance strategy, slow writes, backup and recovery on Azure, tagging, row-level security, compute choice, joins, skew and salting, AQE and Photon, cost levers, declarative pipelines, Asset Bundles, Lakeflow Connect, UniForm, Vector Search, dbt, ODCS, egress, and a question bank.
Bronze, silver and gold, one order through every layer, nine rules, products per platform.
Every fast-lane and platform-lane option in one table: path, delay, push or pull, Unity Catalog, cost, verdict and why, with links.
IoT and ordinary sources, seconds and minutes: roles first, then options per part with pros, cons and a verdict; SCD, DLT, Auto Loader, replays, SLA report, cost.
Real-time mode (GA) moves data in milliseconds; Lakehouse//RT (beta) answers queries in milliseconds. Limits, cost against Flink and DynamoDB, and where each fits.
From a database change to a number on screen in a second: designs, products and cost.
Unit, contract, data and simulation tests, and where each runs.
Seventeen kinds of test on a reference build: when each runs, what blocks a release, what is proven, and how each works.
Static and offline checks that catch PySpark and SQL problems before they run.
The whole picture with a product in every box, and one ETL followed end to end.