Home › Data engineering › Fundamentals

Data engineering · free guides

Data engineering fundamentals

The ideas every data engineer leans on, whatever the platform. Each page starts with the plain physics, shows the topic as a mind map, lists the decisions with a verdict, and links to the CostSherlock rules that catch the mistakes in practice.

Performance fundamentals

The core physics of data engineering performance: disk vs memory vs network, driver and executors, more or fewer tasks, row vs column storage, compression, partitioning, clustering, file sizes, shuffles, skew, waiting and locking, and why we build performance objects in gold.

Cost fundamentals

Where data platform money goes and why: compute, storage, network, licences and people; idle and over-sized clusters, wasteful queries, small files, refresh frequency, anomalies and the habits that keep a bill predictable.

Processing fundamentals

The basics of how data is processed: batch vs streaming, lazy plans and actions, narrow and wide steps, partitions, idempotency, incremental loads, joins, windows, state, retries, ordering and late data, and when a single machine is enough.

Security

The security basics a data engineer must own: identity and least privilege, encryption, secrets, network paths, sensitive data classification, masking, row and column control, audit, lineage, data sharing and retention.

Storage fundamentals

Storage basics for data engineers: object storage, files vs tables, row vs column formats, compression, partitioning and clustering, file sizing, table formats and transactions, versions and clean-up, tiers and cost.

Database families

The families of data stores a data engineer meets and what each is built for: relational, NoSQL (key-value, document, wide-column), MPP warehouses, big data open table formats, object storage, graph and vector databases. How to choose by access pattern.

How to use these

When something is wrong, pick the page by the symptom

SymptomStart with
The job is slowPerformance fundamentals
The bill is high or surprisingCost fundamentals
Results are wrong, duplicated or lateProcessing fundamentals
Someone saw data they should notSecurity fundamentals
Tables are slow, large or messyStorage fundamentals
Choosing where data should liveDatabase families