Home › Data engineering › Fundamentals
Data engineering · free guides
Data engineering fundamentals
The ideas every data engineer leans on, whatever the platform. Each page starts with the plain physics, shows the topic as a mind map, lists the decisions with a verdict, and links to the CostSherlock rules that catch the mistakes in practice.
Performance fundamentals
The core physics of data engineering performance: disk vs memory vs network, driver and executors, more or fewer tasks, row vs column storage, compression, partitioning, clustering, file sizes, shuffles, skew, waiting and locking, and why we build performance objects in gold.
Cost fundamentals
Where data platform money goes and why: compute, storage, network, licences and people; idle and over-sized clusters, wasteful queries, small files, refresh frequency, anomalies and the habits that keep a bill predictable.
Processing fundamentals
The basics of how data is processed: batch vs streaming, lazy plans and actions, narrow and wide steps, partitions, idempotency, incremental loads, joins, windows, state, retries, ordering and late data, and when a single machine is enough.
Security
The security basics a data engineer must own: identity and least privilege, encryption, secrets, network paths, sensitive data classification, masking, row and column control, audit, lineage, data sharing and retention.
Storage fundamentals
Storage basics for data engineers: object storage, files vs tables, row vs column formats, compression, partitioning and clustering, file sizing, table formats and transactions, versions and clean-up, tiers and cost.
Database families
The families of data stores a data engineer meets and what each is built for: relational, NoSQL (key-value, document, wide-column), MPP warehouses, big data open table formats, object storage, graph and vector databases. How to choose by access pattern.
How to use these
When something is wrong, pick the page by the symptom
| Symptom | Start with |
|---|---|
| The job is slow | Performance fundamentals |
| The bill is high or surprising | Cost fundamentals |
| Results are wrong, duplicated or late | Processing fundamentals |
| Someone saw data they should not | Security fundamentals |
| Tables are slow, large or messy | Storage fundamentals |
| Choosing where data should live | Database families |