Home › Data engineering › Fundamentals › Processing
Data engineering fundamentals · free guide
Processing fundamentals: how data jobs actually run
Before tuning anything, it helps to know what a processing engine actually does with your code. This page explains the ideas every data engineer leans on: plans, partitions, shuffles, incremental work, state and failure, in plain words and organised as a mind map.
Start here
The data physics
Five plain facts that explain almost everything below. Read these and the rest follows.
The mind map
The whole topic on one screen
Each card is a branch. Walk them in order when you are stuck: the question tells you what the branch is about, and the first check tells you where to look.
1. Execution model
What actually runs, and when?
Lazy plans, actions, stages, tasks, the optimiser, cost-based choices
First check: Read the query plan
2. Batch and streaming
How often and how much at a time?
Full, incremental, micro-batch, continuous, triggers, watermarks
First check: Freshness need vs cost
3. Parallelism
How is the work split?
Partitions, narrow vs wide, shuffle, broadcast, skew
First check: Partition count and size
4. Shaping operations
What do the operations do?
Joins, aggregations, windows, sorts, distinct, explode
First check: Row count before vs after
5. State and history
What must be remembered?
Streaming state, change tracking, SCD2, deduplication, watermarks
First check: State size growth
6. Correctness under failure
Is a rerun safe?
Idempotency, exactly-once vs at-least-once, atomic commits, retries, checkpoints
First check: Run it twice and compare
7. Choosing the engine
Is a cluster needed?
Single-node engines, SQL warehouse, Spark, streaming engines
First check: Data size and concurrency
Execution model
From your code to tasks on cores
Knowing these five words explains most performance and cost behaviour.
Lazy evaluation: transformations describe; an action runs.
| Word | Plain meaning | Why you care |
|---|---|---|
| Transformation | A step that describes new data from old (filter, join, select) | Free until something asks for the result |
| Action | A step that demands a result (count, collect, write, show) | Each action triggers a real job with planning overhead |
| Plan and optimiser | The engine rewrites your steps to be cheaper | Pushes filters down, picks join methods; defeated by opaque code like UDFs |
| Stage | A run of steps with no data movement between them | A shuffle ends one stage and starts the next |
| Task | One stage run over one partition | Tasks are the unit of parallelism and of failure retry |
Batch and streaming
How much at a time, and how often
The choice is a freshness question first, an engineering question second. Streaming is not 'better batch'; it is a different cost and complexity profile.
Freshness is a business question first. Cost and complexity rise as you go right.
| Style | Plain meaning | Good when | Watch out for | Verdict |
|---|---|---|---|---|
| Full refresh | Recompute everything each run | Small data, simple logic | Cost grows with history | Fine until it hurts |
| Incremental batch | Process only new or changed data | Most daily and hourly pipelines | Needs a reliable change marker and late-data plan | Default for growing data |
| Micro-batch streaming | Small batches every few seconds or minutes | Near-real-time needs | Cluster always on; small files | Pursue when freshness pays |
| Continuous streaming | Row-by-row processing | Sub-second needs | Highest complexity and cost | Only where a must |
| Triggered / available-now | Streaming code that runs on a schedule and stops | You want streaming semantics without an always-on cluster | Latency equals the schedule | Often the sweet spot |
Parallelism
Partitions, narrow and wide, and why order of steps matters
Anything you can do before a shuffle makes the shuffle smaller. Filter early, project early, join late.
Why a shuffle is the expensive step: all-to-all movement across the network. Filter before it, not after.
| Concept | Plain meaning | Rule of thumb |
|---|---|---|
| Partition | A slice of the data handled by one task | Aim for roughly 100 to 200 MB |
| Narrow operation | Each output partition depends on one input partition | Cheap; chain them freely |
| Wide operation / shuffle | Output depends on many input partitions | Expensive; do after filtering |
| Broadcast | Send a small table to every worker | Use for small dimensions |
| Skew | Uneven partitions | One slow task sets the finish time |
| Repartition vs coalesce | Repartition reshuffles; coalesce merges without a full shuffle | Coalesce to reduce, repartition to rebalance |
Shaping operations
What joins, windows and aggregations really do to row counts
Most wrong results and most explosions come from not knowing how many rows an operation produces.
| Operation | Rows out vs rows in | Common trap | Check |
|---|---|---|---|
| Filter / select | Same or fewer | Filtering after an expensive step | Move filters first |
| Aggregate / group-by | Fewer | Group key with huge cardinality | Count distinct keys first |
| Inner join | Matches only | Duplicate keys multiply rows (fan-out) | Check key uniqueness on the small side |
| Left join | At least all left rows | Null-filled columns hide missing matches | Count unmatched rows |
| Window function | Same rows, extra columns | Window over the whole table with no partition key | Always partition the window |
| Explode / flatten | More | Output explodes before aggregating | Aggregate soon after |
| Distinct | Fewer | Used to hide duplicate fan-out | Fix the join, not the output |
| Sort | Same | Global sort for no reason | Sort only the final slice |
State and history
What the engine must remember
Anything that depends on earlier rows needs memory somewhere: a running total, a deduplication set, the last known version of a record.
| Need | How it is remembered | Risk | Control |
|---|---|---|---|
| Deduplicate a stream | Keep keys seen so far | State grows forever | Bound it with a watermark |
| Window aggregates over time | Open windows held until closed | Late data after close | Watermark and allowed lateness |
| Change tracking (SCD2) | Previous version rows in the table | Late and out-of-order changes | Sequence by source change time, not arrival |
| Checkpointing | Where the job stopped | Lost checkpoint means reprocessing | Protect and version it |
Correctness under failure
Make every rerun harmless
A pipeline you cannot safely rerun is a pipeline you cannot fix.
Safe reruns: overwrite or merge on a key, never blindly append.
| Property | Plain meaning | How to get it | Verdict |
|---|---|---|---|
| Idempotent write | Running twice gives the same table | Overwrite a partition, or merge on a key | Always |
| Atomic commit | Readers see all of a write or none | Table formats with transactions | Use a transactional format |
| At-least-once vs exactly-once | Duplicates possible vs not | Idempotent sink plus deterministic processing | Design for at-least-once, dedupe at the sink |
| Bounded retries | Retry some failures, not forever | Cap attempts; alert after | Always |
| Deterministic logic | Same input, same output | Avoid random and current-time inside transformations | Pursue |
| Reconciliation | Counts and totals match source | Automatic check after every load | Pursue |
Choosing the engine
Do you need a cluster?
A distributed engine pays start-up, planning and shuffle overhead on every job. That is worth it when data is large or concurrency is high. For small and medium data it can be the slowest choice.
| Data and need | Fits | Why | Verdict |
|---|---|---|---|
| Under a few GB, one user or job | Single-node engine (DuckDB-style) | No cluster start-up, no shuffle, reads columnar files directly | Pursue; cheapest and fastest |
| Tens of GB to a few TB, many analysts | SQL warehouse | Concurrency management and caching | Pursue |
| Many TB, complex transforms | Distributed engine (Spark) | Scale-out for shuffles and joins | Pursue |
| Seconds-level freshness | Streaming engine | Continuous processing | Only where a must |
In practice
CostSherlock rules for this topic
CostSherlock is our open set of cost and performance rules. Each rule below has a plain meaning here; the rule id is its name in CostSherlock. They are grouped by the same branches as the mind map, so a finding tells you which branch to open.
Execution model
| CostSherlock rule | What it catches, in plain words |
|---|---|
code_spark_action_inside_loop | An action per loop pass |
code_spark_action_result_discarded | An action whose result is thrown away |
code_persist_without_unpersist | Cached data never released |
aqe_disabled_in_sql | Adaptive execution disabled |
uc_cluster_cbo_disabled | Cost-based optimiser disabled |
Batch and streaming
| CostSherlock rule | What it catches, in plain words |
|---|---|
delta_full_refresh_pattern | Full rewrite where incremental would do |
dlt_continuous_for_batch_workload | Continuous for batch data |
uc_streaming_writes_tiny_batch_size | Streaming writing tiny batches |
auto_loader_unbounded_max_files | Ingest batch size unbounded |
Parallelism
| CostSherlock rule | What it catches, in plain words |
|---|---|
partition_skew | Skewed partitions |
uc_query_underpartitioned | Too few partitions |
uc_query_excessive_task_count | Too many tasks |
excessive_shuffle_partitions_config | Shuffle partition setting off |
code_hardcoded_repartition_debug_value | A debug repartition value left in code |
Shaping operations
| CostSherlock rule | What it catches, in plain words |
|---|---|
cross_join | Cross join |
excessive_joins | Too many joins in one query |
self_join | Self join where a window would do |
window_without_partition | Window without partition key |
explode_amplifies_without_aggregation | Explode with no aggregate |
multiple_distinct_aggregations | Several distinct aggregations |
null_compared_with_equals | NULL compared with = |
not_in_with_subquery | NOT IN with a subquery |
union_instead_of_union_all | UNION instead of UNION ALL |
Correctness under failure
| CostSherlock rule | What it catches, in plain words |
|---|---|
update_without_where | UPDATE with no filter |
delete_without_where | DELETE with no filter |
dbt_incremental_without_unique_key | Incremental model without a key |
nondeterministic_function_in_filter | Non-deterministic function in a filter |
merge_without_partition_predicate | MERGE scanning the whole target |
FAQ
Questions people ask
What does lazy evaluation mean?
Transformations only describe work. The engine waits until you ask for a result (an action) so it can optimise the whole plan together.
Why is a shuffle expensive?
Rows are written to disk, sent over the network to other workers, read back and sorted or grouped, and every worker waits for the slowest. It is the most costly step in most jobs.
Batch or streaming?
Start from how fresh the answer must be. If hourly is fine, incremental batch is simpler and cheaper. Use streaming only where the business acts within seconds.
What does idempotent mean in practice?
If a job runs twice by accident, the table ends up the same. Overwrite a partition or merge on a key instead of blindly appending.
Do I need Spark for my data?
Not if it fits on one machine. A single-node engine skips cluster start-up and shuffle overhead. Use a cluster when size or concurrency demands it.
Are the size thresholds fixed?
No. They are rules of thumb to start from. Measure your own workload.
Related
Where it connects
Performance fundamentals · Cost fundamentals · Security fundamentals · Storage fundamentals · Database families · Where data engineering time goes · Who maintains your tables · All data engineering
The numbers and thresholds on this page are rules of thumb and orders of magnitude to build intuition, not benchmarks. Check your platform's current documentation and measure your own workload.