Home › Data engineering › Fundamentals › Processing

Data engineering fundamentals · free guide

Processing fundamentals: how data jobs actually run

Before tuning anything, it helps to know what a processing engine actually does with your code. This page explains the ideas every data engineer leans on: plans, partitions, shuffles, incremental work, state and failure, in plain words and organised as a mind map.

Start here

The data physics

Five plain facts that explain almost everything below. Read these and the rest follows.

Your code is a plan, not a program.Engines like Spark read your code, build a plan, optimise it, and only then run it. Nothing runs until an action asks for a result.
Work is cut into pieces.Data is split into partitions and each piece is processed by one task. Parallelism is simply many tasks at once.
Some steps stay put, some move data.Filtering and mapping rows stay within a partition (narrow). Joins and group-bys must bring related rows together (wide, a shuffle).
Things fail, so rerunning must be safe.Machines die and sources hiccup. A well-built job can run twice and leave the same result (idempotent).
Time has two meanings.When something happened (event time) and when you saw it (processing time) differ. Late and out-of-order data is normal.

The mind map

The whole topic on one screen

Each card is a branch. Walk them in order when you are stuck: the question tells you what the branch is about, and the first check tells you where to look.

Processing

1. Execution model

What actually runs, and when?

Lazy plans, actions, stages, tasks, the optimiser, cost-based choices

First check: Read the query plan

2. Batch and streaming

How often and how much at a time?

Full, incremental, micro-batch, continuous, triggers, watermarks

First check: Freshness need vs cost

3. Parallelism

How is the work split?

Partitions, narrow vs wide, shuffle, broadcast, skew

First check: Partition count and size

4. Shaping operations

What do the operations do?

Joins, aggregations, windows, sorts, distinct, explode

First check: Row count before vs after

5. State and history

What must be remembered?

Streaming state, change tracking, SCD2, deduplication, watermarks

First check: State size growth

6. Correctness under failure

Is a rerun safe?

Idempotency, exactly-once vs at-least-once, atomic commits, retries, checkpoints

First check: Run it twice and compare

7. Choosing the engine

Is a cluster needed?

Single-node engines, SQL warehouse, Spark, streaming engines

First check: Data size and concurrency

Execution model

From your code to tasks on cores

Knowing these five words explains most performance and cost behaviour.

Your codefilter, join, writePlannothing runs yetOptimiserreorders, pushesfilters downStagescut at eachshuffleOnly an action (write, count, collect, show) makes the engine run the plan. Each action is a new job with planning cost.

Lazy evaluation: transformations describe; an action runs.

WordPlain meaningWhy you care
TransformationA step that describes new data from old (filter, join, select)Free until something asks for the result
ActionA step that demands a result (count, collect, write, show)Each action triggers a real job with planning overhead
Plan and optimiserThe engine rewrites your steps to be cheaperPushes filters down, picks join methods; defeated by opaque code like UDFs
StageA run of steps with no data movement between themA shuffle ends one stage and starts the next
TaskOne stage run over one partitionTasks are the unit of parallelism and of failure retry

Batch and streaming

How much at a time, and how often

The choice is a freshness question first, an engineering question second. Streaming is not 'better batch'; it is a different cost and complexity profile.

How much at a timeFull refresheverything, every runIncremental batchonly new or changedMicro-batchsmall batches, oftenContinuousrow by rowsimple, costly at scalefresh, complexChoose by how fresh the answer must be. Most businesses need incremental batch.

Freshness is a business question first. Cost and complexity rise as you go right.

StylePlain meaningGood whenWatch out forVerdict
Full refreshRecompute everything each runSmall data, simple logicCost grows with historyFine until it hurts
Incremental batchProcess only new or changed dataMost daily and hourly pipelinesNeeds a reliable change marker and late-data planDefault for growing data
Micro-batch streamingSmall batches every few seconds or minutesNear-real-time needsCluster always on; small filesPursue when freshness pays
Continuous streamingRow-by-row processingSub-second needsHighest complexity and costOnly where a must
Triggered / available-nowStreaming code that runs on a schedule and stopsYou want streaming semantics without an always-on clusterLatency equals the scheduleOften the sweet spot

Parallelism

Partitions, narrow and wide, and why order of steps matters

Anything you can do before a shuffle makes the shuffle smaller. Filter early, project early, join late.

Narrow step: stays inside a partitionWide step (shuffle): all to allFilter, select, mapJoin, group-by, sort: data crosses the network

Why a shuffle is the expensive step: all-to-all movement across the network. Filter before it, not after.

ConceptPlain meaningRule of thumb
PartitionA slice of the data handled by one taskAim for roughly 100 to 200 MB
Narrow operationEach output partition depends on one input partitionCheap; chain them freely
Wide operation / shuffleOutput depends on many input partitionsExpensive; do after filtering
BroadcastSend a small table to every workerUse for small dimensions
SkewUneven partitionsOne slow task sets the finish time
Repartition vs coalesceRepartition reshuffles; coalesce merges without a full shuffleCoalesce to reduce, repartition to rebalance

Shaping operations

What joins, windows and aggregations really do to row counts

Most wrong results and most explosions come from not knowing how many rows an operation produces.

OperationRows out vs rows inCommon trapCheck
Filter / selectSame or fewerFiltering after an expensive stepMove filters first
Aggregate / group-byFewerGroup key with huge cardinalityCount distinct keys first
Inner joinMatches onlyDuplicate keys multiply rows (fan-out)Check key uniqueness on the small side
Left joinAt least all left rowsNull-filled columns hide missing matchesCount unmatched rows
Window functionSame rows, extra columnsWindow over the whole table with no partition keyAlways partition the window
Explode / flattenMoreOutput explodes before aggregatingAggregate soon after
DistinctFewerUsed to hide duplicate fan-outFix the join, not the output
SortSameGlobal sort for no reasonSort only the final slice

State and history

What the engine must remember

Anything that depends on earlier rows needs memory somewhere: a running total, a deduplication set, the last known version of a record.

NeedHow it is rememberedRiskControl
Deduplicate a streamKeep keys seen so farState grows foreverBound it with a watermark
Window aggregates over timeOpen windows held until closedLate data after closeWatermark and allowed lateness
Change tracking (SCD2)Previous version rows in the tableLate and out-of-order changesSequence by source change time, not arrival
CheckpointingWhere the job stoppedLost checkpoint means reprocessingProtect and version it

Correctness under failure

Make every rerun harmless

A pipeline you cannot safely rerun is a pipeline you cannot fix.

Job failsmidwayRerunSame table as a clean runidempotent writeBlind append: duplicatesFix: overwrite a partition or merge on a key

Safe reruns: overwrite or merge on a key, never blindly append.

PropertyPlain meaningHow to get itVerdict
Idempotent writeRunning twice gives the same tableOverwrite a partition, or merge on a keyAlways
Atomic commitReaders see all of a write or noneTable formats with transactionsUse a transactional format
At-least-once vs exactly-onceDuplicates possible vs notIdempotent sink plus deterministic processingDesign for at-least-once, dedupe at the sink
Bounded retriesRetry some failures, not foreverCap attempts; alert afterAlways
Deterministic logicSame input, same outputAvoid random and current-time inside transformationsPursue
ReconciliationCounts and totals match sourceAutomatic check after every loadPursue

Choosing the engine

Do you need a cluster?

A distributed engine pays start-up, planning and shuffle overhead on every job. That is worth it when data is large or concurrency is high. For small and medium data it can be the slowest choice.

Data and needFitsWhyVerdict
Under a few GB, one user or jobSingle-node engine (DuckDB-style)No cluster start-up, no shuffle, reads columnar files directlyPursue; cheapest and fastest
Tens of GB to a few TB, many analystsSQL warehouseConcurrency management and cachingPursue
Many TB, complex transformsDistributed engine (Spark)Scale-out for shuffles and joinsPursue
Seconds-level freshnessStreaming engineContinuous processingOnly where a must

In practice

CostSherlock rules for this topic

CostSherlock is our open set of cost and performance rules. Each rule below has a plain meaning here; the rule id is its name in CostSherlock. They are grouped by the same branches as the mind map, so a finding tells you which branch to open.

Execution model

CostSherlock ruleWhat it catches, in plain words
code_spark_action_inside_loopAn action per loop pass
code_spark_action_result_discardedAn action whose result is thrown away
code_persist_without_unpersistCached data never released
aqe_disabled_in_sqlAdaptive execution disabled
uc_cluster_cbo_disabledCost-based optimiser disabled

Batch and streaming

CostSherlock ruleWhat it catches, in plain words
delta_full_refresh_patternFull rewrite where incremental would do
dlt_continuous_for_batch_workloadContinuous for batch data
uc_streaming_writes_tiny_batch_sizeStreaming writing tiny batches
auto_loader_unbounded_max_filesIngest batch size unbounded

Parallelism

CostSherlock ruleWhat it catches, in plain words
partition_skewSkewed partitions
uc_query_underpartitionedToo few partitions
uc_query_excessive_task_countToo many tasks
excessive_shuffle_partitions_configShuffle partition setting off
code_hardcoded_repartition_debug_valueA debug repartition value left in code

Shaping operations

CostSherlock ruleWhat it catches, in plain words
cross_joinCross join
excessive_joinsToo many joins in one query
self_joinSelf join where a window would do
window_without_partitionWindow without partition key
explode_amplifies_without_aggregationExplode with no aggregate
multiple_distinct_aggregationsSeveral distinct aggregations
null_compared_with_equalsNULL compared with =
not_in_with_subqueryNOT IN with a subquery
union_instead_of_union_allUNION instead of UNION ALL

Correctness under failure

CostSherlock ruleWhat it catches, in plain words
update_without_whereUPDATE with no filter
delete_without_whereDELETE with no filter
dbt_incremental_without_unique_keyIncremental model without a key
nondeterministic_function_in_filterNon-deterministic function in a filter
merge_without_partition_predicateMERGE scanning the whole target

FAQ

Questions people ask

What does lazy evaluation mean?

Transformations only describe work. The engine waits until you ask for a result (an action) so it can optimise the whole plan together.

Why is a shuffle expensive?

Rows are written to disk, sent over the network to other workers, read back and sorted or grouped, and every worker waits for the slowest. It is the most costly step in most jobs.

Batch or streaming?

Start from how fresh the answer must be. If hourly is fine, incremental batch is simpler and cheaper. Use streaming only where the business acts within seconds.

What does idempotent mean in practice?

If a job runs twice by accident, the table ends up the same. Overwrite a partition or merge on a key instead of blindly appending.

Do I need Spark for my data?

Not if it fits on one machine. A single-node engine skips cluster start-up and shuffle overhead. Use a cluster when size or concurrency demands it.

Are the size thresholds fixed?

No. They are rules of thumb to start from. Measure your own workload.

Related

Where it connects

Performance fundamentals · Cost fundamentals · Security fundamentals · Storage fundamentals · Database families · Where data engineering time goes · Who maintains your tables · All data engineering

The numbers and thresholds on this page are rules of thumb and orders of magnitude to build intuition, not benchmarks. Check your platform's current documentation and measure your own workload.