Home › Learn › Data platform architecture › DataTwin Studio › Spark internals
Scenes: One batch, end to endDelta Lake internalsSpark internalsProblem LabData qualityPerformance stories · more coming: streaming, Kafka, Unity Catalog, dimensional modelling, cost
DataTwin Studio · Spark internals
Spark internals, recorded
What Spark actually does with your code. Each step shows the code, the result, every stage with its tasks as a timeline (so you can see waves and the one slow task), the query plan, the jobs and the logs. Seven scenarios on 2 million orders, from 'an action becomes tasks' to 'AQE splits a skewed join'.
real Recorded on local[2] Spark 3.5.1 on 2026-10-27. Every stage, task, byte shuffled, plan and timing comes from Spark's own event log and EXPLAIN. Timings are from a laptop with 2 cores: compare them with each other, not with a cluster.
Read alongside
Data platform architecture · Medallion architecture · Open table formats · The small-files problem · Idempotency · Where data costs come from