Home › Learn › Data platform architecture › DataTwin Studio › Performance stories
Scenes: One batch, end to endDelta Lake internalsSpark internalsProblem LabData qualityPerformance stories · more coming: streaming, Kafka, Unity Catalog, dimensional modelling, cost
DataTwin Studio · Performance stories · from slow to fast, measured
Performance stories: the slow job, the clue, the fix
Every data team has a job that takes three hours. It is almost never one problem; it is four small ones stacked up. Story 0 removes them one at a time and measures each fix. Stories 1 to 12 take one problem each: what you see, the clue that gives it away (in the event log, the query plan or the table's metadata), and the fix with its measured gain.
real Recorded on local[2] Spark 3.5.1 + Delta Lake 3.2.0 on 2026-10-29. Every time is measured on a laptop with 2 cores and 1 GB of memory, on millions of rows rather than billions, and the scale is stated on each story. Compare before with after, not with a cluster. Story 12 (the cloud bill) is arithmetic with example prices, and is labelled illustrative.
Read alongside
Data platform architecture · Medallion architecture · Open table formats · The small-files problem · Idempotency · Where data costs come from