Home › Learn › Data platform architecture › DataTwin Studio
Scenes: One batch, end to endDelta Lake internalsSpark internalsProblem LabData qualityPerformance stories · more coming: streaming, Kafka, Unity Catalog, dimensional modelling, cost
DataTwin Studio · scene 1 · one batch, end to end
Watch one batch go through a data platform
A clothing shop's orders file travels from the moment a data engineer onboards the table to the dashboard tile. Every panel shows what that part of the platform holds at that step: the YAML and the metadata database, the Spark jobs and tasks, the rows in each layer, the Delta Lake commit files, the query plan, the logs and the dashboard. Then break it: send the same file again.
real Recorded on local[2] Spark 3.5.1 + Delta Lake 3.2.0 on 2026-10-28. with a real SQLite metadata store. Only the story (who merged the pull request, why the file was re-sent) is made up.
Read alongside
Data platform architecture · Medallion architecture · Open table formats · The small-files problem · Idempotency · Where data costs come from