Home › Learn › Data platform architecture › DataTwin Studio › Data quality
Scenes: One batch, end to endDelta Lake internalsSpark internalsProblem LabData qualityPerformance stories · more coming: streaming, Kafka, Unity Catalog, dimensional modelling, cost
DataTwin Studio · Data quality and profiling · metadata-driven, low-code
Data quality as metadata: one port, two engines
Data quality rules belong in metadata, not in pipeline code. A YAML file per dataset says what to check, how serious a failure is, and what to do about it. One interface (a port) compiles the rules for whichever engine the YAML names, Spark SQL or Deequ today, Great Expectations as another adapter. Change the engine by changing one line. Onboard a dataset by adding one file.
real Recorded on local[2] Spark 3.5.1 + Delta Lake 3.2.0 on 2026-10-29. Deequ 2.0.7 (through PyDeequ 1.4.0) did the Deequ runs, profiling, suggestions and anomaly checks; the metadata DB is a real SQLite database. Great Expectations has an adapter slot in the design but was not installed for these recordings.
Read alongside
Data platform architecture · Medallion architecture · Open table formats · The small-files problem · Idempotency · Where data costs come from