Home › Key concepts › Data quality dimensions
Key concept · Crawl (beginner) · the same on every platform
Data quality dimensions
Six ways data goes wrong, and a check for each.
What it is
Completeness (are values missing?), uniqueness (are there duplicates?), validity (is the value allowed?), consistency (do two sources agree?), accuracy (is it true?) and timeliness (is it fresh?). Each can be written as an automatic check.
See it happen: Data quality as metadata: YAML rules, one port, Spark SQL or Deequ (simulation), recorded from a real run.
Why it matters
Checks at each layer stop bad data before a person or a model uses it.
An example
A check fails the silver load because 3% of orders have no customer id, instead of the gap appearing in a board report.
The common mistake
Checking quality only by eye, after someone complains.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Data qualitychecks that stop bad data before anyone uses it | Glue Data Quality | Purview data quality | Dataplex data quality | Pipeline expectations | Great Expectations or Soda |
Related ideas
Where it shows up
Part of the Crawl level on these paths: Data Engineering (trusting data) · ML Engineering (features and data).