“The numbers are wrong”: this is the sentence that ruins trust in a dashboard. Behind it, the causes vary: a customer counted twice, yesterday’s sales not yet loaded, a mistyped postal code. To act, you first need to name the problem and measure it. The dimensions of data quality give this common vocabulary.
Quality depends on the use
Data is not good or bad in absolute terms: it is fit or unfit for a use. A customer file without phone numbers suits an emailing campaign, not a calling campaign. Before measuring, you therefore ask: who uses this data, for what, and what error would have consequences?
The six usual dimensions
Frameworks do not all list exactly the same dimensions, but these six come up everywhere:
| Dimension | Question | Example of a defect |
|---|---|---|
| Completeness | Is the value filled in? | empty email |
| Validity | Does it respect the expected format or domain? | 4-digit postal code, email without @ |
| Accuracy | Does it match reality? | wrong address, well formed |
| Consistency | Is it compatible with other data? | “France” here, “FR” elsewhere; delivery date before the order |
| Uniqueness | Is each real-world object present only once? | same customer entered twice |
| Freshness (timeliness) | Is it recent enough for the use? | yesterday’s sales missing at 9 AM |
Two frequent confusions:
- validity and accuracy:
marc@ex.fris a valid email, but if it belongs to someone else, it is inaccurate. Validity is checked with a rule; accuracy often requires a reference source or human verification; - uniqueness and technical duplicates: two identical rows are an obvious duplicate; two rows “Marie Dupont” and “M. Dupont” with the same email are the same customer, which is harder to detect.
Some teams add integrity (references point to existing objects), precision (a sufficient level of detail), or accessibility.
From requirement to measurable rule
A vague requirement (“customer data must be clean”) cannot be measured. You translate it into rules:
- Completeness: “email is filled in for at least 98% of active customers”;
- Validity: “the French postal code has exactly 5 digits”;
- Plausibility (a verifiable form of accuracy): “the date of birth is in the past and after 1900”;
- Uniqueness: “no email is shared by two active customers”;
- Freshness: “the previous day’s sales are available every day at 7 AM”.
Each rule has a threshold (100% is not always realistic or necessary) and an owner.
Computing an indicator
A quality indicator is a simple ratio: compliant rows / checked rows. Mind the denominator: the validity of an email is computed on the emails that are filled in, otherwise you mix completeness and validity.
SELECT
100.0 * COUNT(email) / COUNT(*) AS pct_email_renseigne,
100.0 * SUM(CASE WHEN email LIKE '%_@_%' THEN 1 ELSE 0 END)
/ COUNT(email) AS pct_email_valide
FROM clients;
In SQL, COUNT(column) ignores NULLs: this is handy for completeness. Watch out for empty strings, which are not NULL: depending on the source, a missing email can be '', NULL, or 'n/a'.
Tracking quality over time
A one-off measurement is not enough: you compute the indicators at every load and record them. A chart of the trend shows whether quality is degrading (new version of a form, new source) and whether corrective actions work. Transformation tools (dbt tests) and specialized tools (Great Expectations, Soda, or the quality features of cloud platforms) automate these checks. But the choice of rules and thresholds remains a business decision, made with the users of the data.
Exercises
Solutions and tests stay on the server. In the student space, your code is run and checked, and the tutor gives hints before the solution.
Measuring the quality of a customer file
Data (
clients.csv):client_id,email,code_postal,date_naissance,pays 1,amina@ex.fr,69001,1990-04-12,FR 2,,75011,1985-11-30,FR 3,marc@ex.fr,7501,1979-02-15,FR 4,lea@ex.fr,33000,2031-01-01,FR 5,amina@ex.fr,69001,1990-04-12,FR 6,nina@ex.fr,13008,,France 7,sam.ex.fr,44000,1992-07-08,FR 8,paul@ex.fr,59000,1988-09-21,FR 9,ines@ex.fr,31000,1995-03-03,FR 10,,67000,2000-12-24,FRCompute, by hand, in a spreadsheet, or in SQL:
- the completeness of the email (as a % of the 10 rows);
- the number of emails that are filled in but invalid (without an
@preceded and followed by at least one character); - the number of invalid postal codes (not exactly 5 digits);
- the number of impossible dates of birth (in the future);
- the number of rows whose
paysis not in the expectedFRformat; - the number of duplicate rows (same email, same date of birth as another row).
- Email completeness computed over 10 rows
- Email validity computed only on the emails that are filled in
- Each indicator is linked to its dimension (completeness, validity, accuracy/plausibility, consistency, uniqueness)
- The affected rows are identified by their client_id
Which dimension?
At 9 AM, the sales dashboard does not yet show the previous day’s sales, which arrive around 11 AM. Which quality dimension is failing?
- Uniqueness
- Freshness (timeliness)
- Validity
Writing measurable rules
Customer service expresses three expectations: “we must be able to reach customers”, “no duplicate customers”, and “addresses must be up to date”. Translate each one into a measurable rule, with the dimension concerned, the indicator formula, and a threshold.
- Each rule names a dimension (completeness, uniqueness, freshness, or accuracy)
- Each rule has an explicit calculation formula (numerator and denominator)
- Each rule has a numeric threshold
- The address rule specifies how accuracy or freshness is verified (date of last confirmation, address reference database)
End-of-lesson quiz
An email
paul@ex.fractually belongs to another customer. Which dimension is failing?- Validity
- Accuracy
- Completeness
“Country = France” in one table, “FR” in another: which dimension?
- Consistency
- Freshness
- Uniqueness
Why compute email validity only on the emails that are filled in?
- So as not to mix completeness and validity
- Because SQL requires it
- To get a better score
Who sets the acceptance threshold for a quality rule?
- The data engineer alone
- The business users of the data, with the data team
- The quality tool, automatically
The rest of the course
- Data Contracts16 min
- Data Catalog and Lineage16 min
- GDPR and Personal Data in Data Platforms18 min
- Building a Quality Improvement Plan16 min