The Real Cost of Bad Data — And the Practical Path to Fixing It
IBM estimated the cost of poor data quality to the US economy alone at $3.1 trillion per year. For individual organisations, Gartner puts the average annual cost at £12 million. These are large numbers, and they can feel abstract — until you experience a board meeting where two departments present conflicting revenue figures, or a marketing campaign that targets the wrong segment because the CRM hasn't been properly deduplicated.
This article is a practical guide to data quality: what actually goes wrong, how to detect it systematically, and what a sustainable programme looks like.
Why Data Quality Degrades: The Root Causes
Data quality problems are rarely random. They have identifiable root causes, and understanding those causes is the first step to addressing them systematically.
1. Manual Data Entry
Humans make mistakes. Free-text fields invite inconsistency ("Ltd", "Limited", "ltd." are all the same legal entity). Date formats differ by country and by person. Phone numbers get entered with or without country codes. Any system that relies on manual entry will accumulate quality issues over time.
Fix: Constrain inputs. Replace free-text with dropdowns, lookups, and validated fields wherever possible. Where free text is unavoidable, implement real-time validation at the point of entry.
2. System Integration Without Transformation Logic
When you connect two systems that were built independently, their data models rarely align perfectly. Customer IDs don't match. Status codes mean different things. Timestamps are in different timezones. If you simply pipe data from one system to another without robust transformation logic, you inherit all of those inconsistencies.
Fix: Treat integration as a data engineering problem, not a connectivity problem. Every integration point should have explicit mapping logic, documented assumptions, and automated tests.
3. Schema Changes Without Communication
An upstream application team adds a new status code, renames a column, or changes a field from nullable to required. If your pipelines aren't monitoring for schema changes, they either silently corrupt data or fail in ways that take time to diagnose.
Fix: Implement schema change detection on all critical data sources. Tools like dbt's schema tests, Great Expectations, or commercial data observability platforms (Monte Carlo, Bigeye) can automate this.
4. No Ownership
In many organisations, nobody owns the data. The application team owns the system; the data team owns the pipeline; nobody owns the quality of the data as a business asset. Problems fall through the cracks.
Fix: Assign explicit data stewardship. Each critical data domain should have a named owner who is accountable for quality and is empowered to raise and resolve issues.
A Framework for Measuring Data Quality
Data quality is multi-dimensional. The most widely used framework breaks it into six dimensions:
| Dimension | What It Measures | Example Issue | |---|---|---| | Completeness | Are all required fields populated? | 30% of customer records missing email address | | Accuracy | Does the data reflect reality? | Revenue figures that don't reconcile with bank statements | | Consistency | Is the same data consistent across systems? | Customer status is "active" in CRM but "churned" in billing | | Timeliness | Is the data current enough for its intended use? | Sales pipeline updated weekly when daily decisions depend on it | | Uniqueness | Are there duplicate records? | Same customer appears 3 times with slight name variations | | Validity | Does the data conform to defined rules? | Dates in future for completed transactions |
Building a data quality scorecard that tracks these dimensions for your most critical datasets gives you a baseline and a way to measure improvement over time.
What a Sustainable Data Quality Programme Looks Like
Point-in-time data cleaning is not a solution — it's debt management. You clean the data, and within months it degrades again because the root causes haven't been addressed. A sustainable programme has four components:
1. Prevention at Source
The cheapest quality fix is the one you don't have to do. Validate data at the point of entry, constrain fields, and make it easy for users to input correct data. This requires collaboration between the data team and the application teams.
2. Automated Detection in the Pipeline
Integrate data quality checks into your ETL/ELT pipelines as first-class citizens, not afterthoughts. Every pipeline run should validate the data it processes and alert on anomalies before they propagate downstream.
In practice, this means writing dbt tests that check for nulls in critical fields, referential integrity between tables, and expected value ranges. It means setting up row-count anomaly detection to catch when a source suddenly sends 10% of its normal volume. It means dead-letter queues that hold failed records for investigation rather than silently dropping them.
3. Observability and Alerting
Data quality issues that aren't detected quickly are expensive. By the time a bad data problem surfaces in a board report, it may have propagated through dozens of downstream systems and influenced dozens of decisions.
Invest in data observability — the ability to understand the health of your data at any point in time. The minimum viable version of this is a dashboard showing pipeline success rates, record counts, and quality metric trends. More mature implementations use ML-based anomaly detection to catch issues that rule-based checks miss.
4. Governance and Stewardship
Technology alone cannot solve data quality. You need clear processes: how are quality issues reported and prioritised? What is the SLA for resolution? How are improvements tracked? Who approves changes to critical data models?
This doesn't require a large team. A lightweight governance process — a weekly data quality review, a shared issue tracker, and defined escalation paths — is sufficient for most organisations.
How We Approach This at Byte Cognetics
When we take on a new data engineering engagement, a data quality audit is always one of our first steps. We profile the key datasets, identify the most significant quality issues, and work backwards to their root causes. Our pipelines are built with data quality checks as a standard component — not an optional extra.
We've helped clients move from reactive firefighting ("why doesn't our revenue match the bank statement?") to proactive quality management where issues are caught in the pipeline before they ever reach a dashboard.
Concerned about the quality of data underpinning your analytics? Get in touch for a no-obligation data quality assessment.