Data Quality › Limitations and Considerations

Data quality work involves inherent tradeoffs and calls for careful judgment. Perfect data quality is unattainable: even carefully collected data contains measurement error, missing values, and unusual observations. The goal is not perfection but understanding data limitations and managing their impact on conclusions appropriately.

Aggressive data cleaning can introduce its own biases. Excluding too many outliers creates datasets that underrepresent distributional tails, potentially missing important phenomena. Imputing missing data with overly simple methods can create artificial precision and distort relationships. Each quality-related decision involves balancing competing concerns, and different stakeholders may reasonably disagree about appropriate tradeoffs.

The boundary between legitimate extreme values and outliers requiring special treatment is often unclear. Statistical criteria like z-scores or Cook's distance provide screening tools but not definitive answers. An observation three standard deviations from the mean might be an error or a real rare event that warrants special attention precisely because it is unusual. Domain expertise must guide these judgments, supplemented by investigation of individual cases.

Missing data methods rely on untestable assumptions about missingness mechanisms. We can never definitively prove that data are MAR rather than MNAR, because the distinction depends on relationships involving unobserved values. Sensitivity analyses help understand how conclusions might change under alternative assumptions, but uncertainty remains. Report this uncertainty directly rather than implying more certainty than the data support.

Data quality work is time-consuming and rarely produces publishable results on its own, creating incentives to shortcut or skip quality checks. However, conclusions derived from poor-quality data are worse than useless: they are misleading. Building data quality practices into standard workflows, automating checks where possible, and recognizing quality work in project evaluations helps ensure appropriate attention to these foundational activities.