Data Quality › Missing Data

Missing data is ubiquitous in real-world datasets and poses significant challenges for statistical analysis. The appropriate strategy for handling missing values depends critically on the mechanism that generated the missingness, that is, why certain values are absent while others are observed.

The statistical literature distinguishes three mechanisms that generate missing data, each with different implications for analysis. Missing Completely At Random (MCAR) occurs when the probability that a value is missing is unrelated to any observed or unobserved variables in the dataset. If laboratory equipment randomly malfunctions, producing missing measurements that bear no relationship to the samples being processed or any other variables, the missingness is MCAR. Under MCAR, complete case analysis (simply deleting observations with missing values) produces unbiased estimates, though it reduces sample size and statistical power.

Missing At Random (MAR) describes situations where the probability of missingness depends on observed variables but not on the missing values themselves, conditional on the observed data. For example, if younger participants in a longitudinal study are more likely to skip follow-up surveys, but among participants of the same age, the missingness is unrelated to the variables being measured, then the missingness is MAR. Despite the potentially misleading name, MAR does not mean values are missing randomly. It means the missingness pattern can be explained by observed variables. Under MAR, complete case analysis produces biased estimates, but multiple imputation and maximum likelihood methods yield valid inference.

Missing Not At Random (MNAR) represents the most challenging scenario, where missingness depends on the unobserved values themselves even after conditioning on all observed variables. If individuals with higher incomes are less likely to report their income, and this relationship persists even after accounting for other observed characteristics, the missingness is MNAR. No general-purpose statistical method can correct for MNAR without making untestable assumptions about the relationship between missingness and the unobserved values.

Determining which missingness mechanism operates in a given dataset requires both statistical investigation and relevant domain knowledge. Visual exploration provides initial insights: plotting missingness patterns against observed variables can reveal relationships suggesting MAR rather than MCAR. If missing income values occur predominantly among respondents from wealthy neighborhoods, this pattern suggests MAR at minimum.

Formal tests for MCAR exist, though they have limitations. Little's MCAR test examines whether means of observed variables differ between observations with different patterns of missingness. Rejecting the null hypothesis of MCAR provides evidence against that mechanism, but failing to reject does not prove MCAR holds, as the test may simply lack power. More importantly, tests cannot distinguish MAR from MNAR, as the distinction hinges on relationships involving unobserved values.

Domain knowledge often provides the strongest evidence about missingness mechanisms. Understanding how and why data were collected reveals likely sources of missing values. Sensor failures that occur independently of measured values suggest MCAR. Survey fatigue leading to higher non-response for later questions suggests MAR if earlier responses predict dropout. Income non-response related to actual income levels represents MNAR.

The following visualization demonstrates how the three missingness mechanisms affect statistical analysis and why complete case analysis produces biased estimates under MAR and MNAR but not under MCAR.

The choice of method for handling missing data should balance the plausibility of required assumptions against the analysis goals and available information. Complete case analysis, which analyzes only observations with no missing values, is simple and valid under MCAR but wasteful of data and biased under MAR or MNAR. When missingness is minimal and likely MCAR, this approach may be adequate.

Single imputation methods replace each missing value with a single estimated value based on observed data. Mean imputation fills missing values with the variable's average among observed cases, preserving the sample mean but artificially reducing variance and distorting correlations. Regression imputation predicts missing values from other variables using regression models, capturing relationships but still underestimating uncertainty by treating imputed values as if they were observed.

Multiple imputation represents current best practice for handling missing data under MAR assumptions. This approach creates multiple complete datasets by imputing missing values multiple times, each time incorporating appropriate randomness to reflect uncertainty about the true missing values. Each complete dataset is analyzed separately, and results are combined using Rubin's rules to produce final estimates and standard errors that properly account for imputation uncertainty. Multiple imputation requires specifying an imputation model relating missing variables to observed ones, which should be carefully considered based on domain knowledge.

Maximum likelihood methods provide an alternative to imputation that directly estimates parameters from the observed data without explicitly filling in missing values. These methods also assume MAR and can be more efficient than multiple imputation when the analysis model and missingness pattern are compatible. However, they are less flexible for complex analyses requiring multiple steps.

When implementing any missing data method, transparency and sensitivity analysis are important. Document how much data is missing for each variable and the patterns of missingness across variables. Report whether analyses use complete cases, imputation, or maximum likelihood, and justify the choice. For important analyses, conduct sensitivity analyses comparing results across methods. If conclusions change dramatically depending on how missing data is handled, this fragility should be acknowledged.

Consider whether missing data itself carries information that should be preserved. In some contexts, the fact that a value is missing may be more informative than an imputed value. For instance, missing injury data in workplace safety records might indicate absence of injury, while imputation could introduce spurious variation. Creating indicator variables flagging missingness alongside imputed values allows analyses to capture both sources of information.