Data Quality › Bias in Data Collection

Bias refers to systematic deviation of collected data from the true population values or relationships of interest. While measurement error operates at the level of individual observations, bias operates at the systematic level, producing datasets that misrepresent reality in consistent ways. Understanding sources of bias and their implications is fundamental to conducting valid statistical inference.

Selection bias occurs when the mechanism for including individuals in a sample is related to the outcome or relationships being studied. The resulting sample does not represent the target population, leading to estimates that do not generalize. This form of bias is particularly insidious because it cannot be corrected through statistical analysis alone: no amount of sophisticated modeling can recover the information lost by studying an unrepresentative sample.

Survival bias provides a classic example. Studies of successful companies systematically oversample survivors who made it through competitive markets while missing failed companies that disappeared. Analyzing only survivors produces overly optimistic conclusions about business strategies because the sample excludes failures. Similarly, studies of elderly populations automatically exclude individuals who died young, potentially missing important risk factors that only manifest in premature mortality.

Non-response bias arises when individuals who choose to participate in surveys differ systematically from non-respondents. If a satisfaction survey achieves only a 20% response rate, the respondents likely differ in their satisfaction levels from those who ignored the survey. Highly satisfied and highly dissatisfied customers may be overrepresented relative to moderately satisfied customers who feel ambivalent about responding.

Even with perfect response rates, the sampling frame itself may introduce bias. If the list of potential participants used to draw the sample excludes or underrepresents certain groups, the resulting sample will be biased regardless of how randomly individuals are selected from that flawed frame. Historical examples include telephone surveys systematically missing households without landlines and internet surveys excluding populations with limited digital access.

Convenience sampling, where researchers collect data from easily accessible participants, virtually guarantees sampling bias. Recruiting study participants from university students, conducting surveys at shopping malls, or analyzing data from online volunteers all produce samples that differ systematically from the general population in ways that likely relate to the research questions of interest.

Information bias occurs when systematic errors in measuring or classifying variables lead to distorted relationships in the data. Recall bias in retrospective studies exemplifies this problem: individuals asked to remember past exposures or behaviors may systematically misremember in ways correlated with their current health status. Mothers of children with birth defects, for instance, may recall prenatal exposures more thoroughly than mothers of healthy children, creating spurious associations.

Social desirability bias causes respondents to systematically misreport socially sensitive information, understating undesirable behaviors and overstating desirable ones. Self-reported data on income, exercise habits, alcohol consumption, and voting behavior are all susceptible to this bias, with the direction and magnitude varying across populations and contexts.

Preventing bias requires careful attention during study design, as bias cannot be reliably corrected after data collection. Random sampling from a complete sampling frame that covers the target population gives the basis for unbiased inference. When true random sampling is infeasible, probability-based sampling with known inclusion probabilities allows weighting adjustments to recover population estimates.

For unavoidable non-response, collecting data on non-respondents through abbreviated surveys or administrative records enables assessment of non-response bias. If non-respondents differ systematically from respondents on observable characteristics, inverse probability weighting can partially correct for the bias, though unobserved differences remain problematic.

Blinding and objective measurement reduce information bias when feasible. If outcome assessors do not know participants' exposure status, their measurements cannot be influenced by that knowledge. Using administrative records or biomarkers instead of self-report eliminates recall and social desirability biases, though these alternatives introduce their own measurement challenges.