Confidence Intervals and Test Assumptions › Test Assumptions: The Foundation of Valid Inference

Michael BrenndoerferUpdated January 12, 20269 min read
Machine Learning from ScratchAll chapters
Part 2 of 9Hypothesis Testing
Chapter 13Confidence Intervals and Test Assumptions › Test Assumptions: The Foundation of Valid Inference
Reading section 4 of 9 · Test Assumptions: The Foundation of Valid Inference · 9 minSection 4 of 9 · 9 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 13 of 66

Section 4 of 9 · 9 min read

Confidence Intervals and Test Assumptions › Test Assumptions: The Foundation of Valid InferenceOne section · Sections in this chapter4 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Every statistical test is built on assumptions. When these assumptions hold, the test does what it claims: it controls the Type I error rate at the specified level and has known power properties. When assumptions are violated, these guarantees may break down.

Understanding assumptions isn't about rigidly checking boxes. It's about understanding when violations matter, when they don't, and what alternatives exist.

Assumption 1: Independence

Independence is the most important assumption and the one most commonly violated without recognition.

What independence means: Each observation provides unique information. The value of one observation doesn't tell you anything about another observation's value.

Why it matters: Statistical tests calculate standard errors assuming each observation contributes independent information. When observations are correlated, the effective sample size is smaller than the actual sample size. The test "thinks" it has more information than it does, leading to standard errors that are too small, test statistics that are too large, and p-values that are too small.

Common violations:

Out[6]:
Visualization
Scatter plot showing repeated measures from 5 subjects with clear within-subject correlation.
Repeated measures: Multiple measurements from the same subjects are correlated. If Subject 1 has high values, all their measurements tend to be high. The 20 points come from only 5 independent subjects.
Line plot showing autocorrelated time series data.
Time series: Sequential observations are autocorrelated. Today''s value is related to yesterday''s. The 100 points do not represent 100 independent pieces of information.

The consequences: If you analyze repeated measures as if they were independent, your standard errors will be too small (often dramatically so), your p-values will be too small, and you'll make many more false positive claims than you think. A study that appears to have n=100 might effectively have n=10 if each of 10 subjects contributes 10 correlated measurements.

Solutions: Use methods designed for your data structure:

  • Repeated measures: paired t-tests, repeated measures ANOVA, mixed-effects models
  • Clustered data: cluster-robust standard errors, mixed-effects models
  • Time series: ARIMA models, autocorrelation-corrected standard errors

Assumption 2: Normality

Many tests assume the data (or the sampling distribution of a statistic) is normally distributed.

What normality means: The t-test and z-test assume that the sampling distribution of the mean is normal. For small samples, this requires the population itself to be approximately normal. For large samples, the Central Limit Theorem saves us.

The Central Limit Theorem: This remarkable theorem states that the sampling distribution of the mean approaches normality as sample size increases, regardless of the population distribution. This is why t-tests work even when individual observations aren't normal.

Out[7]:
Visualization
Six histograms arranged in two rows showing population distributions and their corresponding sampling distributions of the mean.
The Central Limit Theorem in action. The top row shows three very different population distributions (exponential, uniform, and bimodal). The bottom row shows the sampling distribution of the mean for samples of n=30 from each population. Despite the non-normal populations, the sampling distributions are approximately normal.

How fast does the CLT work? It depends on the population distribution:

Population DistributionSample Size Needed for Approximate Normality
Already normalAny n works
Symmetric, light tailsn ≥ 10 usually sufficient
Moderately skewedn ≥ 30 is a common guideline
Heavily skewedn ≥ 50 to 100 may be needed
Very heavy tailsn ≥ 100+ may be needed

When normality matters most: For small samples (n < 20 or so), the shape of the population distribution affects test validity. Severe skewness or outliers can inflate or deflate Type I error rates.

Checking normality: Use histograms and Q-Q plots to visually assess normality. Q-Q plots compare your data quantiles to theoretical normal quantiles. Points should fall roughly on a straight line if the data are normal.

Out[8]:
Visualization
Q-Q plot showing points along diagonal line for normal data.
Q-Q plot for normal data: Points fall along the diagonal line, indicating the data are approximately normally distributed.
Q-Q plot showing curved pattern for skewed data.
Q-Q plot for right-skewed data: Points curve upward at the right end, indicating a right (positive) skew with heavier right tail than normal.
Q-Q plot showing S-shaped pattern for heavy-tailed data.
Q-Q plot for heavy-tailed data: Points curve away from the line at both ends, indicating heavier tails than the normal distribution.

Assumption 3: Equal Variances (Homoscedasticity)

Two-sample t-tests can assume equal variances in both groups (pooled t-test) or allow unequal variances (Welch's t-test).

Why equal variances matter for the pooled test: The pooled t-test combines variance estimates from both groups to get a single "pooled" variance:

sp2=(n1−1)s12+(n2−1)s22n1+n2−2s_p^2 = \frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}

This pooling is only valid if both groups truly have the same variance. When variances differ, the pooled estimate doesn't accurately represent either group.

What happens when variances differ: The consequences depend on the relationship between variance and sample size:

  • If the group with larger variance has smaller n: Type I error is inflated (you reject too often)
  • If the group with larger variance has larger n: Type I error is deflated (you reject too rarely)
  • If sample sizes are equal: Effects are minimal
Out[9]:
Visualization
Two overlapping histograms showing groups with different variances.
The problem with unequal variances. When Group B has higher variance (shown by wider spread), the pooled variance estimate is a compromise that doesn't accurately represent either group. The standard error of the difference is incorrectly calculated, leading to biased inference.

The solution: Welch's t-test: Welch's t-test doesn't assume equal variances. It calculates standard errors separately for each group and uses a modified degrees of freedom (the Welch-Satterthwaite approximation).

The key insight is that Welch's test is nearly as powerful as the pooled test when variances ARE equal, but much more accurate when they're not. This asymmetry makes Welch's test the recommended default.