Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Confidence Intervals and Test Assumptions › Welch's T-Test: The Safer Default

Michael BrenndoerferUpdated January 12, 20262 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 13Confidence Intervals and Test Assumptions › Welch's T-Test: The Safer Default

Reading section 6 of 9 · Welch's T-Test: The Safer Default · 2 minSection 6 of 9 · 2 min

Section 6 of 9 · 2 min read

Confidence Intervals and Test Assumptions › Welch's T-Test: The Safer DefaultOne section · Sections in this chapter6 / 9 ↓

Sections in this chapter9

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

When comparing two independent groups, Welch's t-test should be your default choice.

The Pooled vs Welch ComparisonLink Copied

Pooled (Student's) t-test assumes equal variances:

t = \frac{\bar{x}_1 - \bar{x}_2}{s_p \sqrt{\frac{1}{n_1} + \frac{1}{n_2}}} \quad \text{where} \quad s_p^2 = \frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}

Welch's t-test allows unequal variances:

t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}

The degrees of freedom for Welch's test are calculated using the Welch-Satterthwaite formula:

df = \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{(s_1^2/n_1)^2}{n_1-1} + \frac{(s_2^2/n_2)^2}{n_2-1}}

This formula typically gives a non-integer value for degrees of freedom.

When to Use WhichLink Copied

Condition	Recommended Test	Reason
Variances clearly equal	Either works	Pooled slightly more powerful
Variances uncertain	Welch's	Allows unequal variances
Variances clearly unequal	Welch's	Pooled will be biased
Default choice	Welch's	Minimal cost if variances are equal

In[11]:

Code

import numpy as np
from scipy import stats

# Example: Comparing two treatments with different variability
np.random.seed(42)

# Treatment A: Lower mean, lower variance
treatment_a = np.random.normal(50, 5, 25)

# Treatment B: Higher mean, higher variance
treatment_b = np.random.normal(55, 12, 30)

print(
    f"Treatment A: mean = {np.mean(treatment_a):.2f}, std = {np.std(treatment_a, ddof=1):.2f}, n = {len(treatment_a)}"
)
print(
    f"Treatment B: mean = {np.mean(treatment_b):.2f}, std = {np.std(treatment_b, ddof=1):.2f}, n = {len(treatment_b)}"
)
print()

# Pooled t-test (assumes equal variances)
t_pooled, p_pooled = stats.ttest_ind(treatment_a, treatment_b, equal_var=True)
print(f"Pooled t-test:  t = {t_pooled:.3f}, p = {p_pooled:.4f}")

# Welch's t-test (allows unequal variances)
t_welch, p_welch = stats.ttest_ind(treatment_a, treatment_b, equal_var=False)
print(f"Welch's t-test: t = {t_welch:.3f}, p = {p_welch:.4f}")

Out[11]:

Console

Treatment A: mean = 49.18, std = 4.78, n = 25
Treatment B: mean = 52.49, std = 10.79, n = 30

Pooled t-test:  t = -1.418, p = 0.1621
Welch's t-test: t = -1.509, p = 0.1389

In this example, Treatment B has more than twice the variance of Treatment A. The Welch's t-test correctly accounts for this, while the pooled test uses an inappropriate compromise variance estimate.