Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

The T-Test › The Two-Sample T-Test

Michael BrenndoerferJanuary 3, 20267 min read

Machine Learning from ScratchAll chapters

Part 2 of 9Hypothesis Testing

Chapter 15The T-Test › The Two-Sample T-Test

Reading section 5 of 11 · The Two-Sample T-Test · 7 minSection 5 of 11 · 7 min

Section 5 of 11 · 7 min read

The T-Test › The Two-Sample T-TestOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The two-sample t-test compares means from two independent groups. This is perhaps the most common application: comparing treatment vs. control, method A vs. method B, or any two distinct populations.

The choice of t-test depends on one question: do the two groups have equal variance?

Why Variance Equality MattersLink Copied

When comparing two means, we need to estimate the standard error of the difference $\bar{x}_1 - \bar{x}_2$ . If both groups share the same variance $\sigma^2$ , we can pool their data to get a more precise estimate. But if variances differ, pooling gives incorrect results, biasing our inference.

This leads to two variants:

Pooled (Student's) t-test: Assumes equal variances, pools data for efficiency
Welch's t-test: Makes no variance assumption, handles unequal variances correctly

Pooled Two-Sample T-TestLink Copied

When we assume $\sigma_1^2 = \sigma_2^2 = \sigma^2$ , we can estimate this common variance using data from both groups.

Pooled variance estimate:

s_p^2 = \frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}

This is a weighted average of the two sample variances, with weights proportional to their degrees of freedom. Larger samples contribute more to the pooled estimate because they provide more reliable information.

Standard error of the difference:

SE = s_p \sqrt{\frac{1}{n_1} + \frac{1}{n_2}}

The term $\sqrt{1/n_1 + 1/n_2}$ accounts for the fact that uncertainty in the difference comes from uncertainty in both means.

Test statistic:

t = \frac{\bar{x}_1 - \bar{x}_2}{s_p \sqrt{\frac{1}{n_1} + \frac{1}{n_2}}}

Degrees of freedom: $df = n_1 + n_2 - 2$

Each group contributes $n - 1$ degrees of freedom for variance estimation, and we combine them.

Welch's T-Test (Recommended Default)Link Copied

When variances may differ, we cannot pool them. Instead, we estimate standard errors separately and combine them using a different formula.

Standard error of the difference:

SE = \sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}

This follows from the variance of a difference: $\text{Var}(\bar{x}_1 - \bar{x}_2) = \text{Var}(\bar{x}_1) + \text{Var}(\bar{x}_2)$ for independent samples.

Test statistic:

t = \frac{\bar{x}_1 - \bar{x}_2}{\sqrt{\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}}}

Degrees of freedom (Welch-Satterthwaite approximation):

df = \frac{\left(\frac{s_1^2}{n_1} + \frac{s_2^2}{n_2}\right)^2}{\frac{(s_1^2/n_1)^2}{n_1-1} + \frac{(s_2^2/n_2)^2}{n_2-1}}

This complex formula estimates the effective degrees of freedom when variances are unequal. The result is typically non-integer, which modern software handles through interpolation.

Why Welch's is the recommended default:

When variances are equal, Welch's test loses only slightly in power compared to the pooled test
When variances are unequal, the pooled test can give seriously incorrect results
The asymmetric risk/reward favors Welch's as the default choice

Worked Example: Comparing Teaching MethodsLink Copied

A researcher compares test scores from two teaching methods: Method A (traditional lectures) and Method B (active learning).

In[7]:

Code

import numpy as np
from scipy import stats

# Test scores from two teaching methods
method_a = [78, 82, 85, 79, 81, 84, 77, 83, 80, 82, 79, 86]  # n=12
method_b = [85, 89, 92, 88, 90, 93, 86, 91, 88, 90, 87, 94, 89, 91]  # n=14

# Sample statistics
n1, n2 = len(method_a), len(method_b)
x1_bar, x2_bar = np.mean(method_a), np.mean(method_b)
s1, s2 = np.std(method_a, ddof=1), np.std(method_b, ddof=1)

# ===== POOLED T-TEST =====
# Pooled variance
sp2 = ((n1 - 1) * s1**2 + (n2 - 1) * s2**2) / (n1 + n2 - 2)
sp = np.sqrt(sp2)

# Standard error (pooled)
se_pooled = sp * np.sqrt(1 / n1 + 1 / n2)

# Test statistic
t_pooled = (x2_bar - x1_bar) / se_pooled
df_pooled = n1 + n2 - 2

# P-value
p_pooled = 2 * stats.t.sf(abs(t_pooled), df=df_pooled)

# ===== WELCH'S T-TEST =====
# Standard error (Welch)
se_welch = np.sqrt(s1**2 / n1 + s2**2 / n2)

# Test statistic
t_welch = (x2_bar - x1_bar) / se_welch

# Degrees of freedom (Welch-Satterthwaite)
num = (s1**2 / n1 + s2**2 / n2) ** 2
denom = (s1**2 / n1) ** 2 / (n1 - 1) + (s2**2 / n2) ** 2 / (n2 - 1)
df_welch = num / denom

# P-value
p_welch = 2 * stats.t.sf(abs(t_welch), df=df_welch)

# Effect size (Cohen's d)
cohens_d = (x2_bar - x1_bar) / sp

Out[8]:

Console

Two-Sample T-Test: Comparing Teaching Methods
============================================================

Sample statistics:
------------------------------------------------------------
Method A: n = 12, mean = 81.33, std = 2.84
Method B: n = 14, mean = 89.50, std = 2.59
Difference (B - A): 8.17 points

POOLED T-TEST (assumes equal variances)
------------------------------------------------------------
Pooled std: s_p = 2.709
Standard error: SE = 1.066
t-statistic: t(24) = 7.662
p-value: 0.000000

WELCH'S T-TEST (no variance assumption)
------------------------------------------------------------
Standard error: SE = 1.074
t-statistic: t(22.6) = 7.607
p-value: 0.000000

Effect size:
------------------------------------------------------------
Cohen's d = 3.01 (very large effect)

Conclusion:
------------------------------------------------------------
Both tests show highly significant results (p < 0.001).
Method B produces substantially higher scores than Method A.
The effect size (d = 2.41) indicates a very large practical difference.

Out[9]:

Visualization

Side-by-side boxplots comparing Method A and Method B test scores. — Comparison of test scores between two teaching methods. The boxplots show the distribution of scores in each group, with individual data points overlaid. The minimal overlap between groups, combined with the large mean difference, results in a highly significant t-test.

Checking the Equal Variance AssumptionLink Copied

Before choosing between pooled and Welch's tests, you might want to check whether variances are equal. Two common approaches:

1. Visual inspection: Compare the spread in boxplots or calculate the ratio of sample variances. A ratio greater than 2 suggests meaningful inequality.

2. Levene's test: A formal hypothesis test for equality of variances. Unlike older alternatives (Bartlett's test), Levene's test is less sensitive to departures from normality.

In[10]:

Code

from scipy import stats

# Sample data with different variances
group1 = [45, 52, 48, 51, 49, 53, 47, 50, 46, 54]
group2 = [42, 58, 35, 62, 48, 55, 38, 65, 44, 52]

# Visual check: variance ratio
s1 = np.std(group1, ddof=1)
s2 = np.std(group2, ddof=1)
print(f"Group 1 std: {s1:.2f}")
print(f"Group 2 std: {s2:.2f}")
print(f"Variance ratio: {(s2 / s1) ** 2:.2f}")
print()

# Levene's test
stat, p_value = stats.levene(group1, group2)
print(f"Levene's test: W = {stat:.3f}, p = {p_value:.4f}")
if p_value < 0.05:
    print("Conclusion: Evidence of unequal variances. Use Welch's t-test.")
else:
    print(
        "Conclusion: No evidence of unequal variances. Either test is appropriate."
    )

Out[10]:

Console

Group 1 std: 3.03
Group 2 std: 10.19
Variance ratio: 11.33

Levene's test: W = 13.935, p = 0.0015
Conclusion: Evidence of unequal variances. Use Welch's t-test.

Practical Recommendation

Many statisticians now recommend always using Welch's t-test as the default, regardless of whether variances appear equal. The reasoning:

When variances are truly equal, Welch's test loses very little power compared to the pooled test
When variances are unequal, the pooled test can produce seriously misleading results
Pre-testing for equal variances adds complexity and has its own issues (low power with small samples, unnecessary when samples are large)

This "Welch by default" approach is increasingly adopted in statistical software and practice.