The T-Test › Why Do We Need the T-Test?

In the previous chapter on z-tests, we learned to compare means when the population standard deviation $\sigma$ is known. The test statistic was:

This follows a standard normal distribution because the denominator $\sigma / \sqrt{n}$ is a fixed, known quantity. Only the numerator varies from sample to sample.

The problem is that you almost never know $\sigma$. In practice, you estimate the standard deviation from your sample data, giving you the sample standard deviation $s$. Early researchers simply substituted $s$ for $\sigma$ and continued using normal distribution critical values. This seemed reasonable, but it led to systematic errors. They rejected the null hypothesis more often than they should have, especially with small samples.

Gosset discovered why: when you estimate $\sigma$ from the same sample you're testing, you introduce additional variability into the test statistic. The sample standard deviation $s$ is itself a random variable that fluctuates from sample to sample. Sometimes $s$ underestimates $\sigma$, making your test statistic too large. Sometimes $s$ overestimates $\sigma$, making it too small. This extra variability means the test statistic no longer follows a normal distribution. It follows a distribution with heavier tails, one that appropriately penalizes us for not knowing the true population variance.