Probability Basics › Central Limit Theorem

While the Law of Large Numbers tells us that sample means converge to population means, the Central Limit Theorem (CLT) tells us something remarkable: regardless of the shape of the underlying distribution, the distribution of sample means approaches a normal distribution as the sample size increases. This result is a common theorems in statistics and gives the basis for much of inferential statistics.

Formally, if $X_1, X_2, \ldots, X_n$ are independent random variables from any distribution with mean $\mu$ and finite variance $\sigma^2$, then the distribution of the standardized sample mean approaches a standard normal distribution as $n$ increases:

Where:

• $\bar{X} = \frac{1}{n}\sum_{i=1}^n X_i$ is the sample mean
• $\mu$ is the population mean
• $\sigma$ is the population standard deviation
• $n$ is the sample size

The Central Limit Theorem explains why the normal distribution appears so frequently in practice. Even when the underlying process doesn't follow a normal distribution, averages and totals often do. Heights, test scores, measurement errors, and countless other phenomena approximately follow normal distributions because they result from combining many small independent effects. This convergence typically occurs quite rapidly; sample sizes of 30 or more often suffice for reasonable normal approximation, even when the underlying distribution is quite non-normal.

The CLT has several practical implications for statistical inference:

1. Confidence Intervals: When estimating a population mean, the CLT tells us that the sampling distribution of $\bar{X}$ is approximately normal. This allows us to construct confidence intervals of the form $\bar{X} \pm z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}$, where $z_{\alpha/2}$ is the appropriate critical value from the standard normal distribution. We can make probabilistic statements like "we are 95% confident the true mean lies within this interval" without knowing the exact shape of the population.

   For example, suppose a factory produces light bulbs and we want to estimate the average lifespan. We take a random sample of 100 bulbs and find a sample mean of 1,200 hours with a known population standard deviation of 100 hours. Using the CLT, we know the sampling distribution of the mean is approximately normal. For a 95% confidence interval, we use $z_{0.025} = 1.96$, giving us $1200 \pm 1.96 \times \frac{100}{\sqrt{100}} = 1200 \pm 19.6$ hours. We can state that we are 95% confident the true average lifespan of all bulbs lies between 1,180.4 and 1,219.6 hours.

2. Hypothesis Testing: The CLT enables us to perform z-tests and t-tests for means. Under the null hypothesis, we can calculate how likely our observed sample mean would be if the true mean were some hypothesized value. The test statistic $z = \frac{\bar{X} - \mu_0}{\sigma / \sqrt{n}}$ follows a standard normal distribution, allowing us to compute p-values and make decisions about statistical significance.

   Consider a coffee shop chain that claims their medium cups contain exactly 16 ounces of coffee. A consumer group suspects the cups are underfilled and collects a random sample of 50 cups, finding a sample mean of 15.7 ounces. Assuming the population standard deviation is 0.8 ounces, we can test the null hypothesis $H_0: \mu = 16$ against $H_a: \mu < 16$. The test statistic is $z = \frac{15.7 - 16}{0.8 / \sqrt{50}} = \frac{-0.3}{0.113} = -2.65$. Looking up this value in the standard normal table gives a p-value of about 0.004. Since this is less than the typical significance level of 0.05, we reject the null hypothesis and conclude there is strong evidence that the cups contain less than 16 ounces on average.

3. Quantifying Uncertainty: The standard error $\sigma / \sqrt{n}$ provides a measure of how much sample means vary from sample to sample. This quantifies the precision of our estimates. Larger samples yield smaller standard errors and more precise estimates.

   To illustrate, imagine polling voters about their support for a candidate. If we survey 100 voters and find 55% support, the standard error is approximately $\sqrt{0.55 \times 0.45 / 100} = 0.05$ or 5 percentage points. If we instead survey 2,500 voters and find the same 55% support, the standard error drops to $\sqrt{0.55 \times 0.45 / 2500} = 0.01$ or just 1 percentage point. This is why major polls use large sample sizes. The larger sample gives us much more precision in our estimate, even though both samples show the same proportion of support.

4. Justification for Normal-Theory Methods: ANOVA and regression are among the parametric techniques that assume normality of residuals or sampling distributions. The CLT explains why these methods remain effective even when underlying data are not perfectly normal. We are often working with means or sums of many observations, which are approximately normal regardless of the original distribution.

   For instance, in linear regression we assume that residuals are normally distributed, but this assumption is really about the sampling distribution of coefficient estimates. Even if individual residuals are somewhat skewed, each coefficient estimate is essentially a weighted average of many data points. Thanks to the CLT, these averages tend toward normality as sample size increases. This is why regression analysis often produces reliable results even with moderately non-normal data, as long as the sample size is reasonably large and there are no extreme outliers.

Example: Central Limit Theorem in Action

This visualization demonstrates the remarkable power of the Central Limit Theorem. We start with a highly skewed exponential distribution and show how the distribution of sample means becomes increasingly normal as sample size grows. Even with samples as small as n=5, we see movement toward normality, and by n=30, the distribution of means is nearly perfectly normal despite the non-normal underlying population.

One more time in the most simple terms, the Central Limit Theorem tells us something remarkable: if you take many samples from any population (regardless of its original shape) and calculate the average of each sample, those averages will form a bell-shaped normal distribution. The larger your samples, the more bell-shaped the distribution of averages becomes. This is why the normal distribution shows up everywhere in statistics: we're often working with averages, and averages tend to be normally distributed even when the underlying data isn't.