Probability Basics › Law of Large Numbers
Machine Learning from ScratchAll chapters Probability Basics › Law of Large NumbersOne section · Sections in this chapter7 / 12 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
The Law of Large Numbers is a key result in probability theory, explaining why statistical methods work in practice. It states that as we collect more and more independent observations from a random process, the sample average converges to the expected value (theoretical mean) of the underlying distribution. This theorem justifies using sample statistics to estimate population parameters.
Intuitively, the Law of Large Numbers tells us that randomness "averages out" in the long run. If you flip a fair coin, the proportion of heads might deviate substantially from 50% in small samples, but as you flip the coin thousands or millions of times, the proportion gets arbitrarily close to 50%. This convergence isn't a guarantee that the sample mean will exactly equal the population mean for any finite sample size, but rather that the probability of large deviations decreases as the sample size increases.
This theorem has great importance in statistical practice. It underlies the validity of sample surveys, where we estimate population characteristics from samples. It justifies Monte Carlo simulation methods, where we approximate expectations by averaging over many random samples. It explains why casino gambling is profitable for the house in the long run, despite short-term variability. Every time you compute a sample mean and treat it as an estimate of a population mean, you're implicitly relying on the Law of Large Numbers.
Example: Demonstrating the Law of Large Numbers
This visualization shows how sample averages converge to the population mean as sample size increases. We simulate multiple sequences of coin flips (with probability 0.6 of heads) and track the cumulative average. Despite initial fluctuations, all paths converge toward the true expected value as more observations are collected.
