Maximum Likelihood Estimation

Suppose we watch twenty visitors arrive at a website and twelve of them click a button. We want to estimate the underlying click probability. Or suppose a sensor returns five readings around 10 volts, and we want to estimate the true voltage and the measurement noise. In both cases we have data and a probability model with unknown parameters. Maximum likelihood estimation gives us a single, general recipe for turning those data into estimates. You choose the parameter values that make the observed data as plausible as possible under the model.

The likelihood is not the probability that a parameter is correct. Instead, we hold the observed data fixed and vary the parameter. For each candidate parameter value, we ask how likely the data would have been if that value had generated them. The maximum likelihood estimate, usually written $\hat\theta$, is the parameter value with the largest likelihood. This reverses our usual perspective. A probability distribution varies the data while its parameter is fixed; a likelihood varies the parameter while the observed data are fixed. Keeping that distinction clear prevents many later mistakes, especially when we compare likelihoods with posterior distributions or confidence intervals.

Taking a logarithm turns a product of many probabilities into a sum of many terms, which is easier to differentiate and far less prone to numerical underflow. The resulting log-likelihood is the central object in this chapter. It connects naturally to loss functions: under independent Gaussian noise with a common positive variance, maximizing the likelihood over the mean parameters is equivalent to minimizing the sum of squared errors we studied in Sum of Squared Errors (SSE). You will derive the main estimators by hand, inspect the curvature that produces standard errors, and then implement the same calculations with NumPy and SciPy. The ideas here also prepare us for the next chapter on gradient descent and numerical optimization, where we study iterative parameter fitting and numerical checks.