Maximum Likelihood Estimation › Summary

Maximum likelihood estimation chooses the parameter values that make the observed data most plausible under a probability model. Under independence, the likelihood is a product of per-observation masses or densities, and the log-likelihood is the sum of their logarithms. We maximize the log-likelihood, or equivalently minimize the negative log-likelihood. Logs are monotone, numerically stable, and easy to differentiate. The likelihood is not a probability distribution over parameters; it is a function of the parameter with the data held fixed.

For Bernoulli data, the maximum likelihood estimate is $\hat p=k/n$ on $[0,1]$; the Wald standard error $\sqrt{\hat p(1-\hat p)/n}$ is an interior large-sample approximation. For a nondegenerate i.i.d. Gaussian sample, the maximum likelihood mean is the sample mean and the variance is $\mathrm{SSE}/n$, with downward finite-sample bias. If $\mathrm{SSE}=0$, no positive-variance Gaussian MLE is attained. Independent Gaussian errors with common variance make likelihood maximization over regression coefficients equivalent to minimizing squared error.

The score and curvature provide both optimization equations and uncertainty estimates. The score is zero at an interior maximum, and the negative second derivative measures information. Identifiability determines whether the parameters can be distinguished at all, while sampling uncertainty describes how much the estimate would vary across repeated samples. Maximum likelihood is general, but it is only as trustworthy as the model, the data, and the numerical checks behind it.