Maximum Likelihood Estimation › Example

We now work through the likelihood calculations for two small examples. The first example is a Bernoulli problem with a count. The second is a Gaussian problem with five observations. In both cases, the same pattern appears. Build the likelihood, take its logarithm, differentiate, solve the score equation, then inspect curvature.

Suppose twenty visitors arrive, and twelve click. We model each visit as an independent Bernoulli trial with unknown click probability $p$. The observed count is $k=12$ and the number of trials is $n=20$. The likelihood is

$$L(p)=p^{12}(1-p)^{8}.$$

where:

• $L(p)$: the likelihood of the click probability $p$ given the observed counts
• $p$: the unknown click probability
• the exponents $12$ and $8$: the numbers of clicks and non-clicks, so the terms carry the success and failure counts

The log-likelihood is

$$\ell(p)=12\log p+8\log(1-p).$$

The score is

$$\ell'(p)=\frac{12}{p}-\frac{8}{1-p}.$$

Setting the score to zero gives

$$\frac{12}{p}=\frac{8}{1-p}.$$

Cross-multiplying gives $12(1-p)=8p$, so $12-12p=8p$, and therefore $12=20p$. Hence

$$\hat p=\frac{12}{20}=0.6.$$

The second derivative is

$$\ell''(p)=-\frac{12}{p^2}-\frac{8}{(1-p)^2}.$$

At $\hat p=0.6$, this is

$$\begin{aligned} \ell''(0.6) &= -\frac{12}{0.6^2} - \frac{8}{(1-0.6)^2} && \text{(substitute } \hat p = 0.6 \text{)} \\ &= -\frac{12}{0.36} - \frac{8}{0.16} && \text{(square the denominators)} \\ &= -33.\overline{3} - 50 && \text{(divide each term)} \\ &= -83.\overline{3} && \text{(add)}. \end{aligned}$$

where:

• $\ell''(0.6)$: the second derivative of the Bernoulli log-likelihood evaluated at $\hat p = 0.6$
• $\frac{12}{0.6^2}$: the magnitude of the negative second-derivative contribution from the 12 successes
• $\frac{8}{(1-0.6)^2}$: the magnitude of the negative second-derivative contribution from the 8 failures

The observed information is therefore $J(0.6) = 83.\overline{3}$, so the approximate standard error is

$$\widehat{\mathrm{se}}(\hat p) =\sqrt{\frac{1}{83.333}} \approx 0.1095.$$

where:

• $\widehat{\mathrm{se}}(\hat p)$: the approximate standard error of the Bernoulli estimate
• $83.333$: the observed information $J(0.6)$ rounded

This matches the formula $\sqrt{\hat p(1-\hat p)/n}=\sqrt{0.6\cdot0.4/20}=\sqrt{0.012}\approx0.1095$.

We can compare the log-likelihood at a few candidate values. At $p=0.5$,

$$\begin{aligned} \ell(0.5) &= 12\log(0.5) + 8\log(0.5) && \text{(substitute } p = 0.5 \text{)} \\ &= 20\log(0.5) && \text{(combine like terms)} \\ &\approx -13.863 && \text{(evaluate)}. \end{aligned}$$

At $p=0.6$,

$$\begin{aligned} \ell(0.6) &= 12\log(0.6) + 8\log(0.4) && \text{(substitute } p = 0.6 \text{)} \\ &\approx -13.460 && \text{(evaluate)}. \end{aligned}$$

At $p=0.7$,

$$\begin{aligned} \ell(0.7) &= 12\log(0.7) + 8\log(0.3) && \text{(substitute } p = 0.7 \text{)} \\ &\approx -13.912 && \text{(evaluate)}. \end{aligned}$$

The largest value is at $p=0.6$, as expected. The differences in log-likelihood are modest, which is another way of seeing that $p=0.5$ is not absurdly implausible with only twenty observations. A rough 95% Wald interval is $0.6\pm1.96(0.1095)$, or approximately $(0.385,0.815)$. The interval is symmetric around $0.6$ and can be unreliable near the boundaries, but it gives a useful first summary of uncertainty.

Suppose five readings from a sensor are

$$y=(9.8,\ 10.3,\ 9.9,\ 10.2,\ 9.8).$$

We model them as independent draws from $N(\mu,\sigma^2)$. The sample mean is

$$\hat\mu=\bar y=\frac{9.8+10.3+9.9+10.2+9.8}{5}=10.0.$$

where:

• $\hat\mu$: the maximum likelihood estimate of the mean
• $\bar y$: the sample mean
• the numerator: the sum of the five observed readings
• the denominator $5$: the number of observations $n$

The deviations from the mean are $-0.2,0.3,-0.1,0.2,-0.2$. Squaring and summing gives

$$\begin{aligned} \mathrm{SSE} &= (y_1-\bar y)^2 + (y_2-\bar y)^2 + (y_3-\bar y)^2 + (y_4-\bar y)^2 + (y_5-\bar y)^2 && \text{(definition)} \\ &= (-0.2)^2 + (0.3)^2 + (-0.1)^2 + (0.2)^2 + (-0.2)^2 && \text{(substitute deviations)} \\ &= 0.04 + 0.09 + 0.01 + 0.04 + 0.04 && \text{(square each term)} \\ &= 0.22. && \text{(sum)} \end{aligned}$$

where:

• $\mathrm{SSE}$: the sum of squared deviations of the readings from the sample mean
• each squared term: one observation's deviation from the mean, squared

The maximum likelihood variance is therefore

$$\hat\sigma^2=\frac{\mathrm{SSE}}{n}=\frac{0.22}{5}=0.044.$$

The maximum likelihood standard deviation is

$$\hat\sigma=\sqrt{0.044}\approx0.2098.$$

where:

• $\hat\sigma$: the maximum likelihood standard deviation
• $0.044$: the maximum likelihood variance estimate

For comparison, the unbiased sample variance divides by $n-1=4$:

$$\begin{aligned} s^2 &= \frac{\mathrm{SSE}}{n-1} = \frac{0.22}{4} = 0.055, \\ s &= \sqrt{s^2} = \sqrt{0.055} \approx 0.2345. \end{aligned}$$

where:

• $s^2$: the unbiased sample variance, which divides by $n-1$ rather than $n$
• $s$: the square root of the unbiased sample variance; $s$ itself is generally biased for $\sigma$
• $n-1 = 4$: the degrees of freedom, one less than the number of observations

The maximum likelihood variance is smaller because it divides by $n$. On average, it underestimates the true variance by a factor of $(n-1)/n=4/5$. With only five observations, the bias is not negligible.

The bias is easiest to see as a ratio. The maximum likelihood variance has expected value $(n-1)\sigma^2/n$, while the unbiased sample variance has expected value $\sigma^2$. The gap closes as $n$ increases.