Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Maximum Likelihood Estimation › Visualizing Maximum Likelihood

Michael BrenndoerferOctober 25, 20265 min read

Machine Learning from ScratchAll chapters

Part 3 of 15Model Fundamentals and Diagnostics

Chapter 29Maximum Likelihood Estimation › Visualizing Maximum Likelihood

Reading section 5 of 10 · Visualizing Maximum Likelihood · 5 minSection 5 of 10 · 5 min

Section 5 of 10 · 5 min read

Maximum Likelihood Estimation › Visualizing Maximum LikelihoodSection view · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

A likelihood curve shows how the plausibility of the observed data changes as we move the parameter. The peak is the maximum likelihood estimate. The width of the curve tells you how much information the data provide about the parameter. To compare curves with different sample sizes, we often shift each log-likelihood so that its maximum is zero. This removes an arbitrary additive constant and makes the shapes easier to compare.

For Bernoulli samples with the same interior observed proportion, the log-likelihood is concave and becomes narrower as sample size grows. The peak stays at the sample proportion, but the data become more informative, so small departures from the peak cost more log-likelihood. That is the visual counterpart of the standard error formula $\sqrt{\hat p(1-\hat p)/n}$ .

ConsoleOut [7]
CollapseExpandn = 5, k = 3, MLE p = 0.600
n = 20, k = 12, MLE p = 0.600
n = 100, k = 60, MLE p = 0.600

PythonIn [8]
CollapseExpandfrom book_plot_style import PALETTE, polish_axes, use_book_style

use_book_style()
fig, ax = plt.subplots(figsize=(6.0, 4.0))
plot_colors = [PALETTE["primary"], PALETTE["amber"], PALETTE["teal"]]
for (k, n, label), ll, color in zip(
    bernoulli_scenarios, bernoulli_curve_values, plot_colors
):
    ax.plot(p_grid_viz, ll, color=color, label=label)
ax.axvline(0.6, color=PALETTE["muted"], linestyle="--", linewidth=1.0)
ax.set_xlabel("Probability p")
ax.set_ylabel("Log-likelihood relative to maximum")
ax.set_title("More data sharpens the Bernoulli likelihood")
ax.legend()
polish_axes(ax)
plt.show()

VisualizationOut [8]
CollapseExpand

Three concave curves over the probability axis, all peaking at 0.6 and becoming narrower as the sample size increases from 5 to 20 to 100. — Bernoulli log-likelihoods for three sample sizes, shifted so each maximum is zero. The curves peak at the same observed proportion of 0.6 but become narrower as the sample size grows from 5 to 20 to 100, reflecting more information about the parameter.

The three curves use the same observed proportion, $k/n=0.6$ , but different sample sizes. The $n=5$ curve is broad, and many values of $p$ are nearly as plausible as the peak. The $n=100$ curve is much narrower, and the data are far more discriminating. This does not mean the estimate is necessarily correct. A narrow likelihood can still be centered on a biased value if the model is wrong. It does mean that, under the assumed model, the parameter is more precisely determined.

For the Gaussian mean, the log-likelihood as a function of $\mu$ is also concave when the variance is fixed. The peak is the sample mean, and the curvature is controlled by the variance and sample size. A larger variance or a smaller sample produces a flatter curve and a larger standard error.

The figure below shows the Gaussian log-likelihood for the mean with the variance held fixed at its maximum likelihood estimate. The concave curve peaks at the sample mean, and its width reflects the information the data provide about $\mu$ .

The readings from the sensor example, the fixed data used throughout this section, are stored once here so that the likelihood curves below can reuse them.

PythonIn [9]
CollapseExpandimport numpy as np

y = np.array([9.8, 10.3, 9.9, 10.2, 9.8])
gaussian_sigma2_viz = np.mean((y - y.mean()) ** 2)
mu_grid_viz = np.linspace(9.6, 10.4, 400)
gaussian_ll_mu = -len(y) / 2 * np.log(2 * np.pi * gaussian_sigma2_viz) - np.sum(
    (y[None, :] - mu_grid_viz[:, None]) ** 2, axis=1
) / (2 * gaussian_sigma2_viz)
gaussian_mu_hat_viz = y.mean()
gaussian_peak_loglik = -len(y) / 2 * np.log(
    2 * np.pi * gaussian_sigma2_viz
) - np.sum((y - gaussian_mu_hat_viz) ** 2) / (2 * gaussian_sigma2_viz)
gaussian_ll_mu_relative = gaussian_ll_mu - gaussian_peak_loglik

PythonIn [10]
CollapseExpandfrom book_plot_style import PALETTE, polish_axes, use_book_style

use_book_style()
fig, ax = plt.subplots(figsize=(6.0, 4.0))
ax.plot(
    mu_grid_viz,
    gaussian_ll_mu_relative,
    color=PALETTE["primary"],
)
ax.axvline(
    gaussian_mu_hat_viz, color=PALETTE["amber"], linestyle="--", linewidth=1.0
)
ax.set_xlabel("Mean mu")
ax.set_ylabel("Log-likelihood relative to maximum")
ax.set_title("The Gaussian mean likelihood peaks at the sample mean")
polish_axes(ax)
plt.show()

VisualizationOut [10]
CollapseExpand

A concave curve over possible mean values, peaking at the sample mean of 10.0. — Gaussian log-likelihood for the mean, with the variance fixed at its maximum likelihood estimate. The concave curve peaks at the sample mean of 10.0, and its width reflects how much information the five readings provide about the mean.

The plotted Gaussian log-likelihood is a downward parabola in $\mu$ , because the variance is held fixed. Its horizontal peak is the estimate and its curvature controls the local precision. Exponentiating the unshifted log-likelihood would give a bell-shaped likelihood, but neither plot by itself assigns posterior probabilities to $\mu$ . Bayesian parameter probabilities also require a prior and normalization.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Comments

Add to the conversation

Stay up to date