Gaussian Mixture Models and Expectation-Maximization › Example

We can carry a full EM iteration by hand on four one-dimensional observations, $x = \{1.0,\ 2.5,\ 4.0,\ 6.0\}$, with two components. To keep the arithmetic transparent we fix the variances at $\sigma_1^2 = \sigma_2^2 = 1.5^2 = 2.25$ and the weights at $\pi_1 = \pi_2 = 0.5$ for the first E-step; the M-step then updates everything.

The normal density at $x$ is $\frac{1}{\sigma\sqrt{2\pi}}\exp\!\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)$, where $\mu$ is the mean and $\sigma^2$ is the variance, here $\mu_1 = 2.0$ and $\mu_2 = 5.0$ with $\sigma = 1.5$. With $\sigma = 1.5$ the leading constant is $\frac{1}{1.5\sqrt{2\pi}} \approx 0.26596$. Starting means are $\mu_1 = 2.0$ and $\mu_2 = 5.0$. For $x = 2.5$ the squared distances are $(2.5-2)^2 = 0.25$ and $(2.5-5)^2 = 6.25$; dividing by $2\sigma^2 = 4.5$ gives approximately $0.0556$ and $1.3889$. Evaluating the densities with the unrounded constant and exponents gives approximately $0.25159$ and $0.06632$. The responsibilities follow from the ratio:

Repeating for all four points gives the E-step below. The two outer points are almost certain, while $x = 4.0$ has a mixed posterior assignment, sitting at $\gamma_{3,1} = 0.3392$ (that is, $\gamma_{i1}$ is close to but below one half, so component 2 is slightly favoured). That contested point is useful because a nearest-centroid method still assigns it deterministically, while the mixture retains both possibilities under its fitted probability model.

The log-likelihood at this parameter value is the sum of the logs of the mixture densities. For $x = 1.0$ the mixture density is $\pi_1 \mathcal{N}(1.0 \mid \mu_1, 2.25) + \pi_2 \mathcal{N}(1.0 \mid \mu_2, 2.25) \approx 0.11028133$ using unrounded densities, whose natural logarithm is approximately $-2.20472063$. Summing the four contributions gives $\ell(\boldsymbol{\theta}^{0}) = -8.074$. That is our baseline. The exact M-step cannot lower it; equality is possible at a stationary configuration. In this example we can check the strict increase numerically.

Now the M-step. Effective counts are the column sums: $N_1 = 0.9656 + 0.7914 + 0.3392 + 0.0344 = 2.1306$ and $N_2 = 4 - 2.1306 = 1.8694$. These are not integer counts. They are the total fractional membership each component has earned and provide the denominators for the M-step's weighted averages. The new mean of component 1 is the responsibility-weighted average of the observations,

and similarly $\mu_2 = (13.5 - 4.5073)/1.8694 = 4.810$, where we used the fact that the two weighted sums must add to the total $\sum_i x_i = 13.5$. The variances use the same weights applied to squared deviations:

giving $\sigma_1 = 1.195$, and the same calculation for component 2 gives $\sigma_2^2 = 1.826$, so $\sigma_2 = 1.351$. The weights become $\pi_1 = 2.1306/4 = 0.533$ and $\pi_2 = 0.467$.

The test of the whole procedure is whether the likelihood improved. Recomputing the mixture density at the new parameters gives $\ell(\boldsymbol{\theta}^{1}) = -7.897$, which is larger than $-8.074$. The log-likelihood increases by about $0.177$ in this step, consistent with the nondecrease guarantee for this exact update. The contested point at $x = 4.0$ has also moved: its responsibility for component 2 increases from about $0.661$ to $0.692$. Both means move closer to this point. The posterior change depends on the updated weights and variances as well as the means; distances to the means alone do not determine it.

The cell below reproduces the hand calculation with the same formulas, so we can confirm the arithmetic and inspect the responsibilities directly.

The printed numbers agree with the rounded hand calculation to three decimal places, and the log-likelihood strictly increased. Further exact EM steps cannot lower it, but that alone does not guarantee a finite limiting fit. Our later implementation uses a small-change stopping rule and explicitly checks for a material likelihood decrease. The example therefore demonstrates the full rhythm of EM: infer soft labels, turn them into weighted estimates, and check that the objective moved in the promised direction.