Gaussian Mixture Models and Expectation-Maximization › Visualizing Gaussian Mixture Models

We begin with the data as generated, coloured by the true component. In a real problem those colours would be unavailable (the whole difficulty is that they are), but seeing them establishes what the model is trying to recover.

The two clouds overlap in the middle and their axes point in different directions, so no single distance threshold separates them cleanly. A circular cluster boundary is visibly the wrong shape for either cloud.

The fitted model replaces the truth with a density. The contours below show the total mixture density; the dashed contours show how each component alone would be shaped.

Each dashed ellipse is a radius-one Mahalanobis contour, not a universal 68% probability region. In two dimensions it encloses about 39% of its Gaussian component's probability. It has the orientation and aspect ratio of that component's covariance matrix: one is long and tilted steeply, the other is rounder and tilted the other way. Those shapes are estimated from the data, not supplied, and they are what a diagonal or spherical covariance type would be unable to represent.

The real output of the fit is not the contours but the responsibilities. Colouring each point by its posterior probability of belonging to component 1 produces a gradual transition through the overlap region instead of a step.

The colour gradient is broad around the overlap, while observations strongly favoured by either component have saturated colours. Points in the pale region have ambiguous component assignments under this fitted model; a downstream decision that treats them as firmly assigned is throwing away information the model has already computed.

K-means, fitted to the same data, has no such region. It produces a boundary and commits.

Comparing this with the previous figure shows where the two methods diverge: near the interface, k-means draws a straight line through points that the mixture model declines to commit on. The pictures summarize different fitted models. The mixture displays its conditional uncertainty; its visual smoothness does not establish that the model is correct.

Finally, inspect the optimization itself. This seeded run records increasing log-likelihood values that flatten; we check that observed behaviour rather than assuming it from the diagonal-addition update.

The exact EM proof explains why likelihood monitoring matters. Here diagonal regularization means we must also check for decreases; a flat right-hand end is a practical stopping signal. Note what the curve does not show: whether this plateau is the highest one available.

Different starting points give a practical check on that question, although finitely many starts cannot settle it. The figure below runs the same algorithm from five random initializations and overlays the trajectories.

The legend reports each run's final log-likelihood. Where the converged curves separate, the lower plateaus are poorer fitted solutions. The finite trajectories do not establish whether a plateau is an exact local optimum. When the curves do coincide, the data happened to be forgiving, and nothing in the algorithm guarantees that it will stay that way. This is the practical argument for n_init.