Gaussian Mixture Models and Expectation-Maximization › Summary

A Gaussian mixture model describes data as a weighted sum of Gaussian densities and treats the component membership of each observation as a latent variable. The model is fitted by maximizing a likelihood that contains the log of a sum, which has no closed-form solution, so we use Expectation-Maximization: compute the posterior probability of each component for each observation (the E-step), then re-estimate the means, covariances, and weights as if those posterior probabilities were fractional counts (the M-step). The updates are weighted versions of the ordinary maximum likelihood estimators, with responsibilities standing in for the unknown indicators, and exact bound-increasing EM updates cannot decrease the objective because the E-step makes the lower bound tight. Our practical diagonal-addition safeguard requires a separate decrease check.

The soft assignments are the point of the exercise. Where k-means commits, the mixture reports a probability, and that probability carries information about ambiguity that a hard boundary discards. The same posterior drives the covariance updates, which is what allows each component to take on its own orientation and aspect ratio, and the resulting density can be evaluated, sampled, and compared across models.

For exact EM the likelihood guarantee is nondecrease, not global optimality or parameter convergence. The likelihood is non-concave, so initialization and restarts determine the answer, and unrestricted mixtures with $K\ge2$ have an unbounded sample likelihood, so a covariance constraint or regularization is needed to prevent collapse. Components are terms in a density, not discovered truths about the world, and the number of components is a model selection decision to be made with penalized criteria rather than training likelihood. For continuous data with roughly elliptical components, a Gaussian mixture gives us a probability density and model-based soft assignments alongside a hard partition.