Linear and Quadratic Discriminant Analysis › Summary

Linear and quadratic discriminant analysis model each class as a multivariate Gaussian and compare its posterior probability. Bayes' rule turns the Gaussian log density into a Mahalanobis-distance term, a log-determinant penalty, and a log-prior term. With the true parameters and priors, choosing the largest score minimizes zero-one misclassification risk under that model; fitted versions estimate this rule. A shared covariance makes the log-determinant and feature-quadratic terms common to all classes, leaving linear score differences in LDA. Class-specific covariances retain quadratic score differences in QDA, which can produce curved or degenerate boundaries.

The two methods trade covariance flexibility against estimation uncertainty. LDA estimates one shared covariance, $K$ means, and $K-1$ priors, for $\frac{d(d+1)}{2}+Kd+(K-1)$ generative parameters. QDA estimates $K$ class-specific covariances, the same number of means, and the same number of priors, for $K\,\frac{d(d+1)}{2}+Kd+(K-1)$ parameters. Here $d$ is the feature dimension and $K$ the class count. QDA increases the covariance parameter count by a factor of $K$, rather than multiplying every parameter count by $K$. Shrinkage toward a common pooled covariance connects class-specific estimates to LDA; shrinkage toward spherical targets has different endpoints and must be interpreted with its target and prior assumptions stated.

Estimating covariances reliably becomes particularly difficult when the sample size per class is small relative to the feature dimension. When samples are small relative to dimension or features are dependent, rank deficiency, poor conditioning, and estimation uncertainty need separate attention; shrinkage is one possible response. If Gaussianity is violated by heavy tails, multimodality, or other shapes, the Bayes-optimality guarantee no longer follows from the Gaussian model, and probability calibration may suffer. The hard-label boundary can nevertheless still coincide with the true Bayes boundary, so misspecification does not prove classification failure. LDA and QDA make their distributional assumptions explicit; using them well requires checking the fitted estimates and evaluating predictions under the conditions that matter for the task.