Linear and Quadratic Discriminant Analysis › Disadvantages

Skew, heavy tails, or multiple modes within a class can make a Gaussian density a poor approximation and can harm probability calibration or classification. LDA and QDA do not adapt their density family to represent arbitrary class shapes. A misspecified density can still give the correct hard-label boundary, so a failure of Gaussianity is a reason to examine held-out performance rather than proof that every prediction is wrong. These methods remain useful when their assumptions are plausible or as baselines for comparison.

Covariance estimation is a major difficulty when the feature dimension is large relative to the available observations per class. In high-dimensional settings, $d$ can rival the number of observations per class. A centered class sample has rank at most $N_k-1$. For $N_k>1$, its empirical covariance is singular when $N_k\le d$; with only one observation, the unbiased covariance is undefined. The ordinary unregularized QDA density formula is then undefined because the covariance inverse does not exist. Even when $N_k > d$, dependent features or estimation noise can leave the covariance singular or poorly conditioned. LDA pools within-class scatter across classes, but its pooled covariance can have rank no greater than $N - K$ and is subject to the same rank and conditioning checks.

For two classes, a nondegenerate LDA boundary is one hyperplane regardless of sample size. It cannot represent a bounded winning region surrounded by the other class or a winning region split across two separated tails. In the multiclass case, each pairwise tie equation is linear and each class winning region is an intersection of half-spaces. QDA can represent additional geometries, but it estimates $K\,\frac{d(d+1)}{2}$ covariance parameters rather than LDA's $\frac{d(d+1)}{2}$. Estimating these separate matrices can add uncertainty in small samples; both methods remain sensitive to outliers, and there is no universal ordering of their fitted errors.

Unregularized LDA and QDA are equivariant under invertible affine transformations when their means and covariances are transformed consistently and calculations are exact. This algebraic property does not require the observed data to be exactly Gaussian. Feature standardization can improve numerical conditioning when units differ substantially, but it does not remove collinearity or guarantee an accurate covariance estimate. Regularization toward a fixed identity target can also change the rule under rescaling, so preprocessing and shrinkage must be considered together.