Linear and Quadratic Discriminant Analysis › Advantages

With the true class means, priors, and shared positive-definite covariance, LDA is the Bayes-optimal classifier for Gaussian classes under zero-one loss. Here, "optimal" means minimizing the expected probability of misclassification under that generative model: at each feature vector, predicting the class with the largest true posterior minimizes conditional error. The Gaussian discriminant derivation explains this population rule. Estimating its parameters introduces uncertainty, so fitted LDA need not attain the Bayes error in a finite sample.

LDA shares covariance information across classes. It estimates one $d \times d$ covariance matrix, $K$ mean vectors, and $K - 1$ priors. For $d = 100$ features and two classes ($K = 2$), the covariance contributes $\frac{100(101)}{2} = 5{,}050$ distinct entries, the means contribute $2 \cdot 100 = 200$, and the priors contribute $K - 1 = 1$, for exactly $5{,}251$ free generative parameters. Ordinary binary logistic regression has $d + 1 = 101$ coefficients including its intercept; fitting two separate one-vs-rest models would store $2(d + 1) = 202$ coefficients, with redundant representations for two complementary classes. The larger generative parameter count does not by itself determine classification efficiency. Efron (1975) studied the asymptotic relative classification efficiency of logistic regression and normal discriminant analysis for two multivariate normal classes with different means and a common covariance. That comparison concerns asymptotic classification performance, rather than a guarantee of more accurate finite-sample coefficient estimates.

QDA gives the corresponding population Bayes rule for Gaussian classes with their own positive-definite covariances, provided the true parameters and priors are used. Fitting it requires $K$ full covariance estimates, each with $\frac{d(d+1)}{2}$ free parameters, instead of one. When class covariances differ, its quadratic score differences can represent curved decision boundaries that a linear classifier cannot express. For independent Gaussian samples with a truly common covariance, the pooled covariance has lower entrywise sampling variance than an unbiased class estimate when that class has at least two observations and the other classes add residual degrees of freedom.

For dense full-covariance matrix kernels, computing within-class scatter from $N$ observations once class subsets are available costs $O(Nd^2)$. QDA then factors $K$ covariance matrices at a cost of $O(Kd^3)$ and scores one new observation in $O(Kd^2)$. LDA factors a single shared covariance and can precompute its class weights: the additional work is $O(d^3 + Kd^2)$, after which a prediction costs $O(Kd)$. These are matrix-kernel bounds; solver choices can change them. The primitive fits below additionally scan all labels once per class, adding O(NK) class-mask comparisons, plus separate input validation and np.unique work. Large feature counts also impose quadratic covariance storage, so an adequate sample size alone does not make either dense method practical.