Linear and Quadratic Discriminant Analysis › Practical Implications

LDA and QDA are candidates when a single Gaussian is a plausible approximation to each observed class. The largest-posterior decision rule used here assumes equal misclassification costs; with unequal costs, choose the label that minimizes posterior expected loss. A single Gaussian cannot exactly represent a multimodal or asymmetric class density, although its hard-label boundary can still be correct. Unmeasured predictors can limit achievable performance for any classifier; omitting features does not by itself invalidate Gaussian class models for the observed features. The central choice between the two methods is whether a shared covariance is credible. Comparing the estimated class covariances by scale, orientation, and uncertainty is more informative than comparing eigenvalues alone, since equal eigenvalues can hide different orientations. Similar covariances support LDA's pooled estimate; material differences support QDA when each class has enough observations to estimate its own covariance. Because both are generative models, the comparison should end with held-out classification and probability checks, not with inspection of the fitted parameters alone.

LDA is also a supervised projection. Under the shared-covariance Gaussian model, posterior comparisons depend only on the span of the covariance-adjusted mean differences, a subspace of dimension at most $\min(d, K - 1)$. Projecting onto that subspace preserves the model's class-posterior information, whereas projecting onto the raw class means generally does not. This can be useful for visualization or for reducing inputs to a downstream classifier, but an accuracy improvement is not guaranteed, especially when the Gaussian model is misspecified. QDA can produce decision regions that are curved or even disjoint, so mapping the fitted regions is useful before deployment.

Start with LDA as a baseline, then fit QDA when the covariance estimates are stable and validation supports the extra flexibility. Use stratified cross-validation for model selection and a separate held-out set for the final comparison. For scikit-learn's default QDA solver when every class has at least $d$ observations, compare reg_param values on a grid such as 0, 0.01, 0.05, 0.1, 0.2, and 0.5; retain the unregularized candidate when it is feasible. When $2 \le N_k < d$, version 1.8's 'eigen' solver with suitable covariance shrinkage provides a different QDA candidate family. Every class still requires at least two observations. For LDA, shrinkage='auto' with solver='lsqr' or solver='eigen' provides a data-driven option, while manual shrinkage values can be tuned in the same way. Set priors explicitly when they are known; otherwise use the empirical priors and remember that they depend on the relative class sampling rates. Cross-validation can compare candidate regularization strengths, but a data-driven covariance estimator follows its own rule and should not be conflated with a manually tuned grid.

Inspect the fitted covariances before trusting the scores. Eigenvalues, condition numbers, and the success or failure of a Cholesky factorization are useful diagnostics; a factorization failure calls for checking rank and numerical range before choosing a remedy, such as rescaling, regularization, dimension reduction, or pooling. A large eigenvalue range is a warning to investigate, not a universal cutoff that mandates shrinkage, because it can reflect units, strong correlation, or estimation noise. Shrinkage has different endpoints that should be described accurately. Blending each class covariance toward a common pooled covariance approaches LDA at full shrinkage; blending toward identity makes the covariances spherical. A common spherical target yields ordinary Euclidean nearest-centroid classification when the priors are equal; unequal priors generally add class-specific score offsets. Class-specific spherical targets with different scales can retain quadratic boundaries. A common identity endpoint is a spherical member of the LDA model family, but it generally differs from the ordinary pooled-covariance LDA fit.

The methods require numeric features and labeled classes, and they assume that a Gaussian density is a reasonable description of each class. Use a task-specific precision target to plan per-class samples. Under independent, identically distributed Gaussian observations within a class, the unbiased variance estimate of one feature has relative standard error $\sqrt{2/(N_k-1)}$, meaning its standard deviation over repeated samples divided by that feature's true variance. This is approximately $14.2\%$ at $N_k=100$ and $10.0\%$ at $N_k=200$. These percentages concern a marginal variance; they do not guarantee accurate or well-conditioned full covariance estimates, classification, or calibration. Compare per-class covariance diagnostics and held-out classification and probability performance to decide whether more data, pooling, or shrinkage is needed. For QDA, N_k > d is necessary for a full-rank empirical covariance but not sufficient, since collinearity, outliers, and noise can still make the estimate singular or ill-conditioned. LDA's pooled covariance has the analogous requirement N - K >= d, with the same caveats. Imbalanced classes make the empirical priors reflect the sampling proportions; changing relative class sampling rates can change the empirical priors inferred from the resulting counts. Under a class-weighted generative likelihood, changing relative class weights can change the effective priors. Multiplying every class count or every class weight by the same positive factor preserves normalized proportions. If priors are supplied explicitly and held fixed, these operations do not change that supplied vector. Treat resampling and class weighting as deliberate modeling choices.

Standardization is not a mathematical prerequisite for exact unregularized affine equivariance, but it can improve numerical conditioning when feature units differ substantially. With shrinkage toward identity or a scaled identity, rescaling can change the target's geometry relative to the data and can change the fitted rule, especially under nonuniform feature rescaling. Uniform rescaling preserves equivariance for a trace-scaled identity target, $\mathbf T=(\operatorname{tr}\hat{\boldsymbol\Sigma}/d)\mathbf I$, when the shrinkage strength is held fixed and means and covariances are transformed consistently. Tune preprocessing and regularization together. Handle missing values before computing means and covariances; imputation can distort the covariance structure and should be checked rather than assumed harmless. A feature that is constant within one class makes that class covariance singular for QDA, and a feature constant within every class makes the pooled covariance singular for LDA. Strong collinearity and extreme outliers can also undermine the estimates; robust covariance estimation addresses a different problem and is outside the scope of this chapter.

A frequent mistake is fitting QDA when each class has too few observations relative to the feature dimension, then treating unstable or singular covariance estimates as meaningful. Another is choosing between LDA and QDA by training accuracy or by comparing eigenvalues alone; equal eigenvalues do not imply equal covariance matrices, and different orientations matter. Ignoring priors is also common: unequal class proportions change relative class scores and can move, create, or remove decision boundaries, while changing relative class sampling rates or class weights can change the empirical or effective priors inferred from the modified data. Common multipliers preserve normalized proportions, and explicitly fixed priors do not change automatically. Assuming that QDA always improves when covariances differ is another pitfall, because the extra covariance estimates can add more variance than the additional flexibility removes.

Identity shrinkage should not be confused with a path to the ordinary pooled-covariance LDA fit: a common identity endpoint belongs to the LDA model family, but generally uses a different shared covariance. Shrinking toward a common pooled covariance approaches LDA at full shrinkage, whereas shrinking toward identity changes the geometry of the class clouds. Evaluating only hard labels is also risky when probabilities are used downstream; noisy covariance estimates can leave accuracy acceptable while calibration degrades. Preprocessing steps such as scaling or imputation should be fit inside cross-validation, not before splitting, or the covariance estimates and performance can be optimistic. Before deployment, map the fitted decision regions, since QDA can assign the same class in disjoint regions.

For dense full-covariance matrix kernels, constructing the within-class scatter once class subsets are available costs O(N d^2). LDA then factors one shared covariance at O(d^3) and can form K linear weight vectors at O(K d^2); prediction costs O(K d). QDA factors K class covariances at O(K d^3), stores O(K d^2) covariance entries, and scores each observation in O(K d^2). These matrix-kernel bounds exclude the primitive fits' O(NK) repeated class-mask comparisons and their separate input validation and np.unique work. Solver choices and regularized covariance models can change the constants and, in some cases, the scaling.

Dense covariance storage grows quadratically with feature dimension, while dense covariance factorization costs grow cubically. Use Cholesky or SVD-based solves rather than forming explicit inverses, and consider dimensionality reduction, a diagonal covariance model, or shrinkage when d is large relative to the available samples. Covariance regularization can make a full-rank fit possible within the solver's stated domain, but it does not remove the dense covariance cost. There is no hardware-independent feature-count cutoff at which one method becomes impractical; benchmark the actual solver, regularization, and preprocessing pipeline. When the number of observations is large but memory is limited, sufficient statistics can compute the scatter matrices in one pass.

Evaluate the fitted model with stratified cross-validation and a final held-out test set. Accuracy and balanced accuracy summarize hard-label performance; log loss, the Brier score, and calibration curves assess probabilistic output; a confusion matrix shows which classes are confused. Compare LDA, QDA, and regularized variants against a simple baseline, and report uncertainty such as bootstrap confidence intervals when differences are small. If probabilities drive downstream decisions, check calibration on data not used to fit the classifier and recalibrate if necessary.

For deployment, store the class means, priors, and covariance factors rather than the raw training data. LDA can precompute its linear weights and intercepts for fast prediction, while QDA stores one factor per class. Monitor the input distribution, missingness, and class priors. Investigate detected drift and use task-relevant assessment to decide whether to refit the model or update its priors. When refitting after a population shift, check the newly estimated covariance conditioning. A fixed model's stored covariance factors do not change just because the incoming population changes. After fitting, scoring one observation reuses stored weights or factors and costs $O(Kd)$ for LDA or $O(Kd^2)$ for dense QDA. For $Q$ observations, total scoring work grows as $O(QKd)$ or $O(QKd^2)$, respectively, and can exceed the cost of one fit. Possible statistical risks include stale parameters, changed priors, and degraded calibration; monitor accuracy or balanced accuracy when only labels matter, and monitor log loss and calibration when probabilities matter.