Linear and Quadratic Discriminant Analysis › Formula

We first combine Gaussian class densities with Bayes' rule, then examine which score terms cancel under a shared covariance. This separates the population decision rule from the covariance estimates and numerical calculations used to fit it.

A discriminant classifier begins with a generative thought experiment. Imagine that nature first chooses a class according to its prior probability, and then draws a feature vector from a probability density that belongs to that class. To classify a new observation, we reverse the order of that story: instead of asking "what does class $k$ tend to look like?", we ask "how plausible is this particular $\mathbf{x}$ under each class, after accounting for how common each class is?" That reversal is Bayes' rule, and the entire geometry of LDA and QDA follows from the density we choose.

The first modeling decision is therefore to describe each class by a full density rather than by a single prototype. An ordinary Euclidean nearest-centroid rule scores a point by its Euclidean distance from a class mean, so equal-length displacements have equal distance scores regardless of direction. A class density can instead describe one direction as tightly concentrated and another as broad or tilted. The multivariate Gaussian is the canonical choice for this job. It is specified by a center and a covariance, its level sets are ellipsoids, and its log-density is quadratic, which is exactly what makes the later decision rules tractable. We assume

$$p(\mathbf{x} \mid y = k) = \mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}_k, \boldsymbol{\Sigma}_k),$$

where $\mathcal{N}$ denotes the multivariate normal density

$$\mathcal{N}(\mathbf{x} \mid \boldsymbol{\mu}, \boldsymbol{\Sigma}) = (2\pi)^{-d/2} |\boldsymbol{\Sigma}|^{-1/2} \exp\!\left(-\tfrac{1}{2} (\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})\right).$$

where:

• $\boldsymbol{\mu}$: the mean vector of the Gaussian, which locates the center of the ellipsoidal cloud
• $\boldsymbol{\Sigma}$: the covariance matrix, which sets the shape and orientation of the cloud
• $|\boldsymbol{\Sigma}|$: the covariance determinant; its square root is proportional to the volume of an ellipsoid with fixed Mahalanobis radius
• $(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu})$: the squared Mahalanobis distance from $\mathbf{x}$ to $\boldsymbol{\mu}$
• $d$: the number of feature dimensions

The mean $\boldsymbol{\mu}_k$ locates the center of a class cloud, and the covariance $\boldsymbol{\Sigma}_k$ determines its shape and orientation. For ellipsoids with the same Mahalanobis radius, volume is proportional to $\sqrt{|\boldsymbol{\Sigma}|}$. A larger determinant therefore means a larger such ellipsoid and a lower Gaussian peak density. The quadratic form $(\mathbf{x} - \boldsymbol{\mu})^\top \boldsymbol{\Sigma}^{-1}(\mathbf{x} - \boldsymbol{\mu})$ is the squared Mahalanobis distance. It measures distance relative to the class covariance, so a large Euclidean displacement along a broad direction can still have a small standardized distance. The density decreases continuously with this distance. There is no single hard neighborhood boundary built into the Gaussian model.

A density alone is not yet a classifier, because classes may occur with different frequencies. Bayes' rule combines the class-conditional density with the prior:

$$p(y = k \mid \mathbf{x}) = \frac{p(\mathbf{x} \mid y = k)\, \pi_k}{p(\mathbf{x})},$$

where:

• $p(y = k \mid \mathbf{x})$: the posterior probability that the observation belongs to class $k$
• $p(\mathbf{x} \mid y = k)$: the class-conditional density of $\mathbf{x}$ given class $k$
• $\pi_k$: the prior probability of class $k$
• $p(\mathbf{x}) = \sum_{j=1}^K p(\mathbf{x} \mid y = j)\, \pi_j$: the marginal density of $\mathbf{x}$ under the generative model, shared by all classes and normalizing the posteriors to sum to one

The denominator is the same for every class, so it cannot change which class has the largest posterior probability. The rule "predict the class with the largest posterior probability" is therefore equivalent to "predict the class with the largest $p(\mathbf{x} \mid y = k)\, \pi_k$." This is a small algebraic observation with a large practical consequence: we never need to compute the marginal density in order to classify.

Comparing products of densities is awkward, and in high dimensions the products can underflow. Taking logarithms preserves the argmax because the logarithm is monotonic. So

$$\log\!\left[p(\mathbf{x} \mid y = k)\, \pi_k\right] = -\tfrac{d}{2} \log(2\pi) - \tfrac{1}{2} \log |\boldsymbol{\Sigma}_k| - \tfrac{1}{2} (\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}_k^{-1} (\mathbf{x} - \boldsymbol{\mu}_k) + \log \pi_k.$$

where:

• $\mathbf{x}$: the feature vector being classified
• $\boldsymbol{\mu}_k$: the mean vector of class $k$
• $\boldsymbol{\Sigma}_k$: the covariance matrix of class $k$
• $\pi_k$: the prior probability of class $k$
• $d$: the number of features
• $|\boldsymbol{\Sigma}_k|$: the determinant of the class covariance

The first term $-\tfrac{d}{2} \log(2\pi)$ is identical for every $k$ and drops out of any comparison. Dropping it, we obtain the discriminant function

$$\delta_k(\mathbf{x}) = -\tfrac{1}{2} (\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}_k^{-1} (\mathbf{x} - \boldsymbol{\mu}_k) - \tfrac{1}{2} \log |\boldsymbol{\Sigma}_k| + \log \pi_k,$$

where:

• $\delta_k(\mathbf{x})$: the discriminant score for class $k$ at point $\mathbf{x}$
• $\mathbf{x}$: the feature vector
• $\boldsymbol{\mu}_k$: the mean vector of class $k$
• $\boldsymbol{\Sigma}_k$: the covariance matrix of class $k$
• $\pi_k$: the prior probability of class $k$
• $(\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}_k^{-1} (\mathbf{x} - \boldsymbol{\mu}_k)$: the squared Mahalanobis distance from $\mathbf{x}$ to class $k$'s mean, in units of class $k$'s own covariance
• $|\boldsymbol{\Sigma}_k|$: the determinant of the class covariance, penalizing diffuse class clouds

The Bayes decision rule is then $\hat{y} = \arg\max_k \delta_k(\mathbf{x})$.

Each term has a clear interpretation, and it is worth reading them as a sequence of questions. The first term rewards closeness to the class mean, measured in the class's own Mahalanobis metric: does $\mathbf{x}$ fall inside the class's ellipsoidal neighborhood? The second term is a normalization penalty: at a fixed feature dimension, a larger covariance determinant lowers the Gaussian peak. The density at a particular point also depends on its Mahalanobis distance from the class mean. The third term rewards frequently observed classes, pulling the decision toward a prior that may be strong. Classification is thus a balance between geometric fit, concentration, and prior belief.

Two facts about this derivation are useful. First, adding the same quantity to every class score, even a quantity depending on $\mathbf{x}$, does not change the predicted class. Second, the difference between two class scores is exactly the corresponding log-posterior ratio, because all common terms cancel:

$$\log \frac{p(y = k \mid \mathbf{x})}{p(y = k' \mid \mathbf{x})} = \delta_k(\mathbf{x}) - \delta_{k'}(\mathbf{x}),$$

where:

• $k, k'$: the two classes being compared
• $\delta_k(\mathbf{x}), \delta_{k'}(\mathbf{x})$: the discriminant scores for classes $k$ and $k'$ at point $\mathbf{x}$, each defined up to the same additive constant
• $p(y = k \mid \mathbf{x}), p(y = k' \mid \mathbf{x})$: the posterior probabilities of classes $k$ and $k'$ given $\mathbf{x}$

The right-hand side is a score difference, and its zero set is the pairwise tie locus. With two classes, a change in the winning label can occur only at a tie; a degenerate tie need not separate two open decision regions. With more than two classes, a tie between $k$ and $k'$ contributes to their shared decision boundary only where their common score is at least as large as every other class score. We now examine how the covariance assumptions determine these score differences.

The LDA assumption is that all classes share the same covariance matrix: $\boldsymbol{\Sigma}_k = \boldsymbol{\Sigma}$ for all $k$. The shared-covariance condition is a model assumption about the data-generating process. It says that all class clouds have the same shape, orientation, and volume, and differ only in where they are centered and how common they are. Under that assumption, the Gaussian ellipsoids are translated copies of one another. The discriminant simplifies because the $-\tfrac{1}{2} \log |\boldsymbol{\Sigma}|$ term is the same for all classes and drops out:

$$\delta_k(\mathbf{x}) = -\tfrac{1}{2} (\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}_k) + \log \pi_k.$$

where:

• $\delta_k(\mathbf{x})$: the discriminant score for class $k$ at point $\mathbf{x}$
• $\mathbf{x}$: the feature vector being classified
• $\boldsymbol{\mu}_k$: the mean vector of class $k$
• $\boldsymbol{\Sigma}$: the shared covariance matrix common to all classes
• $\pi_k$: the prior probability of class $k$
• $(\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}_k)$: the squared Mahalanobis distance from $\mathbf{x}$ to class $k$'s mean in the shared metric

We now expand the quadratic form. In matrix notation,

$$(\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}^{-1} (\mathbf{x} - \boldsymbol{\mu}_k) = \mathbf{x}^\top \boldsymbol{\Sigma}^{-1} \mathbf{x} - 2\, \boldsymbol{\mu}_k^\top \boldsymbol{\Sigma}^{-1} \mathbf{x} + \boldsymbol{\mu}_k^\top \boldsymbol{\Sigma}^{-1} \boldsymbol{\mu}_k.$$

After multiplying the expansion by $-\tfrac{1}{2}$, the term $-\tfrac{1}{2}\mathbf{x}^\top\boldsymbol{\Sigma}^{-1}\mathbf{x}$ is identical for every class and can be dropped. The other terms retain their signs and coefficients; multiplying an entire score by a negative number would reverse its ranking. The shared quadratic term is the cancellation that removes curvature from the score differences. What remains is

$$\delta_k(\mathbf{x}) = \boldsymbol{\mu}_k^\top \boldsymbol{\Sigma}^{-1} \mathbf{x} - \tfrac{1}{2}\, \boldsymbol{\mu}_k^\top \boldsymbol{\Sigma}^{-1} \boldsymbol{\mu}_k + \log \pi_k.$$

This is a linear function of $\mathbf{x}$. Define

$$\mathbf{w}_k = \boldsymbol{\Sigma}^{-1} \boldsymbol{\mu}_k, \qquad b_k = -\tfrac{1}{2}\, \boldsymbol{\mu}_k^\top \boldsymbol{\Sigma}^{-1} \boldsymbol{\mu}_k + \log \pi_k,$$

where:

• $\mathbf{w}_k$: the covariance-adjusted weight vector for class $k$
• $b_k$: the class-$k$ bias (intercept), combining the mean-quadratic term and the log prior
• $\boldsymbol{\Sigma}, \boldsymbol{\mu}_k, \pi_k$: the shared covariance, class-$k$ mean, and class-$k$ prior, as defined earlier

so that $\delta_k(\mathbf{x}) = \mathbf{w}_k^\top \mathbf{x} + b_k$ is a linear score. For two classes, the decision boundary between them is $\delta_0(\mathbf{x}) = \delta_1(\mathbf{x})$, or

$$\mathbf{w}^\top \mathbf{x} + b = 0,$$

with

$$\mathbf{w} = \boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_0 - \boldsymbol{\mu}_1), \qquad b = -\tfrac{1}{2}(\boldsymbol{\mu}_0 + \boldsymbol{\mu}_1)^\top \boldsymbol{\Sigma}^{-1} (\boldsymbol{\mu}_0 - \boldsymbol{\mu}_1) + \log \frac{\pi_0}{\pi_1}.$$

where:

• $\mathbf{w}$: the covariance-adjusted normal to the boundary, pointing toward increasing class-0-versus-class-1 score
• $b$: the bias term that shifts the boundary based on the class means and priors
• $\boldsymbol{\Sigma}, \boldsymbol{\mu}_0, \boldsymbol{\mu}_1, \pi_0, \pi_1$: the shared covariance, class means, and class priors, as defined earlier

The sign convention matters: $\mathbf{w}^\top\mathbf{x} + b$ equals $\delta_0(\mathbf{x})-\delta_1(\mathbf{x})$, so a positive value favors class 0 after accounting for the priors. In general, $\mathbf{w}$ is not parallel to $\boldsymbol{\mu}_0-\boldsymbol{\mu}_1$. If the means differ and the priors are equal, the boundary is the hyperplane through their midpoint on which their Mahalanobis distances are equal. Increasing the prior of one class expands its winning region and moves this hyperplane toward the other class. If the means are identical, the normal is zero and the priors alone decide, or every point is tied when the priors are also equal.

Fisher's original formulation sought a linear combination $\mathbf{w}^\top\mathbf{x}$ maximizing the ratio of between-class to within-class variance. In the two-class case with nonsingular within-class covariance, the between-class scatter of the projected values is $(\mathbf{w}^\top(\boldsymbol{\mu}_0-\boldsymbol{\mu}_1))^2$ and the within-class scatter is $\mathbf{w}^\top\boldsymbol{\Sigma}\mathbf{w}$, so the ratio is

$$J(\mathbf{w}) = \frac{\bigl(\mathbf{w}^\top(\boldsymbol{\mu}_0-\boldsymbol{\mu}_1)\bigr)^2}{\mathbf{w}^\top\boldsymbol{\Sigma}\mathbf{w}},$$

where:

• $\mathbf{w}$: the direction onto which the data are projected
• $\boldsymbol{\mu}_0-\boldsymbol{\mu}_1$: the difference between the class means, setting the between-class separation
• $\boldsymbol{\Sigma}$: the shared (within-class) covariance, setting the within-class spread

When the class means differ, a maximizing nonzero direction is proportional to $\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_0-\boldsymbol{\mu}_1)$. Its sign and nonzero scale do not change the variance ratio. If the means coincide, $J(\mathbf w)=0$ for every nonzero direction, so this criterion does not select an informative direction. Under the shared-covariance Gaussian model with distinct means, the maximizing direction gives the generative classifier when paired with its prior-dependent threshold. The direction alone does not specify that threshold.

LDA has $Kd$ mean parameters and $\frac{d(d+1)}{2}$ shared symmetric-covariance parameters. Estimating the class priors adds $K-1$ free parameters under $\sum_k \pi_k=1$, giving $Kd+\frac{d(d+1)}{2}+(K-1)$ in total. For fixed $K$, the covariance term grows quadratically in $d$, while the mean term grows linearly. In the multiclass setting, LDA needs at most $\min(d,K-1)$ discriminant directions. The centered class means have at most $K-1$ independent directions, and applying an invertible covariance adjustment does not change this rank. The posterior-relevant directions in the original feature coordinates are the span of $\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_k-\boldsymbol{\mu}_{\mathrm{ref}})$, where $\boldsymbol{\mu}_{\mathrm{ref}}$ is one chosen reference-class mean. Projecting onto these directions retains the Gaussian LDA score differences; projecting onto the raw means need not.

QDA relaxes the shared-covariance assumption: each class has its own $\boldsymbol{\Sigma}_k$. This can describe one class that is narrow and another that is broad or differently oriented. The shared-covariance cancellation is no longer available in general. Some terms can still coincide for particular parameter choices, but the general score must retain the full form

$$\delta_k(\mathbf{x}) = -\tfrac{1}{2} (\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}_k^{-1} (\mathbf{x} - \boldsymbol{\mu}_k) - \tfrac{1}{2} \log |\boldsymbol{\Sigma}_k| + \log \pi_k.$$

where:

• $\delta_k(\mathbf{x})$: the discriminant score for class $k$ at point $\mathbf{x}$
• $\mathbf{x}$: the feature vector being classified
• $\boldsymbol{\mu}_k$: the mean vector of class $k$
• $\boldsymbol{\Sigma}_k$: the class-specific covariance matrix for class $k$
• $\pi_k$: the prior probability of class $k$
• $(\mathbf{x} - \boldsymbol{\mu}_k)^\top \boldsymbol{\Sigma}_k^{-1} (\mathbf{x} - \boldsymbol{\mu}_k)$: the squared Mahalanobis distance from $\mathbf{x}$ to class $k$'s mean, in class $k$'s own metric
• $|\boldsymbol{\Sigma}_k|$: the determinant of the class $k$ covariance, penalizing diffuse class clouds

When $\boldsymbol{\Sigma}_k^{-1}$ varies with $k$, the $\mathbf{x}^\top\boldsymbol{\Sigma}_k^{-1}\mathbf{x}$ contribution is class dependent and must be retained. The log-determinant penalty must likewise be retained whenever the covariance determinants differ; different covariance matrices can still have the same determinant. The general discriminant is quadratic in $\mathbf{x}$, with linear and second-order terms in its coordinates. Class-specific covariance therefore permits boundaries beyond a single flat cut, while particular parameter choices can still produce simpler or degenerate boundaries.

For two classes ($k = 0$ and $k = 1$), the boundary $\delta_0(\mathbf{x}) = \delta_1(\mathbf{x})$ is

$$-\tfrac{1}{2} \mathbf{x}^\top (\boldsymbol{\Sigma}_0^{-1} - \boldsymbol{\Sigma}_1^{-1}) \mathbf{x} + (\boldsymbol{\mu}_0^\top \boldsymbol{\Sigma}_0^{-1} - \boldsymbol{\mu}_1^\top \boldsymbol{\Sigma}_1^{-1}) \mathbf{x} + c_0 = 0,$$

which is the equation $\delta_0(\mathbf{x}) - \delta_1(\mathbf{x}) = 0$ written with the class-0 terms leading.

where:

• $\mathbf{x}$: the feature vector at which the boundary is evaluated
• $\boldsymbol{\Sigma}_0, \boldsymbol{\Sigma}_1$: the covariance matrices of classes 0 and 1 (both assumed invertible)
• $\boldsymbol{\mu}_0, \boldsymbol{\mu}_1$: the mean vectors of classes 0 and 1
• $c_0$: the constant term, defined explicitly below

The constant $c_0$ collects the terms that do not depend on $\mathbf{x}$. Writing out $\delta_0(\mathbf{x}) - \delta_1(\mathbf{x}) = 0$ in full (the form matching the displayed boundary equation) and keeping only the terms without $\mathbf{x}$ gives

$$c_0 = -\tfrac{1}{2} \boldsymbol{\mu}_0^\top \boldsymbol{\Sigma}_0^{-1} \boldsymbol{\mu}_0 + \tfrac{1}{2} \boldsymbol{\mu}_1^\top \boldsymbol{\Sigma}_1^{-1} \boldsymbol{\mu}_1 - \tfrac{1}{2} \log |\boldsymbol{\Sigma}_0| + \tfrac{1}{2} \log |\boldsymbol{\Sigma}_1| + \log \frac{\pi_0}{\pi_1}.$$

where:

• $\mathbf{x}$: the feature vector
• $\boldsymbol{\Sigma}_0, \boldsymbol{\Sigma}_1$: the covariance matrices of classes 0 and 1
• $\boldsymbol{\mu}_0, \boldsymbol{\mu}_1$: the class mean vectors
• $\pi_0, \pi_1$: the class prior probabilities
• $c_0$: the constant term collecting all contributions that do not depend on $\mathbf{x}$ (the mean-quadratic and log-determinant terms plus the log prior ratio)

The boundary equation has the form of a general quadratic in $\mathbf{x}$: the symmetric matrix coefficient of the second-order term is $-\tfrac{1}{2}(\boldsymbol{\Sigma}_0^{-1} - \boldsymbol{\Sigma}_1^{-1})$, the linear coefficient is $\boldsymbol{\mu}_0^\top \boldsymbol{\Sigma}_0^{-1} - \boldsymbol{\mu}_1^\top \boldsymbol{\Sigma}_1^{-1}$, and $c_0$ is the scalar constant.

In two dimensions, a quadratic tie equation can describe conic sections such as ellipses, hyperbolas, and parabolas; in higher dimensions, the corresponding objects are quadrics. The eigenstructure of $\boldsymbol{\Sigma}_0^{-1}-\boldsymbol{\Sigma}_1^{-1}$ controls the quadratic curvature, but the linear and constant terms also matter. Together they determine the boundary's location, whether it is degenerate, and whether it exists at all. Some QDA models produce bounded class regions, with a narrow class winning near its center and a broad class winning in the tails. QDA can also reduce to a linear rule when the quadratic difference vanishes.

To see where the constants come from, write $\delta_0(\mathbf{x}) - \delta_1(\mathbf{x})$ term by term, consistent with the $\delta_0 - \delta_1 = 0$ sign convention used above. The $\mathbf{x}$-independent contribution is

$$-\tfrac{1}{2} \boldsymbol{\mu}_0^\top \boldsymbol{\Sigma}_0^{-1} \boldsymbol{\mu}_0 + \tfrac{1}{2} \boldsymbol{\mu}_1^\top \boldsymbol{\Sigma}_1^{-1} \boldsymbol{\mu}_1 - \tfrac{1}{2} \log |\boldsymbol{\Sigma}_0| + \tfrac{1}{2} \log |\boldsymbol{\Sigma}_1| + \log \frac{\pi_0}{\pi_1},$$

matching $c_0$ above, while the $\mathbf{x}$-dependent part yields the quadratic and linear coefficients already stated.

The number of free parameters in QDA is $K \cdot d$ (means) plus $K - 1$ (priors) plus $K \cdot \frac{d(d+1)}{2}$ (covariances), so its covariance term grows $K$ times larger than LDA's. For independent Gaussian samples with a truly common covariance, the pooled covariance has lower entrywise sampling variance than an unbiased class estimate when that class has at least two observations and the other classes add residual degrees of freedom. Whether their flexibility improves held-out prediction depends on the population covariances and the available samples; parameter count alone does not determine the prediction error.

The covariance parameter count grows quadratically in the feature dimension and is multiplied by the number of classes for QDA. The next figure isolates that multiplier for a three-class problem, holding means and priors aside so the covariance cost is the only changing quantity.

The formulas above describe the population rule when the means, covariances, and priors are known. In practice, these parameters must be estimated. The maximum-likelihood class priors are the empirical proportions, $\hat{\pi}_k=N_k/N$, where $N_k$ counts observations in class $k$ and $N=\sum_j N_j$. The maximum-likelihood class means are $\hat{\boldsymbol{\mu}}_k=\frac{1}{N_k}\sum_{i:y_i=k}\mathbf{x}_i$. For covariance estimates, this chapter uses the unbiased sample convention rather than the Gaussian maximum-likelihood convention:

$$\hat{\boldsymbol{\Sigma}}_k = \frac{1}{N_k - 1} \sum_{i: y_i = k} (\mathbf{x}_i - \hat{\boldsymbol{\mu}}_k)(\mathbf{x}_i - \hat{\boldsymbol{\mu}}_k)^\top$$

where:

• $\hat{\boldsymbol{\Sigma}}_k$: the estimated covariance matrix of class $k$
• $N_k$: the number of observations in class $k$
• $\hat{\boldsymbol{\mu}}_k$: the empirical mean of class $k$
• $\mathbf{x}_i$: the $i$-th feature vector, associated with label $y_i$

The $N_k-1$ denominator gives the unbiased class covariance estimator for $N_k>1$. When the centered class scatter is positive definite, the Gaussian covariance maximum-likelihood estimator divides that same scatter by $N_k$. If the scatter is singular, no finite positive-definite covariance maximizes this unconstrained Gaussian density likelihood. The relative difference between the two normalization factors vanishes as $N_k$ grows. For LDA, the unbiased pooled covariance estimator is

$$\hat{\boldsymbol{\Sigma}} = \frac{1}{N - K} \sum_{k=1}^K \sum_{i: y_i = k} (\mathbf{x}_i - \hat{\boldsymbol{\mu}}_k)(\mathbf{x}_i - \hat{\boldsymbol{\mu}}_k)^\top,$$

where:

• $\hat{\boldsymbol{\Sigma}}$: the pooled estimate of the shared covariance matrix used by LDA
• $N = \sum_{j=1}^K N_j$: the total number of observations across all classes
• $K$: the number of classes
• $N_k$: the number of observations in class $k$
• $\hat{\boldsymbol{\mu}}_k$: the empirical mean of class $k$
• $\mathbf{x}_i$: the $i$-th feature vector, associated with label $y_i$

This pools the within-class scatter and divides by the residual degrees of freedom $N-K$. When the pooled scatter is positive definite, the shared-covariance Gaussian maximum-likelihood estimator instead divides that same scatter by $N$. If the pooled scatter is singular, the unconstrained positive-definite Gaussian covariance likelihood has no finite maximum. These are different finite-sample normalization conventions; the worked example and primitive implementations below consistently use the unbiased denominators.

Pooling is the estimation analogue of the shared-covariance assumption. Under independent Gaussian samples with a truly common covariance, the pooled estimate has $N-K$ residual degrees of freedom rather than one class's $N_k-1$. For a class with $N_k>1$, pooling has no larger covariance-entry sampling variance under that model; the reduction is strict when the other classes add residual degrees of freedom, so $N-K>N_k-1$. If class covariances differ, pooling instead estimates a compromise matrix.

For a centered class sample with $N_k>1$, $\operatorname{rank}(\hat{\boldsymbol{\Sigma}}_k)\le\min(d,N_k-1)$. Thus $1<N_k\le d$ forces singularity: for $d=100$ and $N_k=50$, there are at most 49 nonzero eigenvalues and the inverse does not exist. Having $N_k>d$ is necessary, but not sufficient, for nonsingularity. LDA's pooled scatter similarly has rank at most $\min(d,N-K)$. Even when these matrices are full rank, small samples and strong feature dependencies can make covariance estimation unreliable. Rank, conditioning, and estimation uncertainty are separate checks.

Shrinkage estimators regularize the covariance by blending the empirical estimate with a low-dimensional target:

$$\hat{\boldsymbol{\Sigma}}_k(\alpha) = (1 - \alpha) \hat{\boldsymbol{\Sigma}}_k + \alpha \mathbf{T},$$

where:

• $\hat{\boldsymbol{\Sigma}}_k(\alpha)$: the shrunk (regularized) covariance estimate for class $k$ at mixing level $\alpha$
• $\alpha$: the mixing (shrinkage) parameter in $[0, 1]$ controlling the bias-variance tradeoff; $\alpha = 0$ recovers the empirical covariance and $\alpha = 1$ yields the pure target
• $\hat{\boldsymbol{\Sigma}}_k$: the empirical covariance of class $k$
• $\mathbf{T}$: the shrinkage target, for example the identity matrix $\mathbf{I}$ or a scaled version $\sigma^2 \mathbf{I}$

At $\alpha=0$, the estimate is the empirical covariance; at $\alpha=1$, it is the target $\mathbf{T}$. If the empirical covariance is positive semidefinite and the target is positive definite, any $\alpha>0$ makes their convex blend positive definite. A shared identity or positive scaled-identity target is spherical. The Ledoit-Wolf (2004) estimator provides a data-driven shrinkage approach. It should not be conflated with manually choosing a regularization strength. In scikit-learn 1.8, QDA's default 'svd' solver uses reg_param; its 'eigen' solver instead supports shrinkage, including 'auto'. LDA supports shrinkage='auto' with 'lsqr' or 'eigen'. The solver and rank limitations are specified below.

Two further limitations matter. A sufficiently extreme observation can dominate an empirical covariance and move the fitted boundary. The empirical mean and covariance have unbounded influence even when the reference distribution is Gaussian. Moving a contaminating point farther away makes its influence grow without bound. Gaussianity therefore does not make these estimators robust to contamination. Robust covariance estimators address a different estimation problem and are outside this chapter's scope.

Discriminant analysis inherits several structural properties from its Gaussian foundation.

• Invertible affine transformations preserve unregularized LDA and QDA predictions in exact arithmetic when their estimates are transformed consistently. If $\mathbf{z}=A\mathbf{x}+\mathbf{b}$ with invertible $A$, then $\boldsymbol{\mu}'_k=A\boldsymbol{\mu}_k+\mathbf{b}$ and $\boldsymbol{\Sigma}'_k=A\boldsymbol{\Sigma}_kA^\top$. Mahalanobis distances are unchanged, while $|\boldsymbol{\Sigma}'_k|=|\det A|^2|\boldsymbol{\Sigma}_k|$. The full Gaussian discriminants therefore change by $-\log|\det A|$, common to every class. Reduced LDA scores, after dropping their shared quadratic term, can acquire a different common term depending on $\mathbf{x}$, but all score differences are still unchanged. This algebraic equivariance does not require Gaussian data; finite precision and a fixed regularization target can limit the invariance in practice.
• Posterior probabilities are given by the softmax of the discriminant scores (the logits):

$$p(y = k \mid \mathbf{x}) = \frac{\exp(\delta_k(\mathbf{x}))}{\sum_{j=1}^K \exp(\delta_j(\mathbf{x}))}.$$

This means LDA and QDA are not just classifiers but probabilistic models: the same machinery that predicts labels can also predict class probabilities. In practice, these probabilities can be miscalibrated if the Gaussian assumption fails, which motivates the calibration methods discussed in later parts of the book. In the implementation below, we compute the class posteriors directly from the discriminant scores via this softmax.

• Under the shared-covariance Gaussian model, the binary posterior is logistic in $\mathbf{x}$. In the earlier sign convention, $\delta_0-\delta_1=\mathbf{w}^\top\mathbf{x}+b$, so $p(y=1\mid\mathbf{x})=\sigma(-\mathbf{w}^\top\mathbf{x}-b)$, where $\sigma(t)=1/(1+\exp(-t))$. Equivalently, its class-1 logit has coefficient $\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_1-\boldsymbol{\mu}_0)$. Logistic regression fits this conditional linear-logit family directly. LDA obtains coefficients by fitting a joint Gaussian model; this imposes assumptions about feature distributions, rather than a smaller family of linear posterior logits.
• QDA does not generally satisfy a linear-logistic relationship. Its log-odds is quadratic in $\mathbf{x}$, so $p(y = 1 \mid \mathbf{x}) = \sigma(f(\mathbf{x}))$ holds with $f(\mathbf{x}) = \log(\pi_1/\pi_0) - \frac{1}{2}\log(|\boldsymbol{\Sigma}_1|/|\boldsymbol{\Sigma}_0|) - \frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_1)^\top\boldsymbol{\Sigma}_1^{-1}(\mathbf{x}-\boldsymbol{\mu}_1) + \frac{1}{2}(\mathbf{x}-\boldsymbol{\mu}_0)^\top\boldsymbol{\Sigma}_0^{-1}(\mathbf{x}-\boldsymbol{\mu}_0)$, a quadratic function of $\mathbf{x}$. The quadratic term enters before the logistic transformation, so the gradient of the log-odds generally varies affinely across feature space.