Linear and Quadratic Discriminant Analysis › Example

We work through a small one-dimensional example with concrete numbers so that every step is visible. The point of the example is not to show a realistic data analysis, but to make the discriminant scores themselves tangible: we can compute them by hand, compare the LDA and QDA decisions, and see exactly where the quadratic boundary comes from. Consider a two-class problem with the following samples:

$$\text{Class 0}:\ \{0, 1, 2, 3\}, \qquad \text{Class 1}:\ \{4, 6, 8, 10\}.$$

Both classes have four observations, so $\hat{\pi}_0 = \hat{\pi}_1 = 0.5$. Equal priors keep the comparison focused on the densities: any difference in the decision must come from the class means and variances, not from one class being more common than the other.

Step 1: compute means and variances.

For class 0, the mean is $\hat{\mu}_0 = (0 + 1 + 2 + 3)/4 = 1.5$, and the unbiased variance is

$$\hat{\sigma}_0^2 = \frac{1}{3}\sum_{i}\bigl(x_i - 1.5\bigr)^2 = \frac{(-1.5)^2 + (-0.5)^2 + (0.5)^2 + (1.5)^2}{3} = \frac{5}{3} \approx 1.667.$$

For class 1, the mean is $\hat{\mu}_1 = (4 + 6 + 8 + 10)/4 = 7$, and the unbiased variance is

$$\hat{\sigma}_1^2 = \frac{(-3)^2 + (-1)^2 + (1)^2 + (3)^2}{3} = \frac{20}{3} \approx 6.667.$$

The empirical variance of class 1 is four times that of class 0, by construction of this small example. These sample spreads make the two fitted covariance conventions differ visibly. Four observations per class do not establish the covariances of an underlying population, so we treat this as a worked calculation rather than evidence that a real population violates LDA's assumption.

Step 2: pooled variance for LDA.

The pooled variance divides the total within-class squared deviations by the degrees of freedom $N - K = 8 - 2 = 6$. The numerator is the sum of the within-class squared deviations: class 0 contributes $5$ and class 1 contributes $20$, for a total of $5 + 20 = 25$. Thus

$$\hat{\sigma}_{\text{pooled}}^2 = \frac{5 + 20}{6} = \frac{25}{6} \approx 4.167.$$

This pooled value is a compromise between the tight class 0 variance and the broad class 1 variance. It is the single shape that LDA will use for both classes.

Step 3: LDA discriminant scores.

Using the exact pooled estimate $\hat{\sigma}^2=25/6\approx4.167$, the one-dimensional linear discriminant is

$$\delta_k(x) = \frac{\hat{\mu}_k}{\hat{\sigma}^2}\, x - \frac{\hat{\mu}_k^2}{2 \hat{\sigma}^2} + \log \hat{\pi}_k$$

where:

• $\hat{\mu}_k$: the estimated mean of class $k$
• $\hat{\sigma}^2 = 25/6 \approx 4.167$: the pooled variance shared by both classes
• $\hat{\pi}_k = 0.5$: the estimated prior probability of class $k$
• $x$: the (scalar) feature value being classified

Working through the two classes:

$$\delta_0(x) = \frac{1.5}{25/6}x - \frac{2.25}{2 \cdot (25/6)} + \log(0.5) \approx 0.36 x - 0.27 - 0.693 = 0.36 x - 0.963,$$ $$\delta_1(x) = \frac{7}{25/6}x - \frac{49}{2 \cdot (25/6)} + \log(0.5) \approx 1.68 x - 5.88 - 0.693 = 1.68 x - 6.573.$$

At a test point $x = 3$:

$$\delta_0(3) \approx 1.08 - 0.963 = 0.117, \qquad \delta_1(3) \approx 5.04 - 6.573 = -1.533.$$

The larger score is $\delta_0$, so LDA predicts class 0. At a test point $x = 5$:

$$\delta_0(5) \approx 1.80 - 0.963 = 0.837, \qquad \delta_1(5) \approx 8.40 - 6.573 = 1.827.$$

Here LDA predicts class 1. The boundary between the two classes is where $\delta_0(x) = \delta_1(x)$, i.e., $0.36x - 0.963 = 1.68x - 6.573$. Solving: subtract $0.36x$ from both sides to get $-0.963 = 1.32x - 6.573$; add $6.573$ to both sides to get $5.61 = 1.32x$; divide by $1.32$ to get $x \approx 4.25$. As expected, this is the midpoint of the two class means, $\tfrac{1}{2}(1.5 + 7) = 4.25$. The equal-prior assumption removes any prior-driven shift, and the shared variance makes the boundary exactly halfway between the centers.

Step 4: QDA discriminant scores.

QDA keeps the class-specific variances. The discriminant is

$$\delta_k^{\text{QDA}}(x) = -\frac{(x - \hat{\mu}_k)^2}{2 \hat{\sigma}_k^2} - \tfrac{1}{2} \log \hat{\sigma}_k^2 + \log \hat{\pi}_k.$$

For class 0:

$$\delta_0^{\text{QDA}}(x) = -\frac{(x-1.5)^2}{2(5/3)}-\tfrac{1}{2}\log(5/3)+\log(0.5) \approx -0.3(x-1.5)^2-0.949.$$

where:

• $\hat{\mu}_0 = 1.5$: the estimated mean of class 0
• $\hat{\sigma}_0^2 = 5/3 \approx 1.667$: the unbiased estimated variance of class 0
• $\hat{\pi}_0 = 0.5$: the estimated prior probability of class 0
• $x$: the test point at which the class-0 QDA score is evaluated

For class 1:

$$\delta_1^{\text{QDA}}(x) = -\frac{(x-7)^2}{2(20/3)}-\tfrac{1}{2}\log(20/3)+\log(0.5) \approx -0.075(x-7)^2-1.642.$$

At $x = 3$:

$$\begin{aligned} \delta_0^{\text{QDA}}(3) &= -0.675-\tfrac{1}{2}\log(5/3)+\log(0.5) \approx -1.624,\\ \delta_1^{\text{QDA}}(3) &= -1.200-\tfrac{1}{2}\log(20/3)+\log(0.5) \approx -2.842. \end{aligned}$$

The quadratic contributions are exactly $-(3-1.5)^2/[2(5/3)]=-0.675$ for class 0 and $-(3-7)^2/[2(20/3)]=-1.200$ for class 1. The combined log-variance and log-prior constants are approximately $-0.948560$ and $-1.641707$, respectively; retaining them until the final rounding gives the scores above.

In each score, the quadratic contribution is minus one-half of the squared standardized distance. The remaining terms are the log-variance penalty and log prior. Both classes are evaluated at the same test point $x=3$.

QDA also predicts class 0 at $x=3$ because $-1.624>-2.842$. At $x=4.25$, where the exact LDA scores tie:

$$\delta_0^{\text{QDA}}(4.25) \approx -3.217, \qquad \delta_1^{\text{QDA}}(4.25) \approx -2.209.$$

QDA favors class 1 at $x=4.25$, while LDA is indifferent there; a claim about different hard labels at that exact point would require an LDA tie-breaking convention. This point is 2.75 raw units from both means. Relative to the class standard deviations, that is about 2.13 standard deviations for class 0 and 1.07 for class 1. The quadratic contributions are $-2.26875$ and $-0.5671875$. Class 1 pays the larger log-variance penalty, approximately $-0.949$ rather than $-0.255$, but its less negative quadratic contribution more than offsets that penalty. Thus the broader class has the higher QDA score at the midpoint.

Step 5: QDA decision boundary.

Solving $\delta_0^{\text{QDA}}(x)=\delta_1^{\text{QDA}}(x)$ gives a quadratic equation. We keep the exact variances and logarithms until the numerical evaluation:

$$-\frac{(x-1.5)^2}{10/3}-\frac{1}{2}\log(5/3)+\log(0.5) = -\frac{(x-7)^2}{40/3}-\frac{1}{2}\log(20/3)+\log(0.5).$$

The equal log-prior terms cancel. Multiplying by $40/3$ and collecting the log-variance difference gives

$$\begin{aligned} 4(x-1.5)^2-(x-7)^2 &=\frac{20}{3}\log\!\left(\frac{20/3}{5/3}\right)\\ &=\frac{20}{3}\log 4\\ &=\frac{40}{3}\log 2. \end{aligned}$$

The left-hand side expands to $4(x^2-3x+2.25)-(x^2-14x+49)=4x^2-12x+9-x^2+14x-49=3x^2+2x-40$. Define the boundary polynomial

$$P(x)=3x^2+2x-40-\frac{40}{3}\log 2.$$

Here $P(x)$ is the exact quadratic whose zeros are the QDA ties, and $\log$ is the natural logarithm. Since $\frac{40}{3}\log 2\approx9.241962$, the constant term is approximately $-49.241962$. The quadratic formula gives

$$\begin{aligned} x &=\frac{-2\pm\sqrt{\,4+12\left(40+\frac{40}{3}\log 2\right)\,}}{6}\\ &\approx -4.398441 \quad\text{or}\quad 3.731774. \end{aligned}$$

To determine the winning regions, subtract the exact class-0 score from the class-1 score:

$$\frac{40}{3}\left(\delta_1^{\text{QDA}}(x)-\delta_0^{\text{QDA}}(x)\right)=P(x).$$

The multiplier is positive, so the score difference has the same sign as $P(x)$. Its positive leading coefficient makes $P(x)$ negative between the roots and positive outside them. Class 0 therefore wins between the two roots, class 1 wins outside, and the endpoints are ties. The wider class wins in both far tails because its quadratic Mahalanobis contribution grows more slowly, while the class-0 penalty along its own metric dominates far from both centers. This two-tail allocation cannot be expressed by a single nondegenerate LDA threshold.

The worked example produces more than isolated labels: it produces two score curves whose crossings define the decision regions. The figure below evaluates the fitted LDA and QDA scores on a grid whose interval spans both QDA roots and the LDA midpoint. Compare the two classes within one method: each method may discard a different term common to its classes, so score heights across methods are not on a shared probability scale.