Linear and Quadratic Discriminant Analysis › Visualizing LDA and QDA

We now illustrate the geometric differences on a synthetic two-class problem with deliberately different covariances per class. Both classes are drawn from two-dimensional Gaussians. Class 1 has larger marginal variances and a stronger positive feature correlation than class 0. This violates LDA's shared-covariance assumption. QDA can represent the population Gaussian Bayes boundary here, although fitting its extra covariance parameters does not guarantee higher training or test accuracy.

The following cells prepare the data, fit both reference classifiers, and compute the decision grid and class probabilities before plotting. For binary classification, decision_function returns the one-dimensional log-posterior ratio $\log p(y=1\mid\mathbf{x})-\log p(y=0\mid\mathbf{x})$. The common normalization cancels exactly in this difference. We place a zero column beside that ratio to make two class scores with the same difference, then apply softmax across the two columns to recover the posterior probabilities.

For this seeded training sample, the displayed accuracies show that QDA's higher value here is an observed result, not a guarantee; Gaussian likelihood fitting does not directly maximize classification accuracy, and a single training sample cannot establish a general ranking. Each mean posterior averages one class's fitted probability across all 160 training observations, and the following figures show both fitted decision regions on that same sample. An average near one-half is consistent with this balanced sample, but it does not measure confidence, class separation, or calibration: both confident predictions split between the classes and uniformly uncertain predictions can average to one-half. A held-out comparison below checks how both fitted rules classify new observations.

The first figure shows LDA's straight boundary, whose normal uses the pooled estimate $\hat{\boldsymbol{\Sigma}}$ and the class means, $\hat{\boldsymbol{\Sigma}}^{-1}(\hat{\boldsymbol{\mu}}_0-\hat{\boldsymbol{\mu}}_1)$. Its shared covariance is estimated from the within-class scatter of both classes. It is a compromise, rather than class 0's covariance imposed on class 1. LDA can represent correlated and tilted Gaussian clouds, but it forces both clouds to have the same estimated shape and orientation. The plot compares this shared-shape restriction with QDA's class-specific fitted shapes.

The second figure shows QDA. The boundary is now curved. QDA allows class 1 to "reach out" along its long axis, so points that fall inside the elongated direction of the class-1 cloud are more likely to be attributed to class 1. QDA's distance penalty is smaller along a class's broad covariance directions than along its narrow directions. The log-determinant penalty depends on the full covariance volume and stays constant as the observation moves. The price of this flexibility is that QDA estimates $K = 2$ full $2 \times 2$ covariances. In higher dimensions, the per-class covariance estimates are more sensitive to sample size. This tradeoff is the subject of the next section.