Linear and Quadratic Discriminant Analysis

In the figures below, modeling the same observations with a shared covariance gives a straight separator, while allowing separate covariances gives a curved one. Suppose we need to classify continuous observations into two or more groups, and we have a modest number of labeled examples. K-nearest neighbors classified observations by local proximity; naive Bayes modeled class-conditional feature distributions under conditional independence; logistic regression gave us a probabilistic model of $p(y \mid x)$ via a link function. Discriminant analysis takes a third route. Instead of modeling the label given the features, we model the features given the label: each class is assumed to be a multivariate Gaussian cloud, and we classify by asking which cloud most plausibly produced the observation.

When the class distributions are Gaussian and their means, covariances, and priors are known, this approach gives the Bayes-optimal rule under zero-one misclassification loss. In practice, LDA and QDA estimate those parameters, so a fitted classifier does not inherit an exact finite-sample optimality guarantee. Linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) differ in one covariance assumption: LDA uses a shared covariance and has linear score differences, while QDA allows class-specific covariances and quadratic score differences. Those quadratic differences can produce curved boundaries, including degenerate cases discussed below.

A name disclaimer is worth making early. In modern machine learning, "LDA" is also used as the acronym for latent Dirichlet allocation, a text-topic model in unsupervised learning. That model is unrelated to the discriminant analysis we develop here. In this chapter, LDA always means linear discriminant analysis. Fisher (1936) introduced a supervised projection criterion. For two Gaussian classes with distinct means and a shared positive-definite covariance, a maximizing Fisher direction is normal to the Bayes decision hyperplane.

Naive Bayes models class-conditional distributions under conditional independence. Here we allow features to be correlated within each class through a full covariance matrix. Whether those matrices are shared or class specific determines whether the score differences are linear or quadratic.

We will work throughout in the two-class setting for the core derivations and extend to $K$ classes when it makes the mathematics clearer. Notation is standard: $\mathbf{x} \in \mathbb{R}^d$ is the feature vector, $y \in \{1, 2, \ldots, K\}$ the class label, $\pi_k$ the prior probability of class $k$, $\boldsymbol{\mu}_k$ the class-conditional mean, and $\boldsymbol{\Sigma}_k$ the class-conditional covariance matrix. We assume $\pi_k > 0$ and symmetric positive-definite covariance matrices throughout the density-based derivations, so the logarithms, determinants, and inverses are defined. In the two-class derivations and examples, we label the classes 0 and 1.