PCA (Principal Component Analysis) › Formula

Now let's build the mathematics of PCA step by step, starting from basic concepts you already know and working toward the full algorithm. The journey from variance to principal components is more natural than you might expect.

Think about a single variable, like the heights of people in a dataset. How spread out are the values? Variance answers this question:

$$\text{Var}(x) = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})^2$$

Here $\bar{x}$ is the mean (average) of $x$. For each data point, we calculate how far it is from the mean, square that distance, and average all the squared distances. The squaring ensures that deviations above and below the mean both contribute positively to the total spread.

Now consider two variables, height $x$ and weight $y$. Do they vary together? Covariance measures this:

$$\text{Cov}(x,y) = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y})$$

If tall people tend to be heavier, then when $x_i - \bar{x}$ is positive (taller than average), $y_i - \bar{y}$ tends to be positive too (heavier than average). Multiplying these together gives positive products, so the covariance is positive. If the variables move in opposite directions, covariance is negative. If they're unrelated, positive and negative products cancel out, giving covariance near zero.

When you have many variables, say $p$ of them, you need to track all pairwise relationships. The covariance matrix $\mathbf{C}$ does exactly this:

$$\mathbf{C} = \frac{1}{n-1} \mathbf{X}^T \mathbf{X}$$

Here $\mathbf{X}$ is your data matrix with $n$ rows (observations) and $p$ columns (variables). We assume you've already mean-centered the data, meaning you subtracted the mean from each column so each variable has mean zero.

What does this matrix contain? The diagonal elements are the variances of each individual variable. The off-diagonal elements are the covariances between pairs of variables. So if you have three variables, your covariance matrix looks like:

$$\mathbf{C} = \begin{bmatrix} \text{Var}(x_1) & \text{Cov}(x_1,x_2) & \text{Cov}(x_1,x_3) \\ \text{Cov}(x_2,x_1) & \text{Var}(x_2) & \text{Cov}(x_2,x_3) \\ \text{Cov}(x_3,x_1) & \text{Cov}(x_3,x_2) & \text{Var}(x_3) \end{bmatrix}$$

This matrix is symmetric because $\text{Cov}(x_i,x_j) = \text{Cov}(x_j,x_i)$. The covariance matrix captures all the linear relationships in your data in one compact object.

PCA answers a specific question: in which direction does the data vary the most?

Imagine drawing a line through your data cloud. You can project each point onto this line by dropping a perpendicular from the point to the line. The projected points form a one-dimensional dataset on the line. Some lines will give you spread-out projections (high variance), others will give you bunched-up projections (low variance). PCA finds the line that maximizes this variance.

Mathematically, a direction is a unit vector $\mathbf{w}$ (length 1). When you project your data $\mathbf{X}$ onto $\mathbf{w}$, you get $\mathbf{X}\mathbf{w}$. The variance of this projection is:

$$\text{Var}(\mathbf{X}\mathbf{w}) = \mathbf{w}^T \mathbf{C} \mathbf{w}$$

This compact formula says: the variance in direction $\mathbf{w}$ equals $\mathbf{w}^T \mathbf{C} \mathbf{w}$, where $\mathbf{C}$ is the covariance matrix. To find the direction of maximum variance, we need to maximize $\mathbf{w}^T \mathbf{C} \mathbf{w}$ subject to the constraint that $\mathbf{w}$ has unit length ($\mathbf{w}^T \mathbf{w} = 1$).

Why the constraint? Without it, we could make the variance arbitrarily large just by making $\mathbf{w}$ longer. Requiring unit length keeps the problem well-defined.

This is a constrained optimization problem. We solve it using Lagrange multipliers, a technique you may remember from calculus. We form the Lagrangian:

$$\mathcal{L} = \mathbf{w}^T \mathbf{C} \mathbf{w} - \lambda(\mathbf{w}^T \mathbf{w} - 1)$$

The term $\lambda(\mathbf{w}^T \mathbf{w} - 1)$ enforces our constraint. Taking the derivative with respect to $\mathbf{w}$ and setting it to zero:

$$\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = 2\mathbf{C}\mathbf{w} - 2\lambda\mathbf{w} = 0$$

Simplifying by dividing by 2:

$$\mathbf{C}\mathbf{w} = \lambda\mathbf{w}$$

This is the eigenvalue equation. The solutions $\mathbf{w}$ are the eigenvectors of $\mathbf{C}$, and the values $\lambda$ are the eigenvalues. What does this mean? The principal components are eigenvectors of the covariance matrix. The variance along each principal component is the corresponding eigenvalue. Larger eigenvalues mean more variance in that direction.

Once you've found all the eigenvectors, you can transform your data into the new coordinate system:

$$\mathbf{Y} = \mathbf{X}\mathbf{W}$$

Here $\mathbf{W}$ is a matrix whose columns are the eigenvectors of $\mathbf{C}$. Each column is a principal component (a direction of variance). The matrix $\mathbf{Y}$ contains your data in the new coordinate system. Each column of $\mathbf{Y}$ is a principal component score, telling you the position of each observation along that component.

The eigenvalues tell you how much variance each component captures. To find what proportion of total variance the $k$-th component explains:

$$\frac{\lambda_k}{\sum_{j=1}^{p} \lambda_j}$$

If the first eigenvalue is $\lambda_1 = 5$ and the sum of all eigenvalues is $10$, then the first principal component explains $5/10 = 50\%$ of the total variance. This helps you decide how many components to keep. If the first three components explain 95% of variance, you might drop the rest.

PCA has several mathematical properties worth knowing:

Orthogonality: Principal components are perpendicular (orthogonal) to each other. This means they capture different, non-overlapping patterns in the data. Each component adds new information rather than repeating what previous components captured.

Variance preservation: The total variance in the original data equals the sum of all eigenvalues. When you keep the first $k$ components, you capture the maximum possible variance among all sets of $k$ orthogonal directions. No other choice of $k$ perpendicular directions would capture more variance.

Optimality for reconstruction: When you project your data onto $k$ components and reconstruct it back in the original space, PCA minimizes the mean squared reconstruction error. Among all possible $k$-dimensional linear projections, PCA provides the best approximation to the original data.