PCA (Principal Component Analysis) › Example

Let's work through PCA by hand on a tiny dataset. This will make the abstract formulas concrete. We'll calculate each step explicitly so you see exactly how the algorithm works.

Suppose we have four observations of two variables:

$$\mathbf{X} = \begin{bmatrix} 1 & 2 \\ 2 & 3 \\ 3 & 4 \\ 4 & 5 \end{bmatrix}$$

Each row is an observation, each column is a variable. Notice the strong relationship: the second variable is always one more than the first.

PCA requires mean-centered data. First, calculate the mean of each variable:

• Mean of variable 1: $\bar{x}_1 = \frac{1+2+3+4}{4} = 2.5$
• Mean of variable 2: $\bar{x}_2 = \frac{2+3+4+5}{4} = 3.5$

Now subtract these means from each observation:

$$\mathbf{X}_{\text{centered}} = \begin{bmatrix} 1-2.5 & 2-3.5 \\ 2-2.5 & 3-3.5 \\ 3-2.5 & 4-3.5 \\ 4-2.5 & 5-3.5 \end{bmatrix} = \begin{bmatrix} -1.5 & -1.5 \\ -0.5 & -0.5 \\ 0.5 & 0.5 \\ 1.5 & 1.5 \end{bmatrix}$$

Each column now has mean zero. The data is centered at the origin.

Next, compute the covariance matrix using $\mathbf{C} = \frac{1}{n-1} \mathbf{X}_{\text{centered}}^T \mathbf{X}_{\text{centered}}$. With $n=4$ observations, we divide by $n-1=3$:

$$\mathbf{C} = \frac{1}{3} \begin{bmatrix} -1.5 & -0.5 & 0.5 & 1.5 \\ -1.5 & -0.5 & 0.5 & 1.5 \end{bmatrix} \begin{bmatrix} -1.5 & -1.5 \\ -0.5 & -0.5 \\ 0.5 & 0.5 \\ 1.5 & 1.5 \end{bmatrix}$$

Let's calculate element by element:

• $C_{11} = \frac{1}{3}[(-1.5)^2 + (-0.5)^2 + (0.5)^2 + (1.5)^2] = \frac{1}{3}[2.25 + 0.25 + 0.25 + 2.25] = \frac{5}{3} \approx 1.67$

This is the variance of variable 1.

• $C_{12} = C_{21} = \frac{1}{3}[(-1.5)(-1.5) + (-0.5)(-0.5) + (0.5)(0.5) + (1.5)(1.5)] = \frac{1}{3}[2.25 + 0.25 + 0.25 + 2.25] = \frac{5}{3} \approx 1.67$

This is the covariance between variables 1 and 2. It's positive and large, confirming the strong positive relationship we saw.

• $C_{22} = \frac{1}{3}[(-1.5)^2 + (-0.5)^2 + (0.5)^2 + (1.5)^2] = \frac{5}{3} \approx 1.67$

This is the variance of variable 2.

The covariance matrix is:

$$\mathbf{C} = \begin{bmatrix} 1.67 & 1.67 \\ 1.67 & 1.67 \end{bmatrix}$$

Notice that the covariance equals the variances. This happens when two variables are perfectly correlated, as ours nearly are.

Now we solve the eigenvalue equation $\mathbf{C}\mathbf{w} = \lambda\mathbf{w}$. Rearranging, we get $(\mathbf{C} - \lambda\mathbf{I})\mathbf{w} = 0$, where $\mathbf{I}$ is the identity matrix.

For non-trivial solutions (where $\mathbf{w} \neq 0$), the matrix $\mathbf{C} - \lambda\mathbf{I}$ must be singular, meaning its determinant equals zero. This gives the characteristic equation:

$$\det(\mathbf{C} - \lambda\mathbf{I}) = 0$$

Substituting our covariance matrix:

$$\det\begin{bmatrix} 1.67-\lambda & 1.67 \\ 1.67 & 1.67-\lambda \end{bmatrix} = 0$$

The determinant of a 2×2 matrix $\begin{bmatrix} a & b \\ c & d \end{bmatrix}$ is $ad - bc$, so:

$$(1.67-\lambda)(1.67-\lambda) - (1.67)(1.67) = 0$$ $$(1.67-\lambda)^2 - (1.67)^2 = 0$$

This factors as a difference of squares:

$$[(1.67-\lambda) - 1.67][(1.67-\lambda) + 1.67] = 0$$ $$(-\lambda)(3.34 - \lambda) = 0$$

So $\lambda = 0$ or $\lambda = 3.34$. These are our two eigenvalues.

Now find the eigenvector for each eigenvalue.

For $\lambda_1 = 3.34$:

Substitute into $(\mathbf{C} - \lambda\mathbf{I})\mathbf{w} = 0$:

$$\begin{bmatrix} 1.67-3.34 & 1.67 \\ 1.67 & 1.67-3.34 \end{bmatrix}\begin{bmatrix} w_1 \\ w_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$$ $$\begin{bmatrix} -1.67 & 1.67 \\ 1.67 & -1.67 \end{bmatrix}\begin{bmatrix} w_1 \\ w_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$$

The first row gives $-1.67w_1 + 1.67w_2 = 0$, which simplifies to $w_1 = w_2$. So the eigenvector has equal components. To make it unit length, we need $w_1^2 + w_2^2 = 1$. Since $w_1 = w_2$, we have $2w_1^2 = 1$, so $w_1 = \frac{1}{\sqrt{2}}$:

$$\mathbf{w}_1 = \begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{bmatrix} \approx \begin{bmatrix} 0.707 \\ 0.707 \end{bmatrix}$$

This is the first principal component. It points equally in both variable directions, capturing their shared variation.

For $\lambda_2 = 0$:

$$\begin{bmatrix} 1.67 & 1.67 \\ 1.67 & 1.67 \end{bmatrix}\begin{bmatrix} w_1 \\ w_2 \end{bmatrix} = \begin{bmatrix} 0 \\ 0 \end{bmatrix}$$

This gives $1.67w_1 + 1.67w_2 = 0$, so $w_1 = -w_2$. With unit length:

$$\mathbf{w}_2 = \begin{bmatrix} \frac{1}{\sqrt{2}} \\ -\frac{1}{\sqrt{2}} \end{bmatrix} \approx \begin{bmatrix} 0.707 \\ -0.707 \end{bmatrix}$$

This is the second principal component. It points in the direction perpendicular to the first. Note that $\lambda_2 = 0$ means this direction has zero variance, which makes sense because our data lies along a perfect line.

Finally, project the data onto the principal components using $\mathbf{Y} = \mathbf{X}_{\text{centered}}\mathbf{W}$:

$$\mathbf{Y} = \begin{bmatrix} -1.5 & -1.5 \\ -0.5 & -0.5 \\ 0.5 & 0.5 \\ 1.5 & 1.5 \end{bmatrix} \begin{bmatrix} 0.707 & 0.707 \\ 0.707 & -0.707 \end{bmatrix}$$

Let's compute a few entries to see the pattern:

• $Y_{11} = (-1.5)(0.707) + (-1.5)(0.707) = -1.061 - 1.061 = -2.121$
• $Y_{12} = (-1.5)(0.707) + (-1.5)(-0.707) = -1.061 + 1.061 = 0$
• $Y_{21} = (-0.5)(0.707) + (-0.5)(0.707) = -0.354 - 0.354 = -0.707$
• $Y_{22} = (-0.5)(0.707) + (-0.5)(-0.707) = -0.354 + 0.354 = 0$

Following this pattern for all rows:

$$\mathbf{Y} = \begin{bmatrix} -2.121 & 0 \\ -0.707 & 0 \\ 0.707 & 0 \\ 2.121 & 0 \end{bmatrix}$$

Look at what happened. The second column (second principal component) is all zeros. This makes perfect sense. Our original data was perfectly correlated, lying along a single line. All the variance is captured by the first principal component (the direction along that line). The second principal component, perpendicular to that line, has no variance at all.

If you wanted to reduce dimensionality, you could keep only the first column and represent your data in one dimension instead of two, losing zero information.