Principal Component Analysis and Factor Extraction › Implementing PCA from Scratch
Quantitative FinanceAll chapters Principal Component Analysis and Factor Extraction › Implementing PCA from ScratchOne section · Sections in this chapter4 / 13 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's implement PCA step by step to solidify the mathematical concepts. We'll use a simple example before moving to financial applications.
import numpy as np
np.random.seed(42)
# Generate correlated data: two underlying factors drive three observed variables
n_obs = 200
factor1 = np.random.randn(n_obs)
factor2 = np.random.randn(n_obs)
# Create three observed variables as combinations of factors plus noise
x1 = 2 * factor1 + 0.5 * factor2 + 0.3 * np.random.randn(n_obs)
x2 = 1.5 * factor1 - 0.8 * factor2 + 0.3 * np.random.randn(n_obs)
x3 = 1.8 * factor1 + 0.3 * factor2 + 0.3 * np.random.randn(n_obs)
# Stack into data matrix
X = np.column_stack([x1, x2, x3])Now we center the data and compute the covariance matrix:
# Center the data by subtracting column means
X_centered = X - X.mean(axis=0)
# Compute sample covariance matrix
cov_matrix = np.cov(X_centered, rowvar=False)Sample Covariance Matrix: [[3.8158 2.1197 3.2931] [2.1197 2.5623 2.0789] [3.2931 2.0789 3.0186]]
The covariance matrix shows strong positive correlations among all three variables, consistent with a common factor structure.
Next, we compute eigenvalues and eigenvectors:
# Compute eigendecomposition
eigenvalues, eigenvectors = np.linalg.eigh(cov_matrix)
# Sort by decreasing eigenvalue (eigh returns in ascending order)
sort_idx = np.argsort(eigenvalues)[::-1]
eigenvalues = eigenvalues[sort_idx]
eigenvectors = eigenvectors[:, sort_idx]
# Calculate variance explained
total_var = eigenvalues.sum()
explained_variance_pct = 100 * eigenvalues / total_var
cumulative_variance_pct = np.cumsum(explained_variance_pct)Eigenvalues (variance of each PC): PC1: 8.2744 PC2: 1.0416 PC3: 0.0806 Variance explained by each component: PC1: 88.1% PC2: 11.1% PC3: 0.9% Cumulative variance explained: First 1 PCs: 88.1% First 2 PCs: 99.1% First 3 PCs: 100.0%
The first principal component captures the dominant common movement across all three variables, explaining about 85% of total variance. The second captures most of the remaining variance, leaving very little for the third. This confirms that our three observed variables are effectively driven by two underlying factors.
The eigenvectors (loading vectors) tell us how each original variable contributes to each component:
Principal Component Loadings:
PC1 PC2 PC3
x1: -0.6585 0.4351 -0.6141
x2: -0.4609 -0.8782 -0.1279
x3: -0.5949 0.1988 0.7788The first component has positive loadings on all variables, representing their common factor. The second component has loadings of opposite signs, capturing the differential between certain variables. This corresponds to the second factor in our data-generating process.
Finally, we project the data onto the principal components:
# Project data onto principal components
pc_scores = X_centered @ eigenvectors
# Calculate statistics for verification
pc_means = pc_scores.mean(axis=0)
pc_stds = pc_scores.std(axis=0)
pc_corr = np.corrcoef(pc_scores, rowvar=False)Principal Component Statistics: PC1 - Mean: 0.000000, Std: 2.8693 PC2 - Mean: 0.000000, Std: 1.0180 PC3 - Mean: -0.000000, Std: 0.2832 Correlation between PCs: Corr(PC1, PC2): -0.000000 Corr(PC1, PC3): -0.000000 Corr(PC2, PC3): 0.000000
The principal components are uncorrelated by construction, and their standard deviations equal the square roots of the corresponding eigenvalues.
