Multicollinearity › Principal Component Regression (PCR): Transforming Correlated Variables

Michael BrenndoerferApril 13, 20256 min read
Machine Learning from ScratchAll chapters
Part 3 of 9Foundations
Chapter 28Multicollinearity › Principal Component Regression (PCR): Transforming Correlated Variables
Reading section 9 of 13 · Principal Component Regression (PCR): Transforming Correlated Variables · 6 minSection 9 of 13 · 6 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 28 of 66

Section 9 of 13 · 6 min read

Multicollinearity › Principal Component Regression (PCR): Transforming Correlated VariablesOne section · Sections in this chapter9 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Principal Component Regression (PCR) is another useful technique for handling multicollinearity. Instead of working with the original correlated variables, PCR maps them into a set of uncorrelated principal components that capture the most important information in the data.

The key insight is that principal components are orthogonal (uncorrelated) by construction, eliminating multicollinearity entirely. We can then use these components in regression analysis, often achieving better performance with fewer variables.

PCR works by:

  1. Standardizing the original variables
  2. Computing principal components that capture maximum variance
  3. Selecting the optimal number of components (often 95% of variance)
  4. Fitting regression using these uncorrelated components

This approach is particularly useful when we want to retain most of the original information while eliminating multicollinearity. Let's see how PCR performs with our health data:

Out[16]:
Visualization
Scree plot showing explained variance ratio for each principal component in descending order, helping identify the optimal number of components to retain.
Principal Component Analysis scree plot showing the proportion of variance explained by each component. The plot helps identify the optimal number of components to retain, balancing between capturing sufficient variance and avoiding overfitting. The elbow point indicates where additional components provide diminishing returns in terms of variance explained.
Cumulative explained variance plot with 95% threshold line, showing that multiple components are needed to capture 95% of total variance in multicollinear data.
Cumulative variance explained by principal components with 95% threshold line. This visualization shows how many components are needed to capture 95% of the total variance in the data. The red dashed line indicates the 95% threshold, helping determine the optimal number of components for Principal Component Regression while maintaining most of the original information.
Out[17]:
Visualization
Bar chart of original variable coefficients with VIF labels above each bar, showing R-squared and MSE metrics in the title.
Original variable coefficients from linear regression with VIF labels showing multicollinearity severity. High VIF values indicate variables that are highly correlated with other predictors, making coefficient estimates unstable and difficult to interpret.
Bar chart of principal component regression coefficients with reduced dimensionality, showing improved model stability metrics.
Principal Component Regression coefficients after dimensionality reduction. By using uncorrelated principal components, PCR eliminates multicollinearity while retaining most of the variance in the data, resulting in more stable and interpretable coefficient estimates.
Out[18]:
Console
Number of components to retain 95% variance: 4

Model Comparison:
============================================================
Metric               Original     PCR          Improvement 
------------------------------------------------------------
R-squared            0.532        0.532        -0.000      
MSE                  21.372       21.391       -0.019      
Max VIF              103.3        N/A          N/A         

VIF Analysis (Original Model):
Age: 1.3 (Low)
Height: 103.3 (High)
Weight: 3.2 (Low)
BMI: 99.4 (High)
Systolic_BP: 1.3 (Low)