Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Multicollinearity › Detecting Multicollinearity: Correlation and VIF Analysis

Michael BrenndoerferApril 13, 20255 min read

Machine Learning from ScratchAll chapters

Part 3 of 9Foundations

Chapter 28Multicollinearity › Detecting Multicollinearity: Correlation and VIF Analysis

Reading section 2 of 13 · Detecting Multicollinearity: Correlation and VIF Analysis · 5 minSection 2 of 13 · 5 min

Section 2 of 13 · 5 min read

Multicollinearity › Detecting Multicollinearity: Correlation and VIF AnalysisOne section · Sections in this chapter2 / 13 ↓

Sections in this chapter13

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

The first step in understanding multicollinearity is learning to detect it. Multicollinearity reveals itself through the correlation matrix of predictor variables, which shows how strongly each pair of variables is related. The variance inflation factor (VIF) is our primary diagnostic tool for measuring multicollinearity severity.

The VIF is calculated using the coefficient of determination (R²):

\text{VIF}_j = \frac{1}{1 - R_j^2}

Here, $R_j^2$ represents how well variable $j$ can be predicted by other variables. When a variable can be perfectly predicted by others (R² = 1), the VIF becomes infinite, signaling perfect multicollinearity.

Understanding VIF values requires practical guidelines:

VIF < 5: Minimal multicollinearity (acceptable)
VIF 5-10: Moderate multicollinearity (warrants attention)
VIF > 10: High multicollinearity (requires action)

Let's see these concepts in action with a health dataset that demonstrates strong multicollinearity:

Out[3]:

Visualization

Heatmap displaying correlation coefficients between health variables like age, height, weight, BMI, blood pressure, cholesterol, and exercise. Strong correlations appear in red and blue, indicating multicollinearity patterns. — Correlation heatmap showing relationships between predictor variables. Dark red and blue colors indicate strong positive and negative correlations, while light colors suggest weak relationships. This visualization helps identify potential multicollinearity issues by highlighting highly correlated variable pairs.

Bar chart quantifying multicollinearity severity with variance inflation factor values. Variables exceeding 10 show high multicollinearity, while those between 5-10 indicate moderate multicollinearity. — VIF (Variance Inflation Factor) bar chart displaying multicollinearity severity for each predictor variable. Variables with VIF > 10 show high multicollinearity, while those with 5 < VIF ≤ 10 show moderate multicollinearity, indicating correlation issues that require attention.

Multicollinearity Analysis Results:
========================================
Maximum correlation: 0.416
Minimum correlation: -0.985
Maximum VIF: 14198.9
Variables with high VIF (>10): 6
Variables with moderate VIF (5-10): 0
Variables with low VIF (<5): 1