Multicollinearity › Perfect Multicollinearity: The Extreme Case

Michael BrenndoerferApril 13, 20254 min read
Machine Learning from ScratchAll chapters
Part 3 of 9Foundations
Chapter 28Multicollinearity › Perfect Multicollinearity: The Extreme Case
Reading section 3 of 13 · Perfect Multicollinearity: The Extreme Case · 4 minSection 3 of 13 · 4 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 28 of 66

Section 3 of 13 · 4 min read

Multicollinearity › Perfect Multicollinearity: The Extreme CaseOne section · Sections in this chapter3 / 13 ↓
Sections in this chapter13
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Perfect multicollinearity occurs when one variable is a perfect linear combination of others. This is the most extreme form of multicollinearity, where the regression matrix becomes singular and cannot find unique solutions. The system cannot distinguish between variables that provide identical information.

In our health example, BMI is perfectly calculated from Height and Weight using the formula: BMI = weight/(height/100)². This creates perfect multicollinearity because BMI contains no information beyond what Height and Weight already provide.

Let's examine this extreme case:

Out[4]:
Visualization
Four-by-four correlation matrix for height, weight, BMI, and age, where BMI is a deterministic function of height and weight even though its pairwise correlations are below one.
Deterministic multicollinearity scenario where BMI is calculated directly from Height and Weight. Because this relationship is nonlinear, the pairwise correlations do not equal 1 even though BMI contains no independent information beyond its component variables.
Horizontal bar chart showing extreme VIF values for height, weight, and BMI, with the high-VIF threshold marked.
VIF values for the deterministic BMI relationship. Height, Weight, and BMI have extreme variance inflation, demonstrating why mathematical dependence among predictors makes individual regression effects unreliable.
Out[5]:
Console
Perfect Multicollinearity Analysis:
========================================
Correlation Matrix:
        Height  Weight    BMI    Age
Height   1.000     NaN -0.994  0.180
Weight     NaN     NaN    NaN    NaN
BMI     -0.994     NaN  1.000 -0.187
Age      0.180     NaN -0.187  1.000

VIF values:
Height: 90.9
Weight: 75175.8
BMI: 91.2
Age: 1.0

Note: Perfect multicollinearity makes regression analysis impossible
because X2 = 2*X1 + 3, creating a perfect linear relationship.