PCA (Principal Component Analysis) › Implementation in Scikit-learn

Michael BrenndoerferJuly 21, 20253 min read
Machine Learning from ScratchAll chapters
Part 6 of 9Explainability
Chapter 48PCA (Principal Component Analysis) › Implementation in Scikit-learn
Reading section 6 of 9 · Implementation in Scikit-learn · 3 minSection 6 of 9 · 3 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 48 of 66

Section 6 of 9 · 3 min read

PCA (Principal Component Analysis) › Implementation in Scikit-learnOne section · Sections in this chapter6 / 9 ↓
Sections in this chapter9
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Now that you understand the theory, let's see how to use PCA in practice. Scikit-learn handles all the computational details, making PCA easy to apply to real datasets.

In[5]:
Code
from sklearn.datasets import load_wine

# Load the wine dataset
wine = load_wine()
X, y = wine.data, wine.target
feature_names = wine.feature_names

print(f"Original data shape: {X.shape}")
print(f"Number of features: {X.shape[1]}")
In[6]:
Code
# Standardize the data (important for PCA)
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA
pca = PCA()
X_pca = pca.fit_transform(X_scaled)

# Calculate explained variance ratios
explained_variance_ratio = pca.explained_variance_ratio_
cumulative_variance = np.cumsum(explained_variance_ratio)
Out[7]:
Console
Explained variance by each component:
PC 1: 0.362 ( 36.2%) - Cumulative: 0.362 ( 36.2%)
PC 2: 0.192 ( 19.2%) - Cumulative: 0.554 ( 55.4%)
PC 3: 0.111 ( 11.1%) - Cumulative: 0.665 ( 66.5%)
PC 4: 0.071 (  7.1%) - Cumulative: 0.736 ( 73.6%)
PC 5: 0.066 (  6.6%) - Cumulative: 0.802 ( 80.2%)
PC 6: 0.049 (  4.9%) - Cumulative: 0.851 ( 85.1%)
PC 7: 0.042 (  4.2%) - Cumulative: 0.893 ( 89.3%)
PC 8: 0.027 (  2.7%) - Cumulative: 0.920 ( 92.0%)
PC 9: 0.022 (  2.2%) - Cumulative: 0.942 ( 94.2%)
PC10: 0.019 (  1.9%) - Cumulative: 0.962 ( 96.2%)
PC11: 0.017 (  1.7%) - Cumulative: 0.979 ( 97.9%)
PC12: 0.013 (  1.3%) - Cumulative: 0.992 ( 99.2%)
PC13: 0.008 (  0.8%) - Cumulative: 1.000 (100.0%)
In[8]:
Code
# Find number of components for 95% variance
n_components_95 = np.argmax(cumulative_variance >= 0.95) + 1

# Apply PCA with specific number of components
pca_95 = PCA(n_components=n_components_95)
X_pca_95 = pca_95.fit_transform(X_scaled)

# Calculate variance retained
variance_retained = pca_95.explained_variance_ratio_.sum()
Out[9]:
Console

Number of components for 95% variance: 10
Reduced data shape: (178, 10)
Variance retained: 0.962 (96.2%)
Out[10]:
Console
Principal component loadings (first 5 components):
                                PC1    PC2    PC3    PC4    PC5
alcohol                       0.144  0.484 -0.207 -0.018 -0.266
malic_acid                   -0.245  0.225  0.089  0.537  0.035
ash                          -0.002  0.316  0.626 -0.214 -0.143
alcalinity_of_ash            -0.239 -0.011  0.612  0.061  0.066
magnesium                     0.142  0.300  0.131 -0.352  0.727
total_phenols                 0.395  0.065  0.146  0.198 -0.149
flavanoids                    0.423 -0.003  0.151  0.152 -0.109
nonflavanoid_phenols         -0.299  0.029  0.170 -0.203 -0.501
proanthocyanins               0.313  0.039  0.149  0.399  0.137
color_intensity              -0.089  0.530 -0.137  0.066 -0.076
hue                           0.297 -0.279  0.085 -0.428 -0.174
od280/od315_of_diluted_wines  0.376 -0.164  0.166  0.184 -0.101
proline                       0.287  0.365 -0.127 -0.232 -0.158
Out[11]:
Visualization
Scatter plot of wine dataset in PCA space with three colored clusters representing different wine classes.
Wine dataset projected onto first two principal components, colored by wine class. The plot shows how PCA separates the three wine varieties in the reduced dimensional space.

Key Parameters and Methods

Scikit-learn's PCA offers several useful parameters:

n_components: How many components to keep. You can specify this as:

  • An integer (e.g., n_components=3 keeps the first three components)
  • A float between 0 and 1 (e.g., n_components=0.95 keeps enough components to explain 95% of variance)
  • 'mle' for automatic selection using maximum likelihood estimation

whiten: Whether to scale components to have unit variance. This is useful when you plan to use the transformed data in algorithms that assume features have similar scales.

svd_solver: Which algorithm to use for the underlying singular value decomposition. Options include 'auto' (chooses based on data size), 'full' (exact SVD), 'arpack' (iterative for sparse data), and 'randomized' (fast approximation for large datasets).

In[12]:
Code
# PCA with specific number of components
pca_3 = PCA(n_components=3)
X_pca_3 = pca_3.fit_transform(X_scaled)
variance_3 = pca_3.explained_variance_ratio_.sum()

# PCA with variance threshold
pca_var = PCA(n_components=0.95)  # Keep 95% of variance
X_pca_var = pca_var.fit_transform(X_scaled)
variance_var = pca_var.explained_variance_ratio_.sum()

# Whitened PCA
pca_white = PCA(n_components=3, whiten=True)
X_pca_white = pca_white.fit_transform(X_scaled)
variance_white = pca_white.explained_variance_ratio_.sum()
Out[13]:
Console
Different PCA configurations:
3 components: (178, 3), variance: 0.665
95% variance: (178, 10), variance: 0.962
Whitened PCA: (178, 3), variance: 0.665