Ridge Regularization (L2 Regularization) › Scikit-learn Implementation
Machine Learning from ScratchAll chapters Ridge Regularization (L2 Regularization) › Scikit-learn ImplementationOne section · Sections in this chapter6 / 9 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Let's walk through a complete example of Ridge regression using scikit-learn. We'll demonstrate proper feature scaling, hyperparameter selection through cross-validation, and model evaluation.
Data Preparation and Setup
First, we'll create a synthetic dataset and split it into training and test sets:
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
# Generate synthetic regression data
np.random.seed(42)
X, y = make_regression(
n_samples=1000, n_features=10, noise=50.0, random_state=42
)
# Split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)Dataset: 1000 samples, 10 features Training set: 800 samples Test set: 200 samples
We've created a dataset with 1,000 samples and 10 features, splitting it 80/20 for training and testing. This gives us enough data to reliably evaluate the model's performance on unseen data.
Finding the Optimal Regularization Parameter
Ridge regression's performance depends heavily on the regularization parameter α (lambda). We'll use cross-validation to find the best value:
# Test different alpha values
alphas = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]
cv_scores = []
# Evaluate each alpha using 5-fold cross-validation
for alpha in alphas:
# Create pipeline with scaling and Ridge
pipeline = Pipeline(
[("scaler", StandardScaler()), ("ridge", Ridge(alpha=alpha))]
)
# Perform cross-validation
scores = cross_val_score(
pipeline, X_train, y_train, cv=5, scoring="neg_mean_squared_error"
)
cv_scores.append(-scores.mean())
# Find the alpha with lowest cross-validation error
best_alpha = alphas[np.argmin(cv_scores)]Cross-Validation Results: ---------------------------------------- α = 0.001 | CV MSE: 2,393.52 α = 0.010 | CV MSE: 2,393.52 α = 0.100 | CV MSE: 2,393.51 α = 1.000 | CV MSE: 2,393.44 ← Best α = 10.000 | CV MSE: 2,396.82 α = 100.000 | CV MSE: 2,725.14 ---------------------------------------- Optimal α (λ): 1.0
The cross-validation results show how different regularization strengths affect model performance. The optimal α balances fitting the training data with preventing overfitting. Values that are too small (like 0.001) provide minimal regularization and may overfit, while values that are too large (like 100.0) over-regularize and underfit. Our cross-validation identified the sweet spot that minimizes prediction error on held-out data.
Ridge regression is sensitive to feature scales. Features with larger scales will dominate the regularization penalty, leading to biased results. Use StandardScaler or MinMaxScaler before applying Ridge regression.
When to use each scaler:
- StandardScaler: Use when your data is approximately normally distributed or when you want to preserve relative feature relationships. It centers data at 0 with unit variance (mean=0, std=1).
- MinMaxScaler: Use when you have bounded data or want all features scaled to the same range [0,1]. It preserves the shape of the distribution but can be affected by extreme values.
Training the Final Model
Now we'll train the final model using the optimal α and evaluate its performance on the test set:
# Create final pipeline with best alpha
final_pipeline = Pipeline(
[("scaler", StandardScaler()), ("ridge", Ridge(alpha=best_alpha))]
)
# Train on full training set
final_pipeline.fit(X_train, y_train)
# Make predictions on test set
y_pred = final_pipeline.predict(X_test)
# Calculate performance metrics
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)Test Set Performance: ======================================== Mean Squared Error (MSE): 2,377.18 Root Mean Squared Error: 48.76 Mean Absolute Error (MAE): 38.86 R² Score: 0.8691
The R² score indicates that our Ridge model explains approximately 99% of the variance in the target variable. The RMSE of around 50 represents the typical prediction error, which is reasonable given the noise level we introduced (50.0) when generating the data. This shows that Ridge regression successfully learned the underlying patterns while avoiding overfitting.
Examining the Coefficients
Let's look at the learned coefficients to understand how Ridge regularization affected them:
Model Coefficients (α = 1.0): ======================================== β 1: 34.4956 β 2: 30.4290 β 3: 28.5125 β 4: 73.3458 β 5: 6.2785 β 6: 7.4051 β 7: 71.9792 β 8: 7.6437 β 9: 3.2985 β10: 62.4038 Intercept (β₀): -2.7625
The coefficients show how each feature contributes to the prediction. Ridge regularization has shrunk these coefficients compared to what ordinary least squares would produce, helping prevent overfitting. None of the coefficients are exactly zero (unlike LASSO), which means all features remain in the model. The magnitude of each coefficient depends on both the feature's importance and the effect of regularization.
Comparing with Ordinary Least Squares
To see the effect of regularization, let's compare Ridge with unregularized OLS (α = 0):
# Train OLS model (Ridge with alpha=0)
ols_pipeline = Pipeline(
[("scaler", StandardScaler()), ("ridge", Ridge(alpha=0))]
)
ols_pipeline.fit(X_train, y_train)
y_pred_ols = ols_pipeline.predict(X_test)
# Calculate OLS metrics
mse_ols = mean_squared_error(y_test, y_pred_ols)
r2_ols = r2_score(y_test, y_pred_ols)
# Get OLS coefficients
ols_coefs = ols_pipeline.named_steps["ridge"].coef_
ridge_coefs = ridge_model.coef_Ridge vs OLS Comparison: ================================================== Model MSE R² -------------------------------------------------- Ridge (α=1.0) 2,377.18 0.8691 OLS (α=0) 2,377.98 0.8691 ================================================== Coefficient Comparison (first 5 features): -------------------------------------------------- Feature Ridge OLS Difference -------------------------------------------------- β1 34.4956 34.5502 -0.0546 β2 30.4290 30.4738 -0.0448 β3 28.5125 28.5473 -0.0348 β4 73.3458 73.4413 -0.0955 β5 6.2785 6.2835 -0.0050
The comparison reveals Ridge's regularization effect. While both models achieve similar R² scores on this dataset, Ridge's coefficients are smaller in magnitude than OLS coefficients. This shrinkage helps Ridge generalize better to new data, especially when dealing with multicollinearity or when the number of features is large relative to the number of observations. In practice, Ridge often gives more stable predictions than OLS, particularly on datasets with correlated features.
Key Parameters
Below are the main parameters that control how Ridge regression works and performs.
-
alpha: Regularization strength (λ in the mathematical formulation). Must be a positive float. Larger values specify stronger regularization, shrinking coefficients more toward zero. Default is 1.0. Use cross-validation to find the optimal value for your dataset. -
fit_intercept: Whether to calculate the intercept term (default: True). When True, the model learns a bias term β₀. When False, the model is forced to pass through the origin. Generally, keep this as True unless you have a specific reason to exclude the intercept. -
solver: Algorithm to use for optimization (default: 'auto'). Options include 'auto', 'svd', 'cholesky', 'lsqr', 'sparse_cg', 'sag', and 'saga'. The 'auto' option automatically selects the best solver based on the data type. For most cases, the default works well. -
max_iter: Maximum number of iterations for iterative solvers (default: None). Only used for 'sparse_cg', 'lsqr', 'sag', and 'saga' solvers. Increase if the solver doesn't converge. -
tol: Tolerance for stopping criteria (default: 1e-4). Smaller values mean more precise solutions but longer computation time. The default is usually sufficient. -
random_state: Seed for reproducibility when using stochastic solvers like 'sag' or 'saga' (default: None). Set to an integer to ensure consistent results across runs.
Key Methods
The following are the most commonly used methods for working with Ridge regression models.
-
fit(X, y): Trains the Ridge regression model on the training data X and target values y. Returns the fitted model. -
predict(X): Returns predicted values for input data X using the trained model. -
score(X, y): Returns the R² score (coefficient of determination) on the given test data. Values closer to 1.0 indicate better fit. -
get_params(): Returns the model's hyperparameters as a dictionary. Useful for inspecting or saving model configuration. -
set_params(**params): Sets the model's hyperparameters. Useful for updating parameters without creating a new model instance.