Multiple Linear Regression › Implementation
Machine Learning from ScratchAll chapters Multiple Linear Regression › ImplementationOne section · Sections in this chapter7 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
This section provides a step-by-step tutorial for implementing multiple linear regression using both Scikit-learn and NumPy. We'll start with a simple example to demonstrate the core concepts, then progress to a more realistic scenario that shows how to apply the method in practice.
Step 1: Basic Implementation with Scikit-learn
Let's begin by setting up our data and fitting a multiple linear regression model. We'll use the house price dataset from our mathematical example to demonstrate the core concepts.
import numpy as np
from sklearn.linear_model import LinearRegression
# Our house price dataset from the mathematical example
X = np.array([[2000, 1], [0, 3], [4000, 5]]) # Features: [size, age]
y = np.array([6, 10, 16]) # Target: price (in thousands)
# Create and fit the model
model = LinearRegression()
model.fit(X, y)
# Extract model parameters
intercept = model.intercept_
coefficients = model.coef_Now let's examine the fitted coefficients to understand what the model learned:
Model Coefficients: Intercept (β₀): 3.000 Size coefficient (β₁): 0.000333 Age coefficient (β₂): 2.333
The coefficients match our manual calculations exactly! The intercept of 3.0 represents the base price when both size and age are zero. The size coefficient of 0.000333 indicates that each additional square foot increases price by 2{,}333. This demonstrates how multiple linear regression captures the combined effect of multiple features on the target variable.
Step 2: Making Predictions and Evaluating Performance
Let's use the fitted model to make predictions and evaluate its performance:
# Make predictions on the training data
y_pred = model.predict(X)
# Calculate performance metrics
mse = mean_squared_error(y, y_pred)
r2 = r2_score(y, y_pred)Predictions vs Actual: House | Predicted | Actual | Error ------|-----------|--------|------ 1 | 6.0 | 6.0 | +0.0 2 | 10.0 | 10.0 | +0.0 3 | 16.0 | 16.0 | -0.0 Model Performance: Mean Squared Error: 0.000 R-squared: 1.000 Fitted Model: Price = 3.000 + 0.000333 × Size + 2.333 × Age
Perfect predictions with zero error! The of 1.0 indicates that the model explains 100% of the variance in the data. This is expected because we have exactly as many parameters (3) as data points (3), allowing the model to fit the data exactly. In practice with larger datasets, you would typically see some prediction errors due to noise and the inherent complexity of real-world relationships.
Step 3: Understanding the Mathematics with NumPy
For educational purposes, let's implement the multiple linear regression solution from scratch using NumPy to understand the underlying mathematics:
def multiple_linear_regression(X, y):
"""
Fit multiple linear regression using the closed-form OLS solution.
Parameters:
X: Feature matrix (n_samples, n_features)
y: Target vector (n_samples,)
Returns:
coefficients: Array of coefficients [intercept, feature1, feature2, ...]
"""
# Add column of ones for intercept
X_with_intercept = np.column_stack([np.ones(X.shape[0]), X])
# Compute OLS solution: β = (X'X)^(-1) X'y
XtX = X_with_intercept.T @ X_with_intercept
Xty = X_with_intercept.T @ y
coefficients = np.linalg.solve(XtX, Xty)
return coefficients
# Apply to our dataset
X_np = np.array([[2000, 1], [0, 3], [4000, 5]])
y_np = np.array([6, 10, 16])
coefficients_np = multiple_linear_regression(X_np, y_np)NumPy Implementation Results: Intercept: 3.000 Size coefficient: 0.000333 Age coefficient: 2.333
The NumPy implementation produces identical results to Scikit-learn, confirming that both methods use the same mathematical foundation. The np.linalg.solve() function is more numerically stable than computing the matrix inverse explicitly, making it the preferred approach for the multiple linear regression solution.
Step 4: Real-World Example with Larger Dataset
Let's demonstrate multiple linear regression with a more realistic dataset to show how it performs with real-world data:
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
# Generate a realistic dataset
X_large, y_large = make_regression(
n_samples=100, n_features=3, noise=10, random_state=42
)
# Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(
X_large, y_large, test_size=0.2, random_state=42
)
# Optional: Scale features for better numerical stability
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
# Fit model on scaled data
model_large = LinearRegression()
model_large.fit(X_train_scaled, y_train)
# Make predictions
y_train_pred = model_large.predict(X_train_scaled)
y_test_pred = model_large.predict(X_test_scaled)Practical Example Results: Training MSE: 92.84 Test MSE: 123.85 Training $R^2$: 0.986 Test $R^2$: 0.982 Model Coefficients: Intercept: 6.660 Feature 1: 23.325 Feature 2: 81.501 Feature 3: 18.323
This example shows how multiple linear regression performs with a more realistic dataset. The model achieves good performance on both training and test sets, with values around 0.95, indicating that the linear model explains most of the variance in the data. The small difference between training and test performance suggests the model generalizes well without overfitting.
Step 5: Model Interpretation and Validation
Let's create a visualization to better understand how our model performs:
The scatter plots show how well our model predicts the target values. Points close to the red diagonal line indicate accurate predictions. The similar performance on both training and test sets suggests our model generalizes well and doesn't suffer from overfitting.
Key Parameters
Below are the main parameters that affect how the multiple linear regression model works and performs.
fit_intercept: Whether to calculate the intercept for this model. If set to False, no intercept will be used in calculations (i.e., data is expected to be centered). Default: Truecopy_X: If True, X will be copied; else, it may be overwritten. Default: Truen_jobs: The number of jobs to use for the computation. This will only provide speedup for n_targets > 1 and sufficient large problems. Default: None (uses 1 processor)
Key Methods
The following are the most commonly used methods for interacting with the LinearRegression model.
fit(X, y): Fit linear model to training data X and target values y. Returns self for method chainingpredict(X): Predict target values for new data X using the fitted modelscore(X, y): Return the coefficient of determination of the prediction. Best possible score is 1.0get_params(): Get parameters for this estimator. Useful for hyperparameter tuningset_params(**params): Set the parameters of this estimator. Allows parameter modification after initialization