Multiple Linear Regression › Mathematical Foundation

Multiple linear regression extends the simple linear regression model to incorporate multiple predictor variables. The mathematical foundation builds upon the principles of ordinary least squares (OLS) estimation, which is covered in detail in the dedicated OLS chapter.

The multiple linear regression model can be written as:

$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_p x_{ip} + \epsilon_i$$

where:

• $y_i$: observed value of the target variable for the $i$-th observation
• $\beta_0$: intercept (bias term) - the predicted value of $y$ when all features equal zero
• $\beta_1, \beta_2, \ldots, \beta_p$: coefficients (slopes) for each feature, representing the partial effect of each feature
• $x_{i1}, x_{i2}, \ldots, x_{ip}$: values of the $p$ features for the $i$-th observation
• $\epsilon_i$: error term (residual) - the difference between the actual and predicted values

The key insight is that each coefficient $\beta_j$ represents the partial effect of feature $j$ on the target variable, holding all other features constant. This is the fundamental difference from simple linear regression, where we only consider one feature at a time.

When working with $n$ observations and $p$ features, the model can be written compactly using matrix notation:

$$\mathbf{y} = \mathbf{X} \boldsymbol{\beta} + \boldsymbol{\epsilon}$$

where:

• $\mathbf{y}$: $n \times 1$ vector containing all observed target values
• $\mathbf{X}$: $n \times (p+1)$ design matrix containing all feature values with a column of ones for the intercept
• $\boldsymbol{\beta}$: $(p+1) \times 1$ vector of coefficients (including intercept)
• $\boldsymbol{\epsilon}$: $n \times 1$ vector of error terms (residuals)

The design matrix $\mathbf{X}$ has the structure:

$$\mathbf{X} = \begin{bmatrix} 1 & x_{11} & x_{12} & \cdots & x_{1p} \\ 1 & x_{21} & x_{22} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_{n1} & x_{n2} & \cdots & x_{np} \end{bmatrix}$$

The first column of ones enables estimation of the intercept term, while subsequent columns contain the feature values for each observation.

The coefficients in multiple linear regression are estimated using the Ordinary Least Squares (OLS) method, which finds the values of $\beta_0, \beta_1, \ldots, \beta_p$ that minimize the sum of squared differences between observed and predicted values.

The OLS solution is given by the closed-form formula:

$$\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}$$

where:

• $\hat{\boldsymbol{\beta}}$: estimated coefficient vector (including intercept)
• $\mathbf{X}^\top \mathbf{X}$: $(p+1) \times (p+1)$ matrix of feature cross-products
• $(\mathbf{X}^\top \mathbf{X})^{-1}$: inverse of the cross-product matrix
• $\mathbf{X}^\top \mathbf{y}$: $(p+1) \times 1$ vector of feature-target covariances

For a detailed mathematical derivation of this formula, including the normal equations and computational considerations, see the dedicated OLS chapter.

Key PropertiesLink Copied

The OLS solution has several important properties:

• Unbiased: The estimates are unbiased under standard regression assumptions
• Efficient: Among all unbiased linear estimators, OLS has the smallest variance
• Consistent: As sample size increases, the estimates converge to the true values