Multiple Linear Regression › Example

Let's work through a concrete example to see how multiple linear regression works step by step. We'll use a small dataset with two features to make the calculations manageable, and then verify our results using computational tools.

Suppose you're analyzing house prices and have collected data on three houses with two features: size (in square feet) and age (in years). Your target variable is price (in thousands of dollars).

Here's your data:

First, let's organize this data into matrix form. The feature matrix $\mathbf{X}$ includes a column of ones for the intercept:

$$\mathbf{X} = \begin{bmatrix} 1 & 2000 & 1 \\ 1 & 0 & 3 \\ 1 & 4000 & 5 \\ \end{bmatrix} \qquad \mathbf{y} = \begin{bmatrix} 6 \\ 10 \\ 16 \\ \end{bmatrix}$$

Notice that the first column of $\mathbf{X}$ is all ones.that column allows us to estimate the intercept term $\beta_0$.

Now let's compute the OLS solution: $\hat{\boldsymbol{\beta}} = (\mathbf{X}^\top \mathbf{X})^{-1} \mathbf{X}^\top \mathbf{y}$

Step 1: Compute $\mathbf{X}^\top \mathbf{X}$Link Copied

First, we transpose $\mathbf{X}$:

$$\mathbf{X}^\top = \begin{bmatrix} 1 & 1 & 1 \\ 2000 & 0 & 4000 \\ 1 & 3 & 5 \\ \end{bmatrix}$$

Now multiply $\mathbf{X}^\top$ by $\mathbf{X}$:

$$\mathbf{X}^\top \mathbf{X} = \begin{bmatrix} 1 & 1 & 1 \\ 2000 & 0 & 4000 \\ 1 & 3 & 5 \\ \end{bmatrix} \begin{bmatrix} 1 & 2000 & 1 \\ 1 & 0 & 3 \\ 1 & 4000 & 5 \\ \end{bmatrix}$$

Let's compute each element:

• $(1,1)$: $1 \times 1 + 1 \times 1 + 1 \times 1 = 3$
• $(1,2)$: $1 \times 2000 + 1 \times 0 + 1 \times 4000 = 6000$
• $(1,3)$: $1 \times 1 + 1 \times 3 + 1 \times 5 = 9$
• $(2,1)$: $2000 \times 1 + 0 \times 1 + 4000 \times 1 = 6000$
• $(2,2)$: $2000 \times 2000 + 0 \times 0 + 4000 \times 4000 = 20{,}000{,}000$
• $(2,3)$: $2000 \times 1 + 0 \times 3 + 4000 \times 5 = 22{,}000$
• $(3,1)$: $1 \times 1 + 3 \times 1 + 5 \times 1 = 9$
• $(3,2)$: $1 \times 2000 + 3 \times 0 + 5 \times 4000 = 22{,}000$
• $(3,3)$: $1 \times 1 + 3 \times 3 + 5 \times 5 = 35$

So:

$$\mathbf{X}^\top \mathbf{X} = \begin{bmatrix} 3 & 6000 & 9 \\ 6000 & 20{,}000{,}000 & 22{,}000 \\ 9 & 22{,}000 & 35 \\ \end{bmatrix}$$

Step 2: Compute $\mathbf{X}^\top \mathbf{y}$Link Copied

$$\mathbf{X}^\top \mathbf{y} = \begin{bmatrix} 1 & 1 & 1 \\ 2000 & 0 & 4000 \\ 1 & 3 & 5 \\ \end{bmatrix} \begin{bmatrix} 6 \\ 10 \\ 16 \\ \end{bmatrix}$$

Computing each element:

• First row: $1 \times 6 + 1 \times 10 + 1 \times 16 = 32$
• Second row: $2000 \times 6 + 0 \times 10 + 4000 \times 16 = 12{,}000 + 0 + 64{,}000 = 76{,}000$
• Third row: $1 \times 6 + 3 \times 10 + 5 \times 16 = 6 + 30 + 80 = 116$

So:

$$\mathbf{X}^\top \mathbf{y} = \begin{bmatrix} 32 \\ 76{,}000 \\ 116 \\ \end{bmatrix}$$

Step 3: Compute $(\mathbf{X}^\top \mathbf{X})^{-1}$Link Copied

This is the most complex step. For a 3×3 matrix, we need to compute the determinant and the adjugate matrix. Given the large numbers involved, let's use a computational approach for accuracy.

Step 4: Final CalculationLink Copied

Using computational tools (which we'll verify in the implementation section), the final result is:

$$\hat{\boldsymbol{\beta}} = \begin{bmatrix} 3.0 \\ 0.000333 \\ 2.333 \\ \end{bmatrix}$$

The fitted model is:

$$\hat{y} = 3.0 + 0.000333 \times \text{Size} + 2.333 \times \text{Age}$$

where:

-$\hat{y}$: predicted house price (in thousands of dollars)

• $\text{Size}$: house size in square feet
• $\text{Age}$: house age in years

This means:

• Intercept ($\beta_0 = 3.0$): A house with zero size and zero age would cost $3{,}000 (though this interpretation doesn't make practical sense)
• Size coefficient ($\beta_1 = 0.000333$): For each additional square foot, the price increases by $0.33 (0.000333 × 1000)
• Age coefficient ($\beta_2 = 2.333$): For each additional year of age, the price increases by $2{,}333

Let's check how well this model fits our data by calculating predictions for each house:

House 1 (Size = 2000, Age = 1):

$$\hat{y}_1 = 3.0 + 0.000333(2000) + 2.333(1) = 3.0 + 0.666 + 2.333 = 6.0$$

Actual: $y_1 = 6$ → Perfect prediction!

House 2 (Size = 0, Age = 3):

$$\hat{y}_2 = 3.0 + 0.000333(0) + 2.333(3) = 3.0 + 0 + 7.0 = 10.0$$

Actual: $y_2 = 10$ → Perfect prediction!

House 3 (Size = 4000, Age = 5):

$$\hat{y}_3 = 3.0 + 0.000333(4000) + 2.333(5) = 3.0 + 1.332 + 11.665 = 16.0$$

Actual: $y_3 = 16$ → Perfect prediction!

The model provides perfect predictions for this dataset. This is expected because we have exactly as many parameters (3) as data points (3), which allows the model to fit the data exactly. In practice with larger datasets, you would typically see some prediction errors due to noise and the inherent complexity of real-world relationships.