Ordinary Least Squares (OLS) › Example

Let's work through a step-by-step example to see exactly how the Ordinary Least Squares (OLS) solution is computed in practice. We'll use a tiny dataset so that every calculation can be shown explicitly, and you'll see how the matrix algebra connects to the underlying numbers.

Suppose we have the following dataset with four points:

We want to fit a straight line of the form $y = \beta_0 + \beta_1 x$ to this data using OLS.

The design matrix $X$ includes a column of ones (for the intercept) and a column for the $x$ values:

$$X = \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ 1 & 4 \\ \end{bmatrix}$$

The target vector $y$ is:

$$y = \begin{bmatrix} 3 \\ 5 \\ 7 \\ 9 \\ \end{bmatrix}$$

First, we take the transpose of $X$:

$$X^T = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\ \end{bmatrix}$$

Now, multiply $X^T$ by $X$:

$$X^T X = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\ \end{bmatrix} \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \\ 1 & 4 \\ \end{bmatrix} = \begin{bmatrix} 1+1+1+1 & 1+2+3+4 \\ 1+2+3+4 & 1^2+2^2+3^2+4^2 \\ \end{bmatrix} = \begin{bmatrix} 4 & 10 \\ 10 & 30 \\ \end{bmatrix}$$

• The $(1,1)$ entry is the sum of the first column of $X$: $1+1+1+1=4$
• The $(1,2)$ and $(2,1)$ entries are the sum of the $x$ values: $1+2+3+4=10$
• The $(2,2)$ entry is the sum of the squares of $x$: $1^2+2^2+3^2+4^2=1+4+9+16=30$

For a $2 \times 2$ matrix $\begin{bmatrix} a & b \\ c & d \end{bmatrix}$, the inverse is:

$$\frac{1}{ad-bc} \begin{bmatrix} d & -b \\ -c & a \\ \end{bmatrix}$$

Plugging in our values:

• $a=4$, $b=10$, $c=10$, $d=30$
• $ad-bc = 4 \times 30 - 10 \times 10 = 120 - 100 = 20$

So,

$$(X^T X)^{-1} = \frac{1}{20} \begin{bmatrix} 30 & -10 \\ -10 & 4 \\ \end{bmatrix} = \begin{bmatrix} 1.5 & -0.5 \\ -0.5 & 0.2 \\ \end{bmatrix}$$

Multiply $X^T$ by $y$:

$$X^T y = \begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 \\ \end{bmatrix} \begin{bmatrix} 3 \\ 5 \\ 7 \\ 9 \\ \end{bmatrix} = \begin{bmatrix} 3+5+7+9 \\ 1 \times 3 + 2 \times 5 + 3 \times 7 + 4 \times 9 \\ \end{bmatrix} = \begin{bmatrix} 24 \\ 3+10+21+36 \\ \end{bmatrix} = \begin{bmatrix} 24 \\ 70 \\ \end{bmatrix}$$

Now, apply the OLS formula:

$$\hat{\beta} = (X^T X)^{-1} X^T y$$

Multiply the matrices:

$$\hat{\beta} = \begin{bmatrix} 1.5 & -0.5 \\ -0.5 & 0.2 \\ \end{bmatrix} \begin{bmatrix} 24 \\ 70 \\ \end{bmatrix} = \begin{bmatrix} 1.5 \times 24 + (-0.5) \times 70 \\ -0.5 \times 24 + 0.2 \times 70 \\ \end{bmatrix} = \begin{bmatrix} 36 - 35 \\ -12 + 14 \\ \end{bmatrix} = \begin{bmatrix} 1 \\ 2 \\ \end{bmatrix}$$

So:

• $\hat{\beta}_0 = 1$ (intercept)
• $\hat{\beta}_1 = 2$ (slope)

The OLS solution gives us the best-fitting line:

$$y = 1 + 2x$$

• The intercept ($\hat{\beta}_0$) is $1$
• The slope ($\hat{\beta}_1$) is $2$

This means for every unit increase in $x$, $y$ increases by $2$ units, and when $x=0$, $y$ is predicted to be $1$.

Let's check that this line fits all the data points exactly:

• For $x=1$: $\hat{y}_1 = 1 + 2 \times 1 = 3$
• For $x=2$: $\hat{y}_2 = 1 + 2 \times 2 = 5$
• For $x=3$: $\hat{y}_3 = 1 + 2 \times 3 = 7$
• For $x=4$: $\hat{y}_4 = 1 + 2 \times 4 = 9$

All predicted values match the observed $y$ values perfectly.

1. Build the design matrix $X$ and target vector $y$
2. Compute $X^T X$ and $X^T y$
3. Find the inverse $(X^T X)^{-1}$
4. Multiply to get $\hat{\beta} = (X^T X)^{-1} X^T y$
5. Interpret the coefficients in the context of the problem
6. (Optional) Verify predictions and fit

This step-by-step process is the foundation for all OLS regression, whether you have two points or two million. For larger datasets, the calculations are the same, but are handled by computers using efficient matrix operations. In particular, the calculation of the matrix inverse $(X^T X)^{-1}$, which is required to solve for the coefficients, can be computationally intensive for very large or high-dimensional datasets. Modern numerical libraries use optimized algorithms (such as LU or Cholesky decomposition) to compute this inverse efficiently and accurately, or may use more stable techniques like solving the equivalent system of linear equations directly without explicitly forming the inverse.