Polynomial Regression › Example

Let's work through a concrete example to understand how polynomial regression works step-by-step. You'll use a simple dataset and see the complete calculation process.

Consider the following dataset with 5 observations:

We'll fit a quadratic polynomial regression model: $y = \beta_0 + \beta_1 x + \beta_2 x^2$

For a quadratic model, our design matrix $\mathbf{X}$ includes columns for the intercept, $x$, and $x^2$:

$$\mathbf{X} = \begin{bmatrix} 1 & 1 & 1^2 \\ 1 & 2 & 2^2 \\ 1 & 3 & 3^2 \\ 1 & 4 & 4^2 \\ 1 & 5 & 5^2 \end{bmatrix} = \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \\ 1 & 4 & 16 \\ 1 & 5 & 25 \end{bmatrix}$$

The response vector is:

$$\mathbf{y} = \begin{bmatrix} 2.1 \\ 3.9 \\ 8.1 \\ 15.9 \\ 25.1 \end{bmatrix}$$

Now you calculate $\mathbf{X}^T\mathbf{X}$ (the Gram matrix). First, note that $\mathbf{X}^T$ is the transpose of $\mathbf{X}$:

$$\mathbf{X}^T\mathbf{X} = \begin{bmatrix} 1 & 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 & 5 \\ 1 & 4 & 9 & 16 & 25 \end{bmatrix}_{3 \times 5} \begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 4 \\ 1 & 3 & 9 \\ 1 & 4 & 16 \\ 1 & 5 & 25 \end{bmatrix}_{5 \times 3} = \begin{bmatrix} 5 & 15 & 55 \\ 15 & 55 & 225 \\ 55 & 225 & 979 \end{bmatrix}_{3 \times 3}$$

For example, the element in row 1, column 2 is: $(1)(1) + (1)(2) + (1)(3) + (1)(4) + (1)(5) = 15$.

Similarly, calculate $\mathbf{X}^T\mathbf{y}$:

$$\mathbf{X}^T\mathbf{y} = \begin{bmatrix} 1 & 1 & 1 & 1 & 1 \\ 1 & 2 & 3 & 4 & 5 \\ 1 & 4 & 9 & 16 & 25 \end{bmatrix}_{3 \times 5} \begin{bmatrix} 2.1 \\ 3.9 \\ 8.1 \\ 15.9 \\ 25.1 \end{bmatrix}_{5 \times 1} = \begin{bmatrix} 55.1 \\ 223.3 \\ 972.5 \end{bmatrix}_{3 \times 1}$$

For example, the first element is: $(1)(2.1) + (1)(3.9) + (1)(8.1) + (1)(15.9) + (1)(25.1) = 55.1$.

Using the normal equation $\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$, we first compute the inverse of $\mathbf{X}^T\mathbf{X}$:

$$(\mathbf{X}^T\mathbf{X})^{-1} = \begin{bmatrix} 5 & 15 & 55 \\ 15 & 55 & 225 \\ 55 & 225 & 979 \end{bmatrix}^{-1} = \begin{bmatrix} 4.6 & -3.3 & 0.5 \\ -3.3 & 2.671 & -0.429 \\ 0.5 & -0.429 & 0.071 \end{bmatrix}$$

Therefore, the coefficient estimates are:

$$\hat{\boldsymbol{\beta}} = \begin{bmatrix} 4.6 & -3.3 & 0.5 \\ -3.3 & 2.671 & -0.429 \\ 0.5 & -0.429 & 0.071 \end{bmatrix} \begin{bmatrix} 55.1 \\ 223.3 \\ 972.5 \end{bmatrix} = \begin{bmatrix} 2.82 \\ -2.086 \\ 1.314 \end{bmatrix}$$

where:

• $\hat{\beta}_0 = 2.82$ is the estimated intercept
• $\hat{\beta}_1 = -2.086$ is the estimated coefficient for the linear term $x$
• $\hat{\beta}_2 = 1.314$ is the estimated coefficient for the quadratic term $x^2$

Our fitted quadratic polynomial regression model is:

$$\hat{y} = 2.82 - 2.086x + 1.314x^2$$

where:

• $\hat{y}$ is the predicted response value
• $x$ is the predictor variable
• The coefficients indicate a quadratic relationship with both linear and quadratic components

This model shows that the relationship has a negative linear component ($-2.086x$) and a positive quadratic component ($1.314x^2$), creating a U-shaped curve that increases more rapidly for larger values of $x$.

Let's verify our model by calculating predictions for each observation:

• For $x = 1$: $\hat{y} = 2.82 - 2.086(1) + 1.314(1)^2 = 2.82 - 2.086 + 1.314 = 2.05$ (actual: 2.1, residual: $-0.05$)
• For $x = 2$: $\hat{y} = 2.82 - 2.086(2) + 1.314(4) = 2.82 - 4.172 + 5.256 = 3.90$ (actual: 3.9, residual: $0.00$)
• For $x = 3$: $\hat{y} = 2.82 - 2.086(3) + 1.314(9) = 2.82 - 6.258 + 11.826 = 8.39$ (actual: 8.1, residual: $0.29$)
• For $x = 4$: $\hat{y} = 2.82 - 2.086(4) + 1.314(16) = 2.82 - 8.344 + 21.024 = 15.50$ (actual: 15.9, residual: $-0.40$)
• For $x = 5$: $\hat{y} = 2.82 - 2.086(5) + 1.314(25) = 2.82 - 10.430 + 32.850 = 25.24$ (actual: 25.1, residual: $0.14$)

where the residual is defined as $e_i = \hat{y}_i - y_i$ (predicted minus actual).

The model captures the quadratic trend very well, with small residuals. The residual sum of squares is:

$$\text{RSS} = (-0.05)^2 + (0.00)^2 + (0.29)^2 + (-0.40)^2 + (0.14)^2 = 0.265$$

where:

• $\text{RSS}$ is the residual sum of squares
• Each term represents the squared difference between predicted and actual values
• A smaller RSS indicates a better fit to the data

This small RSS value of 0.265 indicates an excellent fit to the data.