Polynomial Regression › Formula

The polynomial regression model extends linear regression by including polynomial terms of the independent variable. For a single variable $x$, the polynomial regression model of degree $d$ is:

$$y = \beta_0 + \beta_1 x + \beta_2 x^2 + \beta_3 x^3 + \cdots + \beta_d x^d + \epsilon$$

where:

• $y$ is the dependent variable (response variable we want to predict)
• $x$ is the independent variable (predictor or feature)
• $\beta_0$ is the intercept (the value of $y$ when $x = 0$)
• $\beta_1, \beta_2, \ldots, \beta_d$ are the regression coefficients for the polynomial terms $x, x^2, \ldots, x^d$
• $\epsilon$ is the error term (random noise, assumed to be normally distributed with mean 0 and constant variance $\sigma^2$)
• $d$ is the degree of the polynomial (the highest power of $x$ in the model)

The polynomial regression model can be written in matrix form as:

$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}$$

where:

• $\mathbf{y} = \begin{bmatrix} y_1 \\ y_2 \\ \vdots \\ y_n \end{bmatrix}$ is the $n \times 1$ vector of response values (observed outcomes)

• $\mathbf{X} = \begin{bmatrix} 1 & x_1 & x_1^2 & \cdots & x_1^d \\ 1 & x_2 & x_2^2 & \cdots & x_2^d \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^d \end{bmatrix}$ is the $n \times (d+1)$ design matrix containing polynomial features up to degree $d$

• $\boldsymbol{\beta} = \begin{bmatrix} \beta_0 \\ \beta_1 \\ \beta_2 \\ \vdots \\ \beta_d \end{bmatrix}$ is the $(d+1) \times 1$ vector of coefficients to be estimated

• $\boldsymbol{\epsilon} = \begin{bmatrix} \epsilon_1 \\ \epsilon_2 \\ \vdots \\ \epsilon_n \end{bmatrix}$ is the $n \times 1$ vector of error terms (assumed to be i.i.d. with mean 0 and constant variance $\sigma^2$)

When you have more than one independent variable (feature), polynomial regression expands to include powers of each variable along with all possible products (interactions) between variables up to the chosen degree. This allows the model to capture much more complex relationships between your predictors and the response.

Suppose you have $p$ independent variables: $x_1, x_2, \ldots, x_p$. The general form of a polynomial regression model of degree $d$ is:

$$y = \beta_0 + \sum_{i=1}^{p} \beta_i x_i + \sum_{i=1}^{p} \sum_{j=i}^{p} \beta_{ij} x_i x_j + \sum_{i=1}^{p} \sum_{j=i}^{p} \sum_{k=j}^{p} \beta_{ijk} x_i x_j x_k + \cdots + \epsilon$$

where:

• $y$ is the response variable
• $p$ is the number of predictor variables
• $x_i$ is the $i$-th predictor variable
• $\beta_0$ is the intercept term
• $\beta_i$ are coefficients for linear terms
• $\beta_{ij}$ are coefficients for quadratic terms and two-way interactions
• $\beta_{ijk}$ are coefficients for cubic terms and three-way interactions
• $\epsilon$ is the error term

Let's break down what this means:

• Linear terms: Each variable appears by itself, just like in standard linear regression. For example, $\beta_1 x_1$, $\beta_2 x_2$, ..., $\beta_p x_p$.
• Quadratic terms: These include both squared terms (like $x_1^2$, $x_2^2$) and products of two different variables (like $x_1 x_2$). The coefficients for these are $\beta_{ii}$ for squares and $\beta_{ij}$ for interactions where $i \neq j$.
• Cubic and higher-order terms: These are all possible products of three or more variables, such as $x_1 x_2 x_3$, $x_2^2 x_3$, etc., up to the chosen degree $d$.
• Interaction terms: Any term that involves the product of two or more different variables (e.g., $x_1 x_2$) is called an interaction term. These allow the effect of one variable to depend on the value of another.

Example: Two Variables, Degree 2Link Copied

If you have two variables ($x_1$ and $x_2$) and use a quadratic (degree 2) polynomial, your model will look like:

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_{11} x_1^2 + \beta_{22} x_2^2 + \beta_{12} x_1 x_2 + \epsilon$$

where:

• $\beta_0$ is the intercept
• $\beta_1$ and $\beta_2$ are coefficients for the linear terms $x_1$ and $x_2$
• $\beta_{11}$ and $\beta_{22}$ are coefficients for the squared terms $x_1^2$ and $x_2^2$
• $\beta_{12}$ is the coefficient for the interaction term $x_1 x_2$
• $\epsilon$ is the error term

Here, you have a total of 6 terms (including the intercept):

• Two linear terms: $x_1$, $x_2$
• Two quadratic (squared) terms: $x_1^2$, $x_2^2$
• One interaction term: $x_1 x_2$

General StructureLink Copied

In general, for $p$ variables and degree $d$, the number of terms in the model (including the intercept) is given by the binomial coefficient:

$$\text{Number of terms} = \binom{p + d}{d} = \frac{(p + d)!}{d! \cdot p!}$$

where:

• $p$ is the number of predictor variables
• $d$ is the polynomial degree
• $\binom{p + d}{d}$ is the binomial coefficient (also written as "p+d choose d")
• The formula counts all possible combinations of variables multiplied together up to degree $d$, including the intercept

For example, with $p = 2$ variables and $d = 2$ degree: $\binom{2+2}{2} = \binom{4}{2} = 6$ terms (which matches our example above: intercept, $x_1$, $x_2$, $x_1^2$, $x_2^2$, $x_1 x_2$).

This rapid combinatorial growth is why polynomial regression with multiple variables can quickly become computationally expensive and prone to overfitting.

The polynomial regression model maintains several important mathematical properties from linear regression. You can estimate the coefficients using the least squares method, which minimizes the sum of squared residuals:

$$\text{RSS} = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \sum_{i=1}^{n} \left(y_i - \sum_{j=0}^{d} \beta_j x_i^j\right)^2$$

where:

• $\text{RSS}$ is the residual sum of squares (the objective function to minimize)
• $y_i$ is the observed response value for observation $i$
• $\hat{y}_i$ is the predicted response value for observation $i$
• $n$ is the number of observations
• $d$ is the polynomial degree
• $\beta_j$ is the coefficient for the $j$-th power term
• $x_i$ is the predictor value for observation $i$

The solution to this minimization problem is given by the normal equation:

$$\hat{\boldsymbol{\beta}} = (\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$$

where:

• $\hat{\boldsymbol{\beta}}$ is the $(d+1) \times 1$ vector of estimated coefficients
• $\mathbf{X}^T$ is the $(d+1) \times n$ transpose of the design matrix
• $(\mathbf{X}^T\mathbf{X})^{-1}$ is the $(d+1) \times (d+1)$ inverse of the Gram matrix
• $\mathbf{y}$ is the $n \times 1$ vector of observed response values

The model maintains the Gauss-Markov theorem properties when the error terms satisfy the standard assumptions (zero mean, constant variance, and uncorrelated errors), making the least squares estimator the best linear unbiased estimator (BLUE).