Simple Linear Regression › Formula

The mathematical foundation of simple linear regression is expressed through a linear equation that describes the relationship between your variables. Let's break this down step by step to understand what each component means and how they work together.

The core formula for simple linear regression is:

$$y = \beta_0 + \beta_1 x + \epsilon$$

where:

• $y$: dependent variable (target/response variable) . what you're trying to predict or explain
• $x$: independent variable (predictor/feature/explanatory variable) . what you're using to make predictions
• $\beta_0$: intercept term (beta-zero) . the value of $y$ when $x = 0$; where the regression line crosses the y-axis
• $\beta_1$: slope coefficient (beta-one) . the change in $y$ for each one-unit increase in $x$ (positive = positive relationship, negative = negative relationship)
• $\epsilon$: error term (epsilon) . the random error component representing the difference between the actual observed value and the model's prediction; captures unmodeled factors, measurement errors, and random variation that influence $y$

In summary, $y$ is the outcome you want to predict, $x$ is the input feature, $\beta_0$ is the baseline value of $y$ when $x$ is zero, $\beta_1$ tells you how $y$ changes as $x$ increases, and $\epsilon$ accounts for everything not explained by the linear relationship.

To better understand what the intercept and slope represent, let's visualize them on a regression line. This diagram will help you see where the intercept appears on the graph and how the slope determines the steepness and direction of the line. Pay attention to how the green triangle shows the relationship between changes in x and changes in y.

To find the best-fitting line, we use the method of least squares, which minimizes the sum of squared differences between observed and predicted values. The least squares objective function is:

$$SSE = \sum_{i=1}^n (y_i - \hat{y}_i)^2 = \sum_{i=1}^n (y_i - \beta_0 - \beta_1 x_i)^2$$

where:

• $SSE$: Sum of Squared Errors (also called Sum of Squared Residuals)
• $y_i$: observed values
• $\hat{y}_i$: predicted values ($\hat{y}_i = \beta_0 + \beta_1 x_i$)
• $\beta_0, \beta_1$: regression coefficients to be estimated
• $n$: number of observations

The goal is to find the values of $\beta_0$ and $\beta_1$ that minimize this sum. The formulas for calculating the optimal coefficients are.

We have covered the basics of the least squares solution in the Sum of Squared Errors (SSE) section. Refer to it for more a more detailed explanation.

Formula for calculating the slope ($\beta_1$)Link Copied

Let's break down the slope formula to understand why it works:

$$\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

where:

• $\beta_1$: slope coefficient (change in y per unit change in x, also called the regression coefficient)
• $x_i$: individual x values
• $\bar{x}$: sample mean of x values
• $y_i$: individual y values
• $\bar{y}$: sample mean of y values
• $n$: number of observations

1. $(x_i - \bar{x})$ measures how far each $x$ value is from the mean of all $x$ values
2. $(y_i - \bar{y})$ measures how far each $y$ value is from the mean of all $y$ values
3. $(x_i - \bar{x})(y_i - \bar{y})$ captures the relationship between these deviations
4. The numerator sums these products, giving us the total covariance between $x$ and $y$
5. The denominator sums the squared deviations of $x$ from its mean, giving us the variance of $x$
6. Dividing covariance by variance gives us the slope that describes the linear relationship

The numerator in the formula for $\beta_1$:

$$\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})$$

calculates the sample covariance between $x$ and $y$ (up to a scaling factor of $n-1$). Here, the "scaling factor" refers to the denominator used when computing the sample covariance: the sum $\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})$ is typically divided by $n-1$ to obtain the average product of deviations, which corrects for bias in estimating the population covariance from a sample. In the regression slope formula, we use just the numerator (the sum), not the average, so the scaling factor $n-1$ is omitted.

Covariance measures how two variables change together: if both $x$ and $y$ tend to be above or below their means at the same time, the covariance is positive; if one tends to be above its mean when the other is below, the covariance is negative. In the context of regression, this term captures the direction and strength of the linear relationship between $x$ and $y$. By dividing this covariance by the variance of $x$ (the denominator), we obtain the slope $\beta_1$, which quantifies how much $y$ changes for a unit change in $x$.

Formula for calculating the intercept ($\beta_0$)Link Copied

$$\beta_0 = \bar{y} - \beta_1 \bar{x}$$

where:

• $\beta_0$: intercept (value of y when x = 0)
• $\bar{y}$: sample mean of y values
• $\beta_1$: slope coefficient
• $\bar{x}$: sample mean of x values

Here, $\beta_0$ is the intercept of the regression line.the value of $y$ when $x = 0$. The formula shows that to find the intercept, you first calculate the mean of the $y$ values ($\bar{y}$) and the mean of the $x$ values ($\bar{x}$), then subtract the product of the slope ($\beta_1$) and the mean of $x$ from the mean of $y$.

• $\bar{x} = \dfrac{1}{n} \sum_{i=1}^n x_i$ is the sample mean of the $x$ values.
• $\bar{y} = \dfrac{1}{n} \sum_{i=1}^n y_i$ is the sample mean of the $y$ values.

This formula ensures that the regression line passes through the point $(\bar{x}, \bar{y})$, meaning the average predicted value equals the average observed value. The intercept $\beta_0$ adjusts the line vertically so that it fits the data according to the least squares criterion.

Here is the direct substitution version of the simple linear regression formula, where you can directly plug in the values of $x_i$ and $y_i$ to compute the predicted value $\hat{y}_i$ for any $x_i$:

Given $n$ data points $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$, the prediction for $y$ at a given $x_i$ is:

$$\hat{y}_i = \beta_0 + \beta_1 x_i$$

where

$$\beta_1 = \frac{\sum_{j=1}^n (x_j - \bar{x})(y_j - \bar{y})}{\sum_{j=1}^n (x_j - \bar{x})^2}$$

and

$$\beta_0 = \bar{y} - \beta_1 \bar{x}$$

with

$$\bar{x} = \frac{1}{n} \sum_{j=1}^n x_j \qquad \bar{y} = \frac{1}{n} \sum_{j=1}^n y_j$$

With direct substitution, the formula for the predicted value $\hat{y}_i$ is:

$$\hat{y}_i = \left[ \frac{\sum_{j=1}^n y_j}{n} - \frac{ \left( \sum_{j=1}^n x_j y_j - \frac{1}{n} \left(\sum_{j=1}^n x_j\right)\left(\sum_{j=1}^n y_j\right) \right) }{ \sum_{j=1}^n x_j^2 - \frac{1}{n} \left(\sum_{j=1}^n x_j\right)^2 } \cdot \frac{\sum_{j=1}^n x_j}{n} \right] + \frac{ \left( \sum_{j=1}^n x_j y_j - \frac{1}{n} \left(\sum_{j=1}^n x_j\right)\left(\sum_{j=1}^n y_j\right) \right) }{ \sum_{j=1}^n x_j^2 - \frac{1}{n} \left(\sum_{j=1}^n x_j\right)^2 } x_i$$

This formula allows you to compute the predicted value $\hat{y}_i$ for any $x_i$ directly from the raw data, using only sums over the $x_j$ and $y_j$ values.

Note: In practice, we would never write the prediction formula in this expanded form. Instead, we would first compute the means $\bar{x}$ and $\bar{y}$, then calculate the slope $\beta_1$ and intercept $\beta_0$, and finally use the compact formula $\hat{y}_i = \beta_0 + \beta_1 x_i$. The expanded version above is shown purely for explanatory purposes, to illustrate how the regression prediction can be written directly in terms of sums over the data. This helps clarify the underlying mechanics, but is not how regression is implemented in real code or statistical software.

The simple linear regression model can also be expressed in a centered form that highlights the relationship between deviations from the mean:

$$\hat{y}_i = \bar{y} + \beta_1 (x_i - \bar{x})$$

This form shows that each prediction $\hat{y}_i$ is the mean of $y$ plus the slope times the deviation of $x_i$ from the mean of $x$. This formulation is mathematically equivalent to the standard form $\hat{y}_i = \beta_0 + \beta_1 x_i$ because:

$$\begin{align*} \hat{y}_i &= \beta_0 + \beta_1 x_i \\ &= (\bar{y} - \beta_1 \bar{x}) + \beta_1 x_i \\ &= \bar{y} - \beta_1 \bar{x} + \beta_1 x_i \\ &= \bar{y} + \beta_1 (x_i - \bar{x}) \end{align*}$$

This centered form is useful for understanding how the regression line relates to the data's center of mass and how predictions depend on deviations from the mean values.