Simple Linear Regression › Understanding Covariance and Correlation

This visualization makes the abstract mathematical concepts of covariance and correlation concrete. The first plot shows how data points in different quadrants contribute to the covariance calculation, while the second plot demonstrates how correlation strength directly determines the regression slope. Understanding these relationships is important for interpreting why the slope formula works and how it captures the linear relationship between variables.

The complete formula derivation for the simple linear regression model, showing the prediction for each observation, is:

$$\begin{align*} \hat{y}_i &= \beta_0 + \beta_1 x_i \\ &= (\bar{y} - \beta_1 \bar{x}) + \beta_1 x_i \\ &= \bar{y} + \beta_1 (x_i - \bar{x}) \end{align*}$$

Where the slope $\beta_1$ is given by:

$$\beta_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2}$$

And the intercept $\beta_0$ is:

$$\beta_0 = \bar{y} - \beta_1 \bar{x}$$

So, the full prediction equation for every data point $i$ is:

$$\hat{y}_i = \bar{y} + \frac{\sum_{j=1}^n (x_j - \bar{x})(y_j - \bar{y})}{\sum_{j=1}^n (x_j - \bar{x})^2} (x_i - \bar{x})$$

Here, the intercept $\beta_0$ does not appear explicitly because it has been incorporated into the formula using the means of $x$ and $y$. Recall that $\beta_0 = \bar{y} - \beta_1 \bar{x}$, so when you expand the prediction equation $\hat{y}_i = \beta_0 + \beta_1 x_i$, you get:

$$\hat{y}_i = (\bar{y} - \beta_1 \bar{x}) + \beta_1 x_i = \bar{y} + \beta_1 (x_i - \bar{x})$$

This form centers the prediction around the mean values, making the intercept implicit in the calculation. Thus, $\beta_0$ is not shown separately because its effect is already included through $\bar{y}$ and the adjustment by $\bar{x}$.

This formula shows that each predicted value $\hat{y}_i$ is the mean of $y$ plus the slope (which measures the relationship between $x$ and $y$) times the deviation of $x_i$ from the mean of $x$.

The least squares solution has several important mathematical properties. First, it's unbiased, meaning that on average, the estimated coefficients will equal the true population values (assuming the model assumptions are met). Second, it's efficient among all unbiased linear estimators, meaning it has the smallest possible variance under the Gauss-Markov conditions (that is, when the errors are uncorrelated, have equal variance, and have zero mean given the predictors.these are known as the "Gauss-Markov assumptions"). Third, the sum of residuals (differences between observed and predicted values) equals zero, and the sum of squared residuals is minimized.

The method also produces a unique solution (unless all $x$ values are identical), typically ensuring that there's only one optimal line for any given dataset. This mathematical elegance makes simple linear regression both theoretically sound and practically useful.

Let's visualize how it finds the line that minimizes the sum of squared residuals. The key insight is that we're looking for the line where the sum of the squared vertical distances from each point to the line is as small as possible. In this visualization, you'll see three different lines fitted to the same data: the optimal line (red) that minimizes the sum of squared residuals, and two suboptimal lines (blue and green) with higher residual sums. The thick red vertical lines clearly show the residuals (distances from points to the optimal line), while the blue and green dotted lines show residuals for the suboptimal models. The gray squares represent the squared residuals for the optimal line - their areas are proportional to the squared error magnitude, making it easy to see why the least squares method chooses the line with the smallest total squared error.