Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Variable Relationships › Simple Linear Regression: Modeling Relationships

Michael BrenndoerferJune 3, 20255 min read

Machine Learning from ScratchAll chapters

Part 1 of 9Statistics 101

Chapter 6Variable Relationships › Simple Linear Regression: Modeling Relationships

Reading section 5 of 11 · Simple Linear Regression: Modeling Relationships · 5 minSection 5 of 11 · 5 min

Section 5 of 11 · 5 min read

Variable Relationships › Simple Linear Regression: Modeling RelationshipsOne section · Sections in this chapter5 / 11 ↓

Sections in this chapter11

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

While correlation measures the strength of a linear relationship, regression goes further by modeling the relationship explicitly. Simple linear regression fits a line through the data to predict a dependent variable $Y$ from an independent variable $X$ . This line represents the best linear approximation of how $Y$ changes as $X$ changes.

The simple linear regression model is:

Y = \beta_0 + \beta_1 X + \epsilon

Where:

$Y$ : Dependent variable (response)
$X$ : Independent variable (predictor)
$\beta_0$ : Intercept (value of $Y$ when $X = 0$ )
$\beta_1$ : Slope (change in $Y$ for unit change in $X$ )
$\epsilon$ : Error term (unexplained variation)

The regression coefficients are typically estimated using ordinary least squares (OLS), which minimizes the sum of squared residuals. The OLS estimates are:

\begin{aligned} \hat{\beta}_1 &= \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = r_{XY} \frac{s_Y}{s_X} \\ \hat{\beta}_0 &= \bar{y} - \hat{\beta}_1 \bar{x} \end{aligned}

Notice that the slope coefficient $\hat{\beta}_1$ is directly related to the correlation coefficient $r_{XY}$ , scaled by the ratio of standard deviations. This connection reveals that regression and correlation are closely related: correlation measures association strength, while regression quantifies the specific form of that association.

The regression line gives several useful capabilities. First, it allows prediction: given a new value of $X$ , we can predict the expected value of $Y$ using $\hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X$ . Second, the slope coefficient $\hat{\beta}_1$ quantifies the average change in $Y$ associated with a one-unit change in $X$ . Third, the regression framework allows for statistical inference: we can test whether the relationship is statistically significant and construct confidence intervals for the coefficients.

The following example demonstrates a simple linear regression fit to data with a clear linear relationship. The regression line captures the average tendency of Y to increase with X, while the residuals represent the unexplained variation around this trend.

Out[5]:

Visualization

Scatter plot with fitted linear regression line. — Simple linear regression fit. The blue points represent observed data, while the red line shows the fitted regression line that minimizes the sum of squared residuals. The equation shown in the plot title indicates the fitted relationship, where the slope coefficient shows that for each unit increase in X, Y increases by approximately 2 units on average. The close fit demonstrates how regression captures the linear relationship between variables.

Understanding what regression optimizes helps build intuition for how the method works. The "least squares" criterion means we're finding the line that minimizes the sum of squared vertical distances from points to the line. These vertical distances are called residuals, and they represent the prediction errors. The following visualization shows these residuals explicitly and illustrates what the regression line accomplishes.

Out[6]:

Visualization

Scatter plot with regression line and residual lines showing vertical distances. — Visualization of regression residuals. The blue points represent observed data, the red line shows the fitted regression line, and the gray vertical lines illustrate the residuals.the vertical distances from each point to the line. Ordinary least squares (OLS) regression finds the line that minimizes the sum of squared lengths of these gray lines. Points above the line indicate observations where the actual Y exceeds the predicted Y (underestimates), while points below indicate predictions that exceed actual values (overestimates). The residuals capture all variation in Y not explained by the linear relationship with X.