Ordinary Least Squares (OLS) › Visualizing OLS

Michael BrenndoerferApril 10, 20254 min read
Machine Learning from ScratchAll chapters
Part 4 of 9Regression Models
Chapter 30Ordinary Least Squares (OLS) › Visualizing OLS
Reading section 3 of 8 · Visualizing OLS · 4 minSection 3 of 8 · 4 min

Book contents

Machine Learning from Scratch9 parts · 66 chaptersView the whole book →
Active chapter 30 of 66

Section 3 of 8 · 4 min read

Ordinary Least Squares (OLS) › Visualizing OLSOne section · Sections in this chapter3 / 8 ↓
Sections in this chapter8
Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Let's take a step back and look at OLS from a geometric perspective.it really helps make sense of why this method is so powerful. Imagine you have a scatterplot of your data points, and you're trying to draw the best-fitting straight line through them. What OLS does is find the line where the total of all the squared vertical distances (the "residuals") from each point to the line is as small as possible. Let's see exactly how this works in practice.

This geometric interpretation reveals why OLS is both mathematically elegant and practically effective. By minimizing squared errors, OLS balances the influence of all data points. You can see how the method balances fitting the data with maintaining statistical stability.

Out[2]:
Visualization
OLS visualization with 20 data points, dashed true relationship, solid fitted line y equals 4.55 plus 1.62 x, vertical residual stems, and an SSE annotation of 43.68.
OLS finds the best-fitting line by minimizing squared vertical distances. The dashed residual stems connect each of the 20 observations to the solid OLS fit, while the dashed true relationship shows the data-generating line. The fitted equation is y = 4.55 + 1.62x and the resulting sum of squared errors is 43.68.

Even though the OLS fit (blue line) is very close to the true relationship (green dashed line), it does not exactly match it. This is because the observed data points contain random noise or error.real-world measurements are rarely perfect. OLS works by finding the line that minimizes the sum of squared vertical distances (residuals) between the observed data and the fitted line, not the true underlying relationship. As a result, the OLS fit represents the best possible linear approximation given the noisy data, but it will generally differ slightly from the true relationship unless there is no noise at all. This distinction highlights a fundamental aspect of statistical modeling: our estimates are always influenced by the variability and imperfections present in real data.