Ordinary Least Squares (OLS) › Ordinary Least Squares (OLS)

Ordinary Least Squares (OLS) is the basic method for estimating parameters in linear regression models. Unlike more complex machine learning algorithms that require iterative optimization, OLS gives a direct, closed-form solution that we can compute exactly using matrix algebra. This makes OLS both mathematically elegant and computationally efficient.

OLS works by finding the line (or hyperplane in multiple dimensions) that minimizes the sum of squared differences between observed values and predicted values. The "least squares" name comes from this minimization of squared errors, which has both mathematical and practical advantages. When we have a dataset with features and a target variable, OLS gives us the best-fitting linear relationship according to this squared error criterion.

In simple terms, imagine we have a scatter plot of data points and want to draw a straight line through them. OLS finds the line that makes the total squared distance from all points to the line as small as possible. This line shows us the general trend in our data and allows us to make predictions for new observations.

OLS vs. Linear RegressionLink Copied

It's important to clarify that Ordinary Least Squares (OLS) is not a type of regression model itself, but rather a method (or formula) for estimating the parameters of a linear regression model. In other words, OLS is the mathematical procedure we use to find the best-fitting line (or hyperplane) for a set of data in linear regression.

• Linear regression refers to the statistical model that describes the relationship between the dependent variable and one or more independent variables.
• OLS is the technique most commonly used to estimate the coefficients (parameters) of that model by minimizing the sum of squared errors.

So, when we talk about "OLS regression," we're really referring to linear regression models whose parameters have been estimated using the OLS method. The OLS formula provides a systematic way to find the line (or hyperplane) that best fits the data according to the least squares criterion.

Note: While OLS is the most common method for linear regression, other estimation methods exist (such as weighted least squares, generalized least squares, or maximum likelihood estimation), but OLS remains the standard approach for basic linear regression problems.

OLS offers several advantages that make it a preferred method for linear regression. First, it provides a closed-form solution, meaning we can compute the optimal coefficients directly using a mathematical formula rather than through iterative optimization. This makes OLS computationally efficient and typically finds the global minimum of the sum of squared errors, unlike iterative methods that might get stuck in local minima.

The method is also highly interpretable, as each coefficient represents the expected change in the target variable for a one-unit increase in the corresponding feature, holding all other features constant. This linear relationship makes it easy to understand and explain the model's predictions to stakeholders. Additionally, OLS has well-established statistical properties, including being the Best Linear Unbiased Estimator (BLUE) under the Gauss-Markov assumptions, which provides theoretical guarantees about the quality of the estimates.

Finally, OLS is well-understood, with extensive theoretical foundation and practical experience accumulated over decades. This method is applicable across a wide range of applications, and it provides a useful baseline against which to compare more complex models.

Despite its strengths, OLS has several important limitations that we should be aware of. The method assumes a linear relationship between features and the target variable, which may not hold in many real-world scenarios. If the true relationship is nonlinear, OLS will provide biased and potentially misleading results.

OLS is also sensitive to outliers, as the squared error penalty means that extreme values have disproportionate influence on the final model. A single outlier can significantly shift the regression line, potentially leading to poor predictions for many of our data points. Additionally, OLS assumes that features are not perfectly correlated (no multicollinearity), as this can make the matrix inversion unstable and lead to unreliable coefficient estimates.

The method also assumes that errors are normally distributed with constant variance (homoscedasticity), and that observations are independent. Violations of these assumptions can lead to biased standard errors and incorrect statistical inferences. Finally, OLS doesn't perform any feature selection, so it will use all provided features even if some are irrelevant or redundant, potentially leading to overfitting in high-dimensional settings.