Sum of Squared Errors (SSE) › SSE in Least Squares Regression

In linear regression, we find the best-fitting line by minimizing SSE. This is why the method is called "least squares" - we're finding the line that results in the least (smallest) sum of squared errors.

For a linear model $\hat{y} = \beta_0 + \beta_1 x$, we solve:

Using squared errors (rather than absolute errors) offers several important mathematical and practical advantages in regression analysis:

1. Differentiability: The squared error function, $(y_i - \hat{y}_i)^2$, is smooth and differentiable everywhere with respect to the model parameters. This property enables optimization algorithms, such as gradient descent, which rely on derivatives to find the minimum error. In contrast, the absolute error function $|y_i - \hat{y}_i|$ is not differentiable at zero, making optimization more challenging and often requiring specialized algorithms.

2. Penalizes large errors more heavily: Squaring the residuals means that larger errors have a disproportionately greater impact on the total SSE. For example, if a prediction is off by 2 units, it contributes $2^2 = 4$ to the SSE, while an error of 1 unit contributes only $1^2 = 1$. This property encourages models to avoid large mistakes, as they are penalized much more than small ones. In contrast, using absolute errors would treat all errors linearly, regardless of their size.

3. Mathematical tractability and closed-form solutions: The use of squared errors leads to elegant mathematical properties that make analysis and computation more straightforward. In particular, minimizing the sum of squared errors in linear regression yields a closed-form solution for the optimal parameters, known as the normal equations. This allows for efficient computation and a clear understanding of how the solution depends on the data. We cover this in the section on ordinary least squares (OLS).

4. Desirable statistical properties: Under the classical assumptions of linear regression.such as errors being independent, identically distributed, and normally distributed with constant variance.the least squares estimator (which minimizes SSE) is the Best Linear Unbiased Estimator (BLUE) according to the Gauss-Markov theorem. This means it has the lowest variance among all linear unbiased estimators. Additionally, when errors are normally distributed, the least squares estimator coincides with the maximum likelihood estimator.

5. Interpretability and connection to variance: The mean squared error (MSE), which is SSE divided by the number of observations, is directly related to the variance of the residuals. This connection makes it easy to interpret the goodness-of-fit of a model and compare models using familiar statistical concepts.

In summary, the use of squared errors in regression is not arbitrary.it is motivated by a combination of mathematical convenience, optimization efficiency, and desirable statistical properties that make it the standard choice in most regression settings.

The normal equations in linear regression are derived by taking the derivative of SSE with respect to the parameters and setting them equal to zero. This process reveals why SSE is central to the entire regression framework.