Regression Analysis for Financial Relationships › The Ordinary Least Squares Framework

Regression analysis models the relationship between a dependent variable and one or more explanatory variables. In finance, we often want to explain asset returns using factor returns, or forecast prices using economic indicators. The power of regression lies in its ability to decompose variation: separating what can be explained by observable factors from what remains unexplained. This decomposition forms the basis for understanding risk, measuring performance, and building predictive models.

Consider a linear relationship between a dependent variable $y$ and an independent variable $x$. The core idea is straightforward: we believe that changes in $x$ produce predictable changes in $y$, but this relationship is obscured by random noise. Our goal is to uncover the underlying systematic relationship despite this noise.

$$y_i = \alpha + \beta x_i + \varepsilon_i$$

where:

• $y_i$: observation of the dependent variable for observation $i$
• $x_i$: corresponding independent variable
• $\alpha$: intercept, representing the expected value of $y$ when $x = 0$
• $\beta$: slope coefficient, representing the change in $y$ for a one-unit change in $x$
• $\varepsilon_i$: error term capturing the variation in $y$ not explained by $x$

The intercept $\alpha$ anchors the regression line at a specific level. In financial contexts, when we regress stock returns on market returns, the intercept represents the return we would expect when the market return is exactly zero. This parameter captures the baseline performance that exists independently of market movements.

The slope coefficient $\beta$ is the heart of the regression. It tells us the marginal effect: for every one-unit increase in $x$, we expect $y$ to change by $\beta$ units on average. In the context of finance, if $\beta = 1.3$ in a stock-versus-market regression, then a 1% market move corresponds to an expected 1.3% move in the stock. This amplification (or dampening, if $\beta < 1$) characterizes the stock's systematic risk profile.

The error term $\varepsilon_i$ represents everything we cannot observe or measure that affects $y$. This includes firm-specific news, measurement error, and the influence of countless unmeasured factors. We assume it has zero mean, constant variance, and is uncorrelated across observations; these are assumptions we'll scrutinize carefully later. These assumptions matter because they determine whether our statistical inference is valid and whether our confidence intervals are trustworthy.

Having specified the model, we need a method to estimate the unknown parameters $\alpha$ and $\beta$. Ordinary least squares takes an intuitive approach: find the line that minimizes the total squared distance between observed values and the line's predictions. Why squared distances? Squaring serves two purposes: it treats positive and negative deviations symmetrically, and it penalizes large errors more heavily than small ones, making the estimator sensitive to outliers but also more efficient when errors are normally distributed.

Ordinary least squares finds the estimators by minimizing the sum of squared residuals $S$:

$$S(\alpha, \beta) = \sum_{i=1}^{n} (y_i - \alpha - \beta x_i)^2$$

where:

• $S$: sum of squared residuals to be minimized
• $y_i, x_i$: dependent and independent variable observations for item $i$
• $\alpha, \beta$: intercept and slope coefficients to be estimated
• $n$: number of observations

This objective function creates a surface in two-dimensional parameter space. The optimal parameter values correspond to the lowest point on this surface, where the total squared error reaches its minimum. Finding this minimum requires calculus: we take derivatives with respect to each parameter and set them equal to zero.

Taking partial derivatives with respect to $\alpha$ and $\beta$ and setting them to zero yields the normal equations:

$$\begin{aligned} \frac{\partial S}{\partial \alpha} &= -2 \sum_{i=1}^n (y_i - \alpha - \beta x_i) = 0 \\ \frac{\partial S}{\partial \beta} &= -2 \sum_{i=1}^n x_i (y_i - \alpha - \beta x_i) = 0 \end{aligned}$$

The first equation has an intuitive interpretation. It says that the sum of residuals must equal zero, meaning the regression line must pass through the center of mass of the data. Positive errors exactly balance negative errors. From the first equation, dividing by $n$ yields $\bar{y} - \alpha - \beta\bar{x} = 0$, which gives $\hat{\alpha} = \bar{y} - \hat{\beta}\bar{x}$. This result confirms that the fitted line passes through the point $(\bar{x}, \bar{y})$, the sample means of both variables.

Substituting this into the second equation eliminates $\alpha$:

$$\sum_{i=1}^n x_i (y_i - (\bar{y} - \hat{\beta}\bar{x}) - \hat{\beta} x_i) = 0$$

Rearranging terms to group $y$ and $x$:

$$\sum_{i=1}^n x_i (y_i - \bar{y}) = \hat{\beta} \sum_{i=1}^n x_i (x_i - \bar{x})$$

Using the algebraic identity $\sum x_i(z_i - \bar{z}) = \sum (x_i - \bar{x})(z_i - \bar{z})$ to center the variables:

$$\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y}) = \hat{\beta} \sum_{i=1}^n (x_i - \bar{x})^2$$

gives:

$$\hat{\beta} = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^{n}(x_i - \bar{x})^2} = \frac{\text{Cov}(x, y)}{\text{Var}(x)}$$ $$\hat{\alpha} = \bar{y} - \hat{\beta}\bar{x}$$

where:

• $\hat{\beta}$: estimated slope coefficient
• $\hat{\alpha}$: estimated intercept
• $\bar{x}, \bar{y}$: sample means of $x$ and $y$
• $\text{Cov}(x, y)$: sample covariance between the independent and dependent variables
• $\text{Var}(x)$: sample variance of the independent variable

The slope coefficient equals the covariance between $x$ and $y$ divided by the variance of $x$. This formula reveals the essential logic of regression. The numerator, covariance, measures how $x$ and $y$ move together. The denominator, variance, normalizes this co-movement by the spread of $x$. The ratio measures how much $y$ moves, on average, when $x$ moves by one unit. If $x$ has high variance, we have more information to estimate the relationship precisely, so we divide by a larger number. The intercept ensures the regression line passes through the point $(\bar{x}, \bar{y})$.

For multiple regression with $k$ explanatory variables, the matrix formulation is essential. The scalar derivation becomes unwieldy with multiple variables, but matrix algebra provides an elegant and general solution. This formulation also reveals the geometric interpretation: OLS finds the projection of the dependent variable onto the space spanned by the explanatory variables.

We write:

$$\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}$$

where:

• $\mathbf{y}$: $n \times 1$ vector of observations
• $\mathbf{X}$: $n \times (k+1)$ matrix of explanatory variables (including a column of ones for the intercept)
• $\boldsymbol{\beta}$: $(k+1) \times 1$ vector of coefficients
• $\boldsymbol{\varepsilon}$: $n \times 1$ vector of errors

The design matrix $\mathbf{X}$ stacks all observations of all explanatory variables. Each row represents one observation, and each column represents one variable. The first column typically consists of ones, which corresponds to the intercept term. This structure allows us to handle any number of explanatory variables using the same mathematical framework.

As we covered in Part I's treatment of linear algebra, the OLS solution minimizes the objective function $S(\boldsymbol{\beta})$. We expand the quadratic form and take the derivative:

$$\begin{aligned} S(\boldsymbol{\beta}) &= (\mathbf{y} - \mathbf{X}\boldsymbol{\beta})'(\mathbf{y} - \mathbf{X}\boldsymbol{\beta}) \\ &= \mathbf{y}'\mathbf{y} - \mathbf{y}'\mathbf{X}\boldsymbol{\beta} - \boldsymbol{\beta}'\mathbf{X}'\mathbf{y} + \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} \\ &= \mathbf{y}'\mathbf{y} - 2\boldsymbol{\beta}'\mathbf{X}'\mathbf{y} + \boldsymbol{\beta}'\mathbf{X}'\mathbf{X}\boldsymbol{\beta} && \text{(since } \mathbf{y}'\mathbf{X}\boldsymbol{\beta} \text{ is scalar)} \end{aligned}$$

The expansion follows from distributing the transpose operation and recognizing that $\mathbf{y}'\mathbf{X}\boldsymbol{\beta}$ produces a scalar. Since a scalar equals its own transpose, we have $\mathbf{y}'\mathbf{X}\boldsymbol{\beta} = \boldsymbol{\beta}'\mathbf{X}'\mathbf{y}$, allowing us to combine these terms.

Taking the derivative with respect to $\boldsymbol{\beta}$:

$$\frac{\partial S}{\partial \boldsymbol{\beta}} = -2\mathbf{X}'\mathbf{y} + 2\mathbf{X}'\mathbf{X}\boldsymbol{\beta}$$

Setting the derivative to zero yields the normal equations:

$$\mathbf{X}'\mathbf{X}\hat{\boldsymbol{\beta}} = \mathbf{X}'\mathbf{y}$$

These normal equations have a clear interpretation. The term $\mathbf{X}'\mathbf{y}$ captures the correlation between each explanatory variable and the dependent variable. The term $\mathbf{X}'\mathbf{X}$ is the Gram matrix of the regressors, encoding their variances and mutual correlations. The normal equations balance these two components.

Assuming $\mathbf{X}'\mathbf{X}$ is invertible:

$$\hat{\boldsymbol{\beta}} = (\mathbf{X}'\mathbf{X})^{-1}\mathbf{X}'\mathbf{y}$$

where:

• $\hat{\boldsymbol{\beta}}$: vector of estimated coefficients
• $\mathbf{X}$: design matrix of explanatory variables
• $\mathbf{y}$: vector of dependent variable observations

Intuitively, this generalizes the scalar slope formula $\text{Cov}(x,y)/\text{Var}(x)$. The term $\mathbf{X}'\mathbf{y}$ captures the covariation between regressors and the dependent variable, while $(\mathbf{X}'\mathbf{X})^{-1}$ normalizes by the variance-covariance structure of the regressors, effectively "dividing" by the variance while adjusting for correlations between explanatory variables. When regressors are correlated with each other, the matrix inverse accounts for this overlap, so each coefficient represents the unique contribution of its variable after controlling for all others.

This closed-form solution is the foundation for all OLS regression analysis. Its existence guarantees that we can always compute the optimal coefficients directly, without iterative optimization. The requirement that $\mathbf{X}'\mathbf{X}$ be invertible means no explanatory variable can be a perfect linear combination of others, a condition known as the absence of perfect multicollinearity.