Time-Series Models for Financial Data › Autoregressive (AR) Models

Autoregressive models capture the idea that today's value depends linearly on past values. This concept reflects a fundamental intuition about how many processes evolve: the current state of a system depends on where it was recently. An AR(p) model expresses the current observation as a weighted sum of the previous $p$ observations plus a noise term. The term "autoregressive" means the series is regressed on its own past values, unlike ordinary regression which uses external variables.

To capture the linear dependence of the current observation on its immediate predecessor, the simplest autoregressive model is AR(1). This model embodies the idea that today's value is pulled toward its previous value, with some random perturbation added. Mathematically:

$$Y_t = c + \phi Y_{t-1} + \varepsilon_t$$

where:

• $Y_t$: value of the series at time $t$
• $Y_{t-1}$: value of the series at time $t-1$
• $c$: constant (intercept)
• $\phi$: autoregressive coefficient
• $\varepsilon_t$: white noise term with mean 0 and variance $\sigma^2$

The white noise terms are assumed independent: $\text{Cov}(\varepsilon_t, \varepsilon_s) = 0$ for $t \neq s$. This assumption places all predictable structure in the autoregressive relationship rather than in the shocks themselves. The parameter $\phi$ controls the strength and direction of persistence. When $\phi$ is positive, high values tend to be followed by high values and low values by low values. When $\phi$ is negative, the series tends to oscillate, with high values followed by low values and vice versa. The magnitude of $\phi$ determines how strong this tendency is.

For AR(1) to be stationary, we require $|\phi| < 1$. This condition ensures that the influence of past shocks decays over time rather than persisting indefinitely or exploding. To see why this condition is necessary, assume stationarity holds and take expectations of both sides of the AR(1) equation:

$$E[Y_t] = c + \phi E[Y_{t-1}] + E[\varepsilon_t]$$

where:

• $E[\cdot]$: expectation operator
• $Y_t$: value of the series at time $t$
• $c$: constant term
• $\phi$: autoregressive coefficient
• $\varepsilon_t$: white noise term

Since $E[Y_t] = E[Y_{t-1}] = \mu$ under stationarity (the mean must be the same at all time periods), and $E[\varepsilon_t] = 0$ by the white noise assumption, we can solve for the unconditional mean:

$$\begin{aligned} \mu &= c + \phi \mu && \text{(stationarity condition)} \\ \mu(1 - \phi) &= c && \text{(rearrange terms)} \\ \mu &= \frac{c}{1-\phi} && \text{(solve for } \mu \text{)} \end{aligned}$$

where:

• $\mu$: mean of the process
• $c$: constant term
• $\phi$: autoregressive coefficient

This solution exists only if $\phi \neq 1$. If $\phi = 1$, the denominator becomes zero and no finite mean exists, indicating a non-stationary process known as a random walk. To compute the variance, we use the independence of $Y_{t-1}$ and $\varepsilon_t$. This independence follows from the assumption that $\varepsilon_t$ is unpredictable given past information:

$$\begin{aligned} \text{Var}(Y_t) &= \text{Var}(c + \phi Y_{t-1} + \varepsilon_t) \\ &= \phi^2 \text{Var}(Y_{t-1}) + \text{Var}(\varepsilon_t) && \text{(independent terms)} \\ &= \phi^2 \text{Var}(Y_{t-1}) + \sigma^2 && \text{(simplify)} \end{aligned}$$

where:

• $\text{Var}(Y_t)$: variance at time $t$
• $c$: constant term
• $\phi$: autoregressive coefficient
• $\varepsilon_t$: white noise term
• $\sigma^2$: variance of the white noise term

Under stationarity with $\text{Var}(Y_t) = \text{Var}(Y_{t-1}) = \gamma_0$, we can substitute and solve for the unconditional variance:

$$\begin{aligned} \gamma_0 &= \phi^2 \gamma_0 + \sigma^2 && \text{(stationarity condition)} \\ \gamma_0(1 - \phi^2) &= \sigma^2 && \text{(rearrange terms)} \\ \gamma_0 &= \frac{\sigma^2}{1-\phi^2} && \text{(solve for } \gamma_0 \text{)} \end{aligned}$$

where:

• $\gamma_0$: variance of the process ($\text{Var}(Y_t)$)
• $\sigma^2$: variance of the white noise term
• $\phi$: autoregressive coefficient

This expression is positive and finite only if $\phi^2 < 1$, meaning $|\phi| < 1$. When $|\phi| \geq 1$, the variance formula would yield either a negative number (impossible) or infinity (the series explodes). The stationarity condition thus emerges naturally from requiring that both the mean and variance be well-defined finite constants.

The autocorrelation function of AR(1) decays geometrically, a signature pattern that helps us identify AR processes in empirical data. To derive the autocovariance, we work with the mean-centered series to simplify calculations. The deviation from the mean satisfies $Y_t - \mu = \phi(Y_{t-1} - \mu) + \varepsilon_t$. Multiplying both sides by $(Y_{t-k} - \mu)$ for $k \geq 1$ and taking expectations yields the autocovariance at lag $k$:

$$\begin{aligned} \gamma_k &= E[(Y_t - \mu)(Y_{t-k} - \mu)] && \text{(definition of autocovariance)} \\ &= E[(\phi(Y_{t-1} - \mu) + \varepsilon_t)(Y_{t-k} - \mu)] && \text{(substitute deviation form)} \\ &= \phi E[(Y_{t-1} - \mu)(Y_{t-k} - \mu)] + E[\varepsilon_t(Y_{t-k} - \mu)] && \text{(linearity of expectation)} \\ &= \phi \gamma_{k-1} + 0 && \text{(uncorrelated error)} \end{aligned}$$

where:

• $\gamma_k$: autocovariance at lag $k$
• $E[\cdot]$: expectation operator
• $Y_t$: value of the series at time $t$
• $\mu$: mean of the process
• $\phi$: autoregressive coefficient
• $\varepsilon_t$: white noise term
• $\gamma_{k-1}$: autocovariance at lag $k-1$

The key insight is that $E[\varepsilon_t(Y_{t-k} - \mu)] = 0$ for $k \geq 1$ because the current shock $\varepsilon_t$ is uncorrelated with past values of the series. This recursion $\gamma_k = \phi \gamma_{k-1}$ yields $\gamma_k = \phi^k \gamma_0$ by repeated substitution, so the autocorrelation function has an elegant closed form:

$$\rho_k = \phi^k$$

where:

• $\rho_k$: autocorrelation at lag $k$
• $\phi$: autoregressive coefficient
• $k$: lag number

When $\phi > 0$, correlations decay exponentially but remain positive at all lags. This positive persistence means high values tend to be followed by somewhat high values, with the memory gradually fading. When $\phi < 0$, correlations alternate in sign while decaying in magnitude. This creates an oscillating pattern where high values tend to be followed by low values, then moderately high values, and so on. The geometric decay pattern is the hallmark of AR(1) and provides a clear diagnostic signature in sample ACF plots.

The AR(p) model extends the first-order case to include $p$ lags, allowing for more complex dependence structures where observations depend on multiple past values:

$$Y_t = c + \phi_1 Y_{t-1} + \phi_2 Y_{t-2} + \cdots + \phi_p Y_{t-p} + \varepsilon_t$$

where:

• $Y_t$: value of the series at time $t$
• $c$: constant term
• $\phi_i$: autoregressive coefficient at lag $i$
• $\varepsilon_t$: white noise error term

Multiple lags can represent dynamics unavailable to an AR(1) model. For instance, an AR(2) model might exhibit cyclical behavior if the coefficients are chosen appropriately. Using the lag operator $L$ where $L Y_t = Y_{t-1}$, we can express the AR(p) model compactly:

$$(1 - \phi_1 L - \phi_2 L^2 - \cdots - \phi_p L^p) Y_t = c + \varepsilon_t$$

where:

• $Y_t$: value of the series at time $t$
• $L$: lag operator such that $L^k Y_t = Y_{t-k}$
• $\phi_i$: autoregressive coefficient at lag $i$
• $c$: constant term
• $\varepsilon_t$: white noise error term

We write this as $\Phi(L) Y_t = c + \varepsilon_t$ where $\Phi(L) = 1 - \phi_1 L - \cdots - \phi_p L^p$ is called the autoregressive polynomial. The lag operator notation provides compactness and mathematical power, allowing us to manipulate time-series equations using algebraic techniques.

Stationarity requires all roots of the characteristic polynomial $\Phi(z) = 0$ to lie outside the unit circle in the complex plane. This generalizes the AR(1) condition $|\phi| < 1$ to higher-order models. For AR(2) with $Y_t = c + \phi_1 Y_{t-1} + \phi_2 Y_{t-2} + \varepsilon_t$, the stationarity conditions can be expressed as three inequalities:

$$\begin{aligned} \phi_1 + \phi_2 &< 1 \\ \phi_2 - \phi_1 &< 1 \\ |\phi_2| &< 1 \end{aligned}$$

where:

• $\phi_1, \phi_2$: autoregressive coefficients of the AR(2) model

These conditions define a triangular region in the $(\phi_1, \phi_2)$ parameter space where stationary AR(2) processes exist. Outside this region, the process either explodes or exhibits non-stationary wandering behavior.

The partial autocorrelation at lag $k$, denoted $\alpha_k$, measures the correlation between $Y_t$ and $Y_{t-k}$ after removing the linear effects of intermediate lags $Y_{t-1}, \ldots, Y_{t-k+1}$. This "partial" correlation isolates the direct relationship between observations $k$ periods apart, controlling for everything in between. The intuition is similar to partial correlation in regression analysis: we want to know how much additional information $Y_{t-k}$ provides about $Y_t$ beyond what is already captured by the intervening observations.

For an AR(p) process, a key identification property holds:

$$\alpha_k = \begin{cases} \neq 0 & \text{for } k \leq p \\ = 0 & \text{for } k > p \end{cases}$$

where:

• $\alpha_k$: partial autocorrelation at lag $k$
• $p$: order of the autoregressive process

This cutoff property follows from the structure of the AR(p) model. Since $Y_t$ depends directly only on $Y_{t-1}$ through $Y_{t-p}$, once we account for these intermediate values, $Y_{t-k}$ for $k > p$ provides no additional predictive information. The PACF cuts off sharply after lag $p$, while the ACF decays gradually. This stark contrast makes the PACF the primary tool for identifying AR order: we look for the lag after which all partial autocorrelations become insignificant.