Forecasting Baselines, Temporal Validation, and Backtesting › Formula

Forecasting notation earns its keep by keeping two times apart: the time when the forecast is made and the time it is meant to describe. If we blur those times, the model can accidentally learn from the very future it is supposed to predict. The formulas below build a precise contract around that separation. We first define the forecast origin and horizon, then the information set, then the three baseline rules, then the backtest that turns a single forecast into repeated evidence, and finally the error measures that summarize that evidence.

Let the observed series be (y_1,\ldots,y_T), where (T) is the last time index with a realized target. We use one-based time subscripts in the equations. In code, Python indices are zero-based, so the observation at time (t) is stored at position (t-1). A forecast made at origin (o) for target time (o+h) is written

$$\hat y_{o+h\mid o}.$$

A one-step-ahead forecast has $h=1$, a four-week-ahead forecast has $h=28$ if the data are daily and the horizon is measured in days, and a forecast for the same weekday next week has $h=7$. The notation $o+h\mid o$ is deliberate: it keeps the target time and the information time separate. Every later formula inherits that separation.

The forecast origin is the last time point whose target value is allowed to influence the forecast. It is a wall, not a suggestion. If we stand at the end of day $o$, the information set for a univariate baseline is

$$\mathcal I_o=\{y_1,y_2,\ldots,y_o\}.$$

This set contains the past and nothing else. It does not contain (y_{o+1}), even if that value happens to be sitting in the same data frame. More generally, if we have covariates (x_s), the information set is

$$\mathcal I_o=\{y_1,\ldots,y_o\}\cup\{x_s:\text{\(x_s\) is known by time \(o\)}\}.$$

A forecast rule is a function

$$\hat y_{o+h\mid o}=f_{o,h}(\mathcal I_o),$$

which maps the available time-stamped records into a real-valued forecast. The braces in $\mathcal I_o$ are shorthand for those records, retaining their times and repeated values. For a baseline, the function is simple. For a machine-learning model, the function may be a fitted regression or tree ensemble, but the domain is still the information set. The target (y_{o+h}) is not in (\mathcal I_o). This is the formal version of an everyday rule: we cannot use next week's sales to predict next week's sales.

The distinction is the time when the feature value became available. A value observed or released only after origin (o) cannot be used there, even if a retrospective table now contains it. For example, a feature “sales the day before the target” can be available for a one-day-ahead forecast made today once today’s sales have been released. For a seven-day-ahead forecast made at the same origin, it would require an unavailable future realization. A feature “sales on the target day” is never available at the origin. Calendar features such as day of week, month, and planned holidays can be known in advance because they are deterministic or pre-scheduled. Planned promotions can also be known in advance, while their future realized effects are unavailable until observed.

It helps to sort features by when their values become known:

• Known before the origin: calendar structure, scheduled holidays, contractual prices, and approved promotion plans.
• Observed by the origin: the latest observed sales, inventory, and traffic.
• Known only after the origin: future realized sales and footfall, weather revisions released after the origin, and aggregates of realized responses that include an unavailable target-day outcome.

Only the first two groups belong in (\mathcal I_o). Values in the third group stay outside that forecast's inputs; after release, they may be used for scoring or retrospective evaluation.

For a direct (h)-step model, a training row is often written as

$$(x_{s,h}, y_{s+h}),$$

where (x_{s,h}) must be constructed only from information available at origin (s). At fitting origin (o), the target (y_{s+h}) must also have been released already; with no reporting delay this requires (s+h\le o). Revised or delayed data require the historical vintage actually available then. The subscript (h) reminds us that the feature set can differ by horizon. For a recursive model, we forecast one step, append the forecast to the history, and forecast again. Recursive forecasts can propagate errors because later forecasts use earlier predictions as inputs. The closed-form baseline implementations here produce every horizon from the same observed history without feeding predictions back into the functions.

A forecast is evaluable only if the target has been observed. For a fixed maximum horizon (H), an origin (o) is valid when

$$o+H\le T.$$

The target block is then (y_{o+1},\ldots,y_{o+H}). This inequality is the basic guard against accidentally scoring a forecast whose target has not happened. It also explains why backtests shorten as (H) grows: the last usable origins must leave room for the entire target block.

The three baselines encode different assumptions about how the future relates to the past. Naive and seasonal naive are conditional means under the corresponding zero-mean innovation random walks. The endpoint drift rule plugs an estimated drift into a random-walk-with-drift forecast. These interpretations explain the mechanisms without promising accuracy for an arbitrary series.

The naive forecast uses the last observed value for every horizon:

$$\hat y^{\text{naive}}_{T+h\mid T}=y_T,\qquad h=1,\ldots,H.$$

Here $T$ is the forecast origin, $y_T$ is its latest observed value, and $h$ is the lead time. Under a random walk

$$y_t=y_{t-1}+\varepsilon_t,\qquad \mathbb E[\varepsilon_t\mid\mathcal I_{t-1}]=0,$$

with integrable innovations, iterated conditional expectation gives $\mathbb E[Y_{T+h}\mid\mathcal I_T]=y_T$. A trend or changing seasonal pattern can make this model unsuitable; the baseline remains a useful matched-horizon comparison.

The seasonal-naive forecast repeats the last observed seasonal cycle. Let $m$ be the seasonal period: $m=7$ for daily data with a weekly pattern, $m=12$ for monthly data with an annual pattern, and $m=4$ for quarterly data.

The floor term selects the correct cycle offset. Define

$$k=\left\lfloor\frac{h-1}{m}\right\rfloor,$$

where (\lfloor\cdot\rfloor) is the floor function, which rounds down to the nearest integer. The seasonal-naive forecast is

$$\hat y^{\text{snaive}}_{T+h\mid T}=y_{T+h-m(k+1)}.$$

The index $T+h-m(k+1)$ is the most recent observation that has the same position in the seasonal cycle as the target. For $h=1$, $k=0$, so the forecast is $y_{T+1-m}$, the same season one cycle earlier. For $h=m$, $k=0$, so the forecast is $y_T$, the last observed value, which is the same season as the target. For $h=m+1$, $k=1$, so the forecast is again $y_{T+1-m}$. This is the most recent observed value at the target's seasonal position, now two cycles before the target.

where:

• $\hat y^{\text{snaive}}_{T+h\mid T}$: the seasonal-naive forecast for target time $T+h$
• $m$: the seasonal period (e.g., 7 for a weekly cycle in daily data)
• $h$: the lead time in periods
• $k=\lfloor (h-1)/m \rfloor$: the index of the block of $m$ future horizons containing $h$, starting at block zero

With integrable innovations satisfying $\mathbb E[\varepsilon_t\mid\mathcal I_{t-1}]=0$, the method is the conditional mean forecast under a seasonal random walk

$$y_t=y_{t-m}+\varepsilon_t.$$

Its weakness is rigidity. Within a forecast block it repeats the last observed seasonal cycle exactly, without smoothing its noise or extrapolating a new level or trend. If the seasonal pattern is strong and stable, that rigidity is a strength. If a promotion or a new competitor changes the pattern, the same rigidity becomes a liability.

The drift baseline extends a straight line from the first observation to the last observation. It requires at least two observations, $T\ge2$. The estimated drift per period is

$$\hat c=\frac{y_T-y_1}{T-1}.$$

where:

• $\hat c$: the estimated drift per period
• $y_T$: the last observation in the history
• $y_1$: the first observation in the history
• $T$: the number of observations (so $T-1$ is the number of intervals between them)

The (h)-step forecast is

$$\hat y^{\text{drift}}_{T+h\mid T}=y_T+h\hat c.$$

For $h=1$, we add one drift increment to the last observation. For $h=2$, we add two increments, and so on.

where:

• $\hat y^{\text{drift}}_{T+h\mid T}$: the drift forecast for target time $T+h$
• $y_T$: the last observed value at the origin
• $\hat c$: the estimated drift per period
• $h$: the lead time in periods

This plugs an estimated drift into the conditional-mean formula for a random walk with drift,

$$y_t=y_{t-1}+c+\varepsilon_t,$$

where:

• $y_t$: the observed value at time $t$
• $y_{t-1}$: the previous period's observation
• $c$: the constant drift per period
• $\varepsilon_t$: the innovation at time $t$, with conditional mean zero

The estimate $\hat c=(y_T-y_1)/(T-1)$ is simple but endpoint-sensitive. A single unusual first or last observation can change the slope substantially. An alternative is to estimate a slope by ordinary least squares on the training history, which uses interior observations and can reduce sensitivity to a noisy endpoint under an appropriate trend model, but then the method is closer to a linear trend model than to the canonical drift baseline. Either version should be stated explicitly when reporting results.

It is useful to remember the assumptions as a compact contrast:

• Naive: the future level is the last level.
• Seasonal naive: the future calendar position repeats the most recent occurrence of that position.
• Drift: the future continues a straight line whose slope is fixed by the observed endpoints.

Each baseline is a hypothesis about the data-generating process. Backtesting asks whether that hypothesis is good enough for the decision at hand.

A fixed-origin holdout forecast gives one forecast origin and one target block. That is often too little evidence to judge a model, and it is easy for one origin to flatter or punish a method by accident. Rolling-origin backtesting repeats the forecast exercise at a sequence of origins

$$o_1<o_2<\cdots<o_K,$$

with (o_k+H\le T) for every origin. At origin (o_k), the model sees information through (o_k), forecasts (H) future values, and is scored on (y_{o_k+1},\ldots,y_{o_k+H}). Then the origin advances and the process repeats. The backtest is a sequence of small, honest forecasting problems, not one large hindsight problem.

In an expanding-window backtest, the training history grows:

$$\mathcal W^{\text{exp}}_{o_k}=\{y_1,\ldots,y_{o_k}\}.$$

This is appropriate when older data remain relevant and when we want to mimic a system that accumulates history over time. The first origin has the smallest training set; later origins have more data. More history can improve estimates under a stable seasonal model, but expanding windows can also mix incompatible regimes.

In a rolling-window backtest, only the most recent (w) observations are used:

$$\mathcal W^{\text{roll}}_{o_k,w}=\{y_{o_k-w+1},\ldots,y_{o_k}\}.$$

This is appropriate when the data-generating process changes and old observations would bias the forecast. The window length (w) is a hyperparameter, and it must be chosen without looking at the test targets. A rolling window throws away information, so it can increase variance, but it can reduce bias after a structural break.

The spacing between origins is also a choice. Using every possible origin maximizes the number of evaluations. Consecutive target blocks share exactly $H-1$ dates, so there is no target overlap when $H=1$. Spacing origins by $H$ periods makes the target blocks disjoint and generally reduces the number of evaluations; the count can be unchanged when $H=1$ or only one origin is evaluable. The number of forecast errors alone does not establish independence among them. If we evaluate 100 daily origins with a 28-day horizon, the target blocks overlap, and the errors can be dependent. We should report the number of origins, the horizon, the window type, and the step, not only the number of error rows.

Once the forecasts are frozen and the targets are revealed, we need a way to turn many forecast errors into evidence. The raw object is the forecast error. For an origin (o) and horizon (h), the forecast error is

$$e_{o,h}=y_{o+h}-\hat y_{o+h\mid o}.$$

The sign convention matters: a positive error means the actual value was above the forecast. If we reverse the sign, bias has the opposite interpretation.

where:

• $e_{o,h}$: the forecast error at origin $o$ and horizon $h$
• $y_{o+h}$: the realized target value at time $o+h$
• $\hat y_{o+h\mid o}$: the forecast made at origin $o$ for target time $o+h$

The sign convention gives a positive error when the actual value exceeds the forecast. The absolute error is $|e_{o,h}|$, and the squared error is $e_{o,h}^2$. Every metric below is a different way of summarizing these elementary errors.

Let (\mathcal O_h) be the set of origins for which horizon (h) is evaluable. The mean absolute error at horizon $h$ is

$$\operatorname{MAE}_h = \frac{1}{|\mathcal O_h|} \sum_{o\in\mathcal O_h}|e_{o,h}|.$$

where:

• $\mathcal O_h$: the set of origins for which horizon $h$ is evaluable
• $|\mathcal O_h|$: the number of such origins
• $e_{o,h}$: the forecast error at origin $o$ and horizon $h$

The root mean squared error is

$$\operatorname{RMSE}_h = \sqrt{ \frac{1}{|\mathcal O_h|} \sum_{o\in\mathcal O_h}e_{o,h}^2 }.$$

where:

• $\mathcal O_h$: the set of origins evaluable at horizon $h$
• $e_{o,h}$: the forecast error at origin $o$ and horizon $h$

MAE is in the original units and is easy to explain. RMSE penalizes large errors more heavily because the errors are squared before averaging. Neither is universally better; they answer different loss questions. The mean error, or bias, is

$$\operatorname{Bias}_h = \frac{1}{|\mathcal O_h|} \sum_{o\in\mathcal O_h}e_{o,h}.$$

where:

• $\mathcal O_h$: the set of origins evaluable at horizon $h$
• $e_{o,h}$: the signed forecast error at origin $o$ and horizon $h$
• $\operatorname{Bias}_h$: positive values indicate under-forecasting, negative values over-forecasting

A model can have low MAE and still be systematically biased. Reporting bias alongside MAE and RMSE prevents that mistake.

Percentage errors are tempting because they are scale-free, but they have a zero problem. The mean absolute percentage error is often written as

$$\operatorname{MAPE}_h = \frac{100}{|\mathcal O_h|} \sum_{o\in\mathcal O_h} \left|\frac{e_{o,h}}{y_{o+h}}\right|.$$

where:

• $e_{o,h}$: the forecast error at origin $o$ and horizon $h$
• $y_{o+h}$: the realized target value at time $o+h$
• $\mathcal O_h$: the set of origins evaluable at horizon $h$

If (y_{o+h}=0), the ratio is undefined. If (y_{o+h}) is near zero, the percentage error can explode even when the absolute error is small. We should not silently drop those rows without reporting the exclusion, because dropping zeros changes the target population. A weighted absolute percentage error,

$$\operatorname{WAPE}_h = \frac{\sum_{o\in\mathcal O_h}|e_{o,h}|} {\sum_{o\in\mathcal O_h}|y_{o+h}|},$$

avoids division by each individual target. It is undefined when the sum of absolute targets is zero and can become unstable when that sum is near zero. Here $\mathcal O_h$ is the set of origins evaluable at horizon $h$, $e_{o,h}$ is the forecast error, and $y_{o+h}$ is the realized target. It is not a universal fix.

The mean absolute scaled error, or MASE, uses a seasonal-naive in-sample scale:

$$\operatorname{MASE}_h = \frac{\operatorname{MAE}_h} { \frac{1}{T-m} \sum_{t=m+1}^{T}|y_t-y_{t-m}| }.$$

where:

• $\operatorname{MAE}_h$: the out-of-sample MAE at horizon $h$
• $T$: the last index of the training history
• $m$: the seasonal period
• $y_t-y_{t-m}$: the seasonal difference between observation $t$ and the same season one cycle earlier

The denominator is the average absolute seasonal difference in the training history. If (m=1), it is the average absolute first difference, which is the naive scale. The numerator is the out-of-sample MAE at horizon (h). A MASE below 1 means the forecast has lower MAE than the seasonal-naive scale on the training data; a MASE above 1 means it is worse on that scale. The scale must be computed without test data. In a rolling backtest, we can compute the scale once from the initial training window, or recompute it at each origin from that origin's history. The first choice keeps the scale fixed and comparable across origins. The second updates it from the history retained at each origin: expanding histories accumulate observations, while rolling histories replace older observations with newer ones. For a per-origin scale $a_o$, average the components $|e_{o,h}|/a_o$; this generally differs from dividing pooled MAE by an average scale. State the convention. If the seasonal differences are all zero, the denominator is zero and MASE is undefined.

Uncertainty should be reported by horizon. At origin $o$, a residual-based interval can use only earlier forecast errors whose targets have already been released. With no reporting delay, define

$$\mathcal A_{o,h}=\{s:s+h\le o\},\qquad q_{p,o,h}=\operatorname{Quantile}_p\{e_{s,h}:s\in\mathcal A_{o,h}\}.$$

Each $e_{s,h}$ must come from a forecast frozen at its own origin $s$. Delayed labels additionally require their release date to be no later than $o$. When enough completed errors are available, the interval is

$$\left[\hat y_{o+h\mid o}+q_{\alpha/2,o,h},\quad\hat y_{o+h\mid o}+q_{1-\alpha/2,o,h}\right].$$

Here $p$ is a quantile probability and $\alpha$ is the desired miscoverage level. Error quantiles have the same units as the target. This heuristic assumes the future error distribution resembles the completed past errors. It gives no finite-sample coverage guarantee under arbitrary serial dependence. Evaluate interval coverage at later untouched origins, updating calibration only with errors completed by each origin. Pooling future errors before constructing earlier intervals would leak outcomes. Overlapping errors may be correlated, so row count alone does not determine uncertainty in the reported coverage fraction.

Reporting should include the forecast origins, the horizon, the seasonal period, the window type, the model names, and the error scale. A single aggregate MAE over a mix of horizons can hide the fact that a model is good one step ahead and poor four weeks ahead. A table by horizon, plus a compact overall summary, is usually more informative. A skill score relative to a baseline is also useful:

$$\operatorname{Skill} = 1-\frac{\operatorname{MAE}_{\text{model}}} {\operatorname{MAE}_{\text{baseline}}}.$$

where:

• $\operatorname{MAE}_{\text{model}}$: the mean absolute error of the model being evaluated
• $\operatorname{MAE}_{\text{baseline}}$: the mean absolute error of the chosen baseline, at the same origins, horizon, and error scale

A positive skill score means the model has lower MAE than the baseline; zero means it ties; negative means it is worse. The baseline and horizon must be stated. If baseline MAE is zero, this ratio is undefined. The skill score is most persuasive when the baseline is the strongest simple benchmark for the same decision, not the weakest straw man.

With integrable, conditionally mean-zero innovations, naive and seasonal naive are conditional means under their respective random-walk models. With known constant drift $c$, the conditional mean is $y_T+hc$; the implemented rule uses $\hat c$ instead. This is a plug-in estimate. Mean reversion, changing seasonality and other departures can make these models unsuitable.

The baselines are also linear functions of the observed history. The naive forecast is the identity function of the last observation and is constant across horizons at a fixed origin. The seasonal-naive forecast selects one of the last (m) observations according to the horizon. The drift forecast is an affine function of the first and last observations. This linearity makes their dependence on the observed history explicit, but also limits their flexibility. Finite inputs and valid history lengths are still required.

Expanding fitting windows are nested when history is appended without revision. Rolling fitting windows are generally not nested because older records leave the selected window. The full information available over time is a different object from the observations deliberately retained for fitting. More history can help in a stable model, while forgetting old regimes can help under change; neither window has a universal variance or accuracy advantage.

Forecast errors from overlapping origins can be dependent. If origin (o) and origin (o+1) both forecast target (o+28), their errors share the same future observation. Their retained fitting histories overlap according to the window policy: expanding histories are nested, while adjacent rolling windows of width $w$ share $w-1$ observations, including none when $w=1$. We can still average them, but we should not treat the average as if it came from independent samples. This is why a backtest with many overlapping origins can look more precise than it really is.