Forecasting Baselines, Temporal Validation, and Backtesting › Example

Before trusting a backtest loop, it is worth shrinking the problem until every number can be checked by hand. A small example exposes the bookkeeping: which observations are in the information set, which are targets, and how each baseline turns the same history into a different forecast.

The series is

We set the forecast origin at (T=8), so the observed history is (y_1,\ldots,y_8), and the target block is (y_9,\ldots,y_{12}). The seasonal period is (m=4), and the maximum horizon is (H=4). The history contains two full cycles and the target block one full cycle. It is also small enough that a reader can verify each forecast against the raw numbers.

For the naive forecast, every horizon repeats (y_8). In symbols,

So the naive path is (30,30,30,30), regardless of the upward movement in the history.

For the seasonal-naive forecast, we use (m=4). The floor formula gives (k=0) for (h=1,2,3,4), so the forecasts are (y_5,y_6,y_7,y_8). These are the last four observations, repeated in order. For (h=1), the target is (y_9), and the most recent same-season observation is (y_5). For (h=4), the target is (y_{12}), and the most recent same-season observation is (y_8). In this example the method carries the shape of the last cycle forward, including its local ups and downs, but it does not add the trend.

For the drift forecast, the estimated drift is

The forecasts are (y_8+h\hat c) for (h=1,2,3,4). The first observation and the last observation determine the slope. If either endpoint is unusual, the drift forecast changes. This is a useful feature to notice before applying drift to noisy operational data: the method is not estimating a trend from all the data, it is drawing a line between two boundary observations.

The code below computes exactly the same three forecasts from the same history. It does not look at the target block when constructing the predictions. The target block is attached only afterward for comparison.

The table below shows the forecasts and the realized targets. Read it row by row and compare it with the formulas. The naive column should be constant, because it repeats $y_8$. The seasonal-naive column should repeat the last four observations, because $k=0$ at every horizon. The drift column should increase by the same increment each step, because it adds $\hat c$ once, twice, three times, and four times.

The table is a useful audit because it separates three questions: Did the model use the right information set? Did the formula implement the intended assumption? Did the realized target happen to favor that assumption? The first two questions are about correctness; the third is about empirical performance.

The error for each model and horizon is actual minus forecast. We summarize with MAE and bias. The MASE denominator uses the seasonal differences in the training history $y_1,\ldots,y_8$, not the future targets. This is the leakage-free scaling convention. If we used the test targets in the denominator, the scale would contain information that the forecast did not have. The scale is therefore part of the training evidence, even though it is computed after the forecasts are made for reporting.

The next code cell reshapes the forecast table into a long format, computes signed and absolute errors, and builds the summary. The reshaping is not cosmetic: it makes the unit of analysis explicit as one model-horizon-forecast record, which is exactly the unit the later backtest will aggregate.

The small-example summaries make the forecasts concrete. The naive forecast is constant at 30, with signed errors 1, 3, 2, and 5; these misses are not monotone in horizon. Seasonal naive has signed errors 7, 6, 6, and 5. Drift extrapolates the endpoint slope and has signed errors −3/7, 1/7, −16/7, and −5/7.

The comparison is intentionally small enough to audit. The naive forecast ignores the upward movement in the series. The seasonal-naive forecast captures the repeating pattern but does not add the trend. The drift forecast adds a trend but ignores the within-season variation. In this example, the best model depends on which of those features dominates the target block. The important lesson is not that one baseline is always best; it is that the formulas, the information set, and the scoring rule are all visible and reproducible.

A signed error curve makes the direction of each miss visible in the small example. Positive values mean the actual value exceeded the forecast, and negative values mean it fell short.

A baseline win describes the evaluated target blocks under the chosen loss. A naive, seasonal-naive, or drift win can motivate further testing of recent level, recurring seasonal shape, or trend, respectively. A sample win alone does not establish that the corresponding pattern persists in the population. Compare a more complex model with the relevant baselines at the same origins and horizons before attributing an improvement to its added complexity.