Forecasting Baselines, Temporal Validation, and Backtesting › Visualizing Forecasting Baselines and Backtests

We generate a daily synthetic demand series with a slow trend, a weekly seasonal pattern, and noise. The baselines expose different parts of that structure: naive retains the latest level, seasonal naive repeats the last cycle, and drift extrapolates an endpoint slope. None is required to win.

We choose a forecast origin and reveal the next 35 days only for display and scoring. The forecasting functions receive history ending at that origin.

The figure below shows the recent history, the forecast origin, the three baseline paths, and the future actual values after the fact. The vertical line is the information boundary. The observed demand values to the left are available at the origin; actual demand values to the right are withheld until scoring. Future dates and the forecast paths issued at the origin are already known.

The naive path is flat because it repeats the last observation. The seasonal-naive path repeats the last seven observations, so it preserves the weekly shape but not the slow trend. The drift path is a straight line. The future actual series combines trend, weekly shape, and noise, so no baseline matches it exactly. This is the point: the figure shows the assumptions of each method rather than pretending one method is universally correct.

A single origin lets us compare realized losses on one target block, but gives limited evidence across forecasting occasions. We now evaluate at many rolling origins with an expanding window. At each origin, we forecast 28 days ahead and record the absolute error at every horizon. The code uses only the history available at that origin. The future actual values are used only for scoring.

The overall table aggregates the same error rows across all horizons, so it should be read as a compact summary rather than as a horizon-specific statement. MAE and RMSE are in demand units, so their magnitudes depend on the scale of the synthetic series, while bias has the same units and carries a direction.

Across the rolling origins, the baseline with the lowest MAE is a useful reference point rather than a final verdict. RMSE is reported on the same rows, so it reflects the same forecasts; a larger RMSE than MAE indicates variation in absolute error magnitudes, whereas equality requires equal absolute errors. On the same rows, RMSE squared equals MAE squared plus the population variance of the absolute errors; the gap alone does not identify a few outliers. The bias column shows whether the baseline tends to over- or under-forecast across all horizons, and the error count alone does not establish independence among those origin-horizon records. The comparison is most meaningful per horizon, which the next figure makes visible.

The next figure compares MAE by horizon. The horizontal axis is the lead time in days. The vertical axis is in demand units. A curve that rises quickly indicates that the forecast loses accuracy as the horizon grows. Low points near multiples of seven show a weekly phase pattern in these errors. Even a flat naive forecast can have such low points when its latest observed phase recurs; this feature alone does not show that a method has modeled seasonality. We should read this figure together with the number of origins and the scale of the series; a small MAE is not meaningful without that context.

The backtest curve is more informative than a single number because it shows where a model fails. A model that is excellent at (h=1) and poor at (h=28) may still be useful for next-day replenishment and useless for monthly planning. A model that is mediocre at every horizon may be easier to maintain. The baseline comparison gives us an accuracy reference. Whether an improvement justifies a more complex method also depends on its operational costs and risks.

Bias is a separate diagnostic from MAE. The next figure uses the same rolling-origin error records and shows whether each baseline tends to over- or under-forecast as the horizon grows.