Forecasting Baselines, Temporal Validation, and Backtesting › Practical Implications

Baselines should be part of every forecasting workflow, not a routine first row in a results table. The naive forecast tests the predictive value of the latest level; seasonal naive tests repetition of the observed seasonal pattern; drift tests endpoint-based trend extrapolation. If a more complex model cannot improve accuracy at the horizon that drives the operational decision, the simpler forecast may be the better choice once maintenance, monitoring, and failure modes are included. The comparison is most informative when the benchmark is the strongest simple method for that decision, not a deliberately weak one.

Temporal validation should mirror the deployment cadence. Match the issuance schedule and the latest response actually available. In the Monday-morning example, sales are final through Sunday: use Sunday response-index origins and score the next 28 daily targets, beginning Monday. For daily next-day forecasting with the current day finalized, use that day as the response-index origin and a one-day horizon.

• The origin schedule, maximum horizon, window type, seasonal period, and step between origins should be reported together, because a single aggregate score over a mix of horizons can hide whether a model is useful at the lead time the business plans for.

• Expanding windows are appropriate when old data remain relevant, rolling windows when the process changes; when a structural break is suspected, reporting both is a useful sensitivity check. Set initial_train larger than the seasonal period and long enough to cover several cycles; in a rolling window it also fixes the window length.

• A step of 1 maximizes evaluations, with $H-1$ shared target dates between adjacent blocks. A step equal to the horizon removes target-block overlap and generally reduces the evaluation count; serial dependence and shared training histories can remain.

Feature availability is one source of leakage. A feature is admissible at origin (o) only if its value is known at (o); rolling averages of realized responses must use only observations available at that origin. Windows over covariates known in advance may include future dates. Scalers, imputers, and any other preprocessing statistics must be fitted on the history available at that origin and then applied forward. Calendar features and approved promotion plans are usually safe because they are known in advance, while unreleased realized sales, traffic, and weather revisions are not. Missing timestamps and irregular gaps need the same treatment in backtesting as in production: filling a gap with a value released after the forecast origin creates leakage. A later observation already available at that origin can be used by the same documented production rule, but not retrospectively at an earlier origin where it was unknown. A training-window statistic is another option and can distort seasonality, so the handling should be documented.

Error metrics should match the decision and be reported by horizon. MAE is natural when the cost of a unit error is roughly linear, RMSE when large errors are disproportionately costly, and bias when under-forecasting and over-forecasting have different consequences. MAPE should be avoided or accompanied by an explicit exclusion rule when the target can be zero, and MASE is useful for scale-free comparison only when its denominator is computed from training history. A single aggregate MAE can conceal a model that is strong one step ahead and poor four weeks ahead, so horizon-specific tables and a compact overall summary are usually more informative than one number. When empirical intervals are reported from backtest residuals, their coverage should be checked by horizon; overlapping origins can induce dependence, so row count alone does not establish precision.

The following pitfalls can invalidate the comparison. Fitting a scaler with values unavailable at the origin, computing a rolling average of realized responses that includes the unavailable target-day outcome, using a promotion flag that was unavailable at the forecast origin, or computing the MASE scale with test targets makes the evaluation invalid for the stated information boundary and can bias it optimistically. Selecting the window, horizon, or model after inspecting the test score also invalidates an untouched-test interpretation, and treating correlated forecast errors as independent can misstate precision. Drift is sensitive to unusual endpoints, seasonal naive is rigid when the seasonal pattern changes, and naive is flat when a trend or promotion is present; these weaknesses should be diagnosed rather than hidden by an aggregate score.

Computational cost and deployment needs should be considered before the validation design is fixed. If deployment refits at every origin, backtest cost grows with the number of origins and the cost of each fit; with other settings fixed, a larger step cannot increase the evaluation count and may reduce computational cost. Adjacent target-block overlap cannot increase as the spacing grows and stays zero once the blocks are disjoint; weaker error dependence is not guaranteed. For long series or many models, origins can be parallelized or errors aggregated in a streaming table, but the origin, horizon, and history boundary must remain explicit in every run. In production, the origin cadence, horizon, window type, seasonal period, and MASE scale should be part of the monitoring contract, and performance should be tracked by horizon, bias, and interval coverage rather than only by a single aggregate. Baselines are useful as a fallback and as a live diagnostic: if a complex model stops beating the relevant simple benchmark, that is a signal to check for leakage, drift, or a changed process before retraining blindly.