Validation Strategies and Data Leakage › Summary

Validation specifies which information the fitted procedure may use and which deployment task the held-out score assesses. A random row split can be appropriate for an i.i.d. task, while an unseen-customer task needs whole-customer hold-outs and future prediction needs prediction-origin restrictions. Repeated entities alone do not make every evaluation invalid.

Expected risk and empirical mean loss coincide in expectation under the stated distribution and fit-independence conditions; nonlinear scores need separate care. Training fits parameters, validation guides choices, and final testing evaluates the frozen choice. Learned preprocessing belongs inside each training fold, and target or future information must be available at the prediction origin. Group disjointness and pipelines address specific boundaries, rather than proving the entire evaluation valid.

We worked through a four-customer, eight-row example with row-split RMSE 1.00 and group-split RMSE about 10.75. The six-customer simulation produces mean random-fold RMSE about 2.30, grouped-fold RMSE about 9.98, and grouped hold-out RMSE about 23.06. The spread between the latter two is a warning about the limited group count and the particular partitions, rather than evidence that two estimates have converged. The overlap and target-encoding experiments also show that a known information leak need not produce a fixed or even favorable score change. Split design and prediction-time feature availability must be checked directly.