Validation Strategies and Data Leakage

A model can predict another observation from a familiar customer much more accurately than an observation from a customer it has never seen. A random row split can mix those two tasks when each customer contributes several records. The resulting score may look useful while answering a different question from the one the model will face at deployment.

Validation starts by defining what the model will know when it makes a prediction. We then construct held-out data that reproduce that situation as closely as the available sample permits. New-customer predictions require holding out whole customers; future predictions require respecting time and label availability. Learned preprocessing belongs inside the training boundary of each evaluation.

Building on Statistical Modelling, Standardization, and Normalization, we examine the evaluation procedure around a fitted model. A 1-nearest-neighbor (1-NN) regressor keeps the examples simple: it predicts with the target of the closest training observation. Its behavior also shows why the numerical effect of a poor split depends on the data and learner.

The chapter develops five parts of an evaluation procedure:

• Training fits parameters, validation guides choices, and final testing assesses the frozen choice.
• Preprocessing is part of the model and must be fitted inside the same training fold as the estimator.
• The split should match the task: preserve useful class proportions, hold out new entities, and respect prediction-time information.
• Out-of-fold predictions provide held-out predictions for covered rows; retain the fold assignments to record which observations each fitted pipeline excluded.
• Reproducibility requires a specified data snapshot, procedure, and fold assignment; randomized procedures also need recorded randomness settings.