Validation Strategies and Data Leakage › Formula

The formulas separate the population quantity we want from the sample quantity we can compute. We first define risk and held-out loss, then describe which observations may contribute to fitting. The grouped and temporal constraints express different deployment tasks; cross-validation repeats the corresponding fitting and evaluation procedure.

Let the collected observations be

$$\mathcal{D} = \{(x_i, y_i, g_i)\}_{i=1}^{n},$$

Here $x_i \in \mathbb{R}^d$ is the feature vector, $y_i \in \mathbb{R}$ is the regression target, and $g_i$ identifies the customer or other group. The examples use the group column to construct splits, rather than as an input to 1-NN. We use squared loss, $L(\hat y,y)=(\hat y-y)^2$; classification uses a loss suited to its target.

Let $\mathcal P$ be the joint distribution of a fresh deployment observation $(x,y)$, and let $f$ be a fixed predictor. Assume the loss $L(f(x),y)$ has a finite expectation. Generalization risk is its expected loss:

$$\mathcal{R}(f) = \mathbb{E}_{(x, y) \sim \mathcal{P}}\left[ L(f(x), y) \right].$$

We generally do not know this population expectation. Validation estimates it from a finite sample. The fitted model and all learned transformations must be evaluated under the information restrictions of the intended task, rather than on rows that contributed to their fitting or selection.

Let $I_\text{train}$, $I_\text{val}$, and $I_\text{test}$ be disjoint sets of row indices. The corresponding data subsets are denoted by $\mathcal D_\text{train}$, $\mathcal D_\text{val}$, and $\mathcal D_\text{test}$. Fit $\hat f$ on the training subset and let $I_\text{test}$ be nonempty. Its empirical test risk is the average loss on the held-out indices:

$$\hat{\mathcal{R}}_\text{test}(\hat f) = \frac{1}{|I_\text{test}|} \sum_{i \in I_\text{test}} L(\hat f(x_i), y_i).$$

Two details in that expression carry all the weight:

• We divided by $|I_\text{test}|$, not by $n$: the estimator is an average over the specific rows in the test set, not over the whole data.
• The model is fixed before those rows are scored; nothing about them influenced the fit.

Sufficient conditions for this additive mean loss to be conditionally unbiased for $\mathcal R(\hat f)$, given the fitted model, are:

• each $(x_i, y_i)$ with $i \in I_\text{test}$ is drawn from the same distribution $\mathcal{P}$ as the future data, and
• each test observation is statistically independent of the fitted model $\hat f$.

Under these conditions, conditioning on the fitted model leaves each test loss with expectation $\mathcal R(\hat f)$, so their mean has that expectation too. Dependence among test rows can still affect its variance. Correlated training and test observations can make the model and test rows statistically dependent even without direct test-data reuse. Distribution shift and adaptive use of test information can invalidate this argument. The result concerns the mean loss; applying a nonlinear operation such as a square root need not preserve unbiasedness.

In practice we divide $\mathcal{D}$ into three disjoint sets with sizes $n_\text{train}$, $n_\text{val}$, and $n_\text{test}$ such that

$$n_\text{train} + n_\text{val} + n_\text{test} = n.$$

• Training fits model parameters under a given hyperparameter setting.
• Validation compares candidate hyperparameters and chooses the best.
• Test assesses the chosen, frozen procedure on data that did not guide that choice.

The third role matters because validation outcomes guide model selection. Selecting the candidate with the best noisy validation result and reporting that same result can introduce optimism. Separate test data assess the choice without participating in it; another option is an outer resampling scheme that includes the entire selection procedure inside each outer training split.

A practical protocol is to inspect the final test result after choices are fixed. Recomputing the same frozen result does not itself change its statistical role. Changing the model, features, threshold, or evaluation protocol in response to that result uses it for validation; the revised procedure then needs fresh uninvolved evaluation data.

Allocations such as $60/20/20$ or $70/15/15$ are examples, not prescriptions. The needed sizes depend on the learning problem, event counts, group structure, and required precision. Cross-validation rotates validation folds within the development data after the final test portion has been reserved.

Preprocessing is part of the fitted procedure. Let $\phi$ be a scaler, imputer, encoder, or feature selector; fit its parameters on training data and fit the predictor on the transformed training rows. If the transform is instead learned from all collected rows, the resulting composite predictor is

$$\hat{f}_\text{leaked}(x) = \hat{f}_\theta\big(\hat{\phi}_\mathcal{D}(x)\big).$$

The transform $\hat\phi_\mathcal D$ has used held-out information. A scaler or unsupervised imputer uses held-out feature values; a supervised selector or target encoder can also use held-out labels. This violates the standard inductive evaluation boundary even when the fitted predictor itself only receives training rows.

For example, a target encoder replaces category $c$ by its mean target. If $c_i$ is the category of row $i$ and $c$ occurs in the collected data, the full-data mean is

$$\text{TE}_\mathcal{D}(c) = \frac{1}{|\{i : c_i = c\}|} \sum_{i : c_i = c} y_i,$$

This mean includes the labels of held-out rows in category $c$. It therefore cannot be used to encode those rows in an inductive test. A training-only mean is

$$\text{TE}_\text{train}(c) = \frac{1}{|\{i \in I_\text{train} : c_i = c\}|} \sum_{i \in I_\text{train} : c_i = c} y_i,$$

Apply this fitted mapping to held-out categories. If a category has no training observations, the denominator is zero, so an explicit fallback is needed; the implementation uses the overall training-target mean. Training rows encoded using their own labels can still overfit even when the outer held-out labels are excluded. Inner cross-fitting can construct training encodings from other training folds; scikit-learn documents this behavior for TargetEncoder.fit_transform, whereas fit followed by transform does not invoke that cross-fitting scheme.

Target leakage occurs when a feature supplies the outcome, or information unavailable when that prediction would have been made. A discharge field recorded after the prediction origin or a refund recorded only after the predicted return are examples when those events have not yet occurred. A historical purchase count can be valid. Audit the computation and availability of each feature at the prediction origin; neither a grouped split nor a modest score proves that this audit has passed.

For an i.i.d. prediction task, a random row split can be appropriate. Exchangeability means the joint distribution is unchanged by permuting row order; it does not mean the rows are independent. Repeated entities and time order require checking whether the available information and dependence match the intended task, rather than treating every row as a fresh independent case.

Stratified splits aim to preserve class proportions across folds, up to integer and allocation constraints. If $p_c(F_k)$ is the fraction of class $c$ among indices in fold $F_k$, the aim is $p_c(F_k)\approx p_c(\mathcal D)$. This reduces variation in class counts, but does not create independence or preserve all sources of uncertainty. A rare class with fewer observations than folds cannot appear in every fold. When groups or time impose other constraints, exact class balance may be infeasible or inappropriate.

Grouped splits hold out whole entities for a new-entity task. To see why this can matter, consider $y_i=\theta_{g_i}+\varepsilon_i$ with mutually independent, zero-mean residuals that are independent of the random group effects, all with finite second moments. Two distinct rows in the same group then have covariance $\operatorname{Var}(\theta_{g_i})$, which is positive when that effect varies. A learner that recognizes the group may use its training history. For an unseen-group test, impose

$$\mathcal{G}_\text{train} \cap \mathcal{G}_\text{test} = \emptyset, \qquad \mathcal{G}_\text{train} = \{g_i : i \in I_\text{train}\}.$$

Every group is assigned wholly to one side. This removes direct entity overlap for the chosen grouping level. Five hundred rows from five customers still provide only five sampled customer histories; their statistical information also depends on within-customer loss dependence and group sizes, so it is not generally equivalent to exactly five independent rows. Group disjointness does not ensure that customers are otherwise independent or representative of future customers.

Temporal splits are appropriate when the task predicts later observations from information available earlier. For a simple chronological hold-out, let $t_i$ be the event time and choose cutoff $t_\text{split}$:

$$t_i \leq t_\text{split}, \quad \forall\, i \in I_\text{train}, \qquad t_j > t_\text{split}, \quad \forall\, j \in I_\text{test}.$$

Training precedes the held-out event period in this construction, but event times alone do not establish label or feature availability. Delayed labels and overlapping outcome windows may require additional exclusions. A known future calendar value can be available at prediction time; an unobserved future outcome cannot. Future observations from existing patients call for chronology with eligible patient history. Future observations from previously unseen patients require both temporal and group hold-outs.

For the following cross-validation notation, use only development data after the final test set has been reserved, and let $n$ now denote the development sample size. Choose an integer $K$ with $2 \le K \le n$ and partition the development indices into $K$ nonempty disjoint folds $F_1,\ldots,F_K$. Fit $\hat f^{(-k)}$ on all development rows outside $F_k$, including fitting every learned transform there. The unweighted mean of the fold mean losses is

$$\hat{\mathcal{R}}^\text{CV}_K = \frac{1}{K} \sum_{k=1}^K \frac{1}{|F_k|} \sum_{i \in F_k} L\big(\hat{f}^{(-k)}(x_i), y_i\big),$$

Here $\hat f^{(-k)}$ is the complete pipeline fitted without fold $k$. For a complete K-fold partition, each development observation appears in exactly one held-out fold. Its out-of-fold prediction is

$$\hat{y}^\text{oof}_i = \hat{f}^{(-k(i))}(x_i), \qquad i = 1, \dots, n,$$

The function $k(i)$ identifies the fold holding out row $i$. An OOF prediction excludes that row from fitting its prediction pipeline, provided preprocessing and any selection are also restricted correctly. OOF values support error analysis, stacking, calibration, and threshold selection. If they guide a choice, they serve as validation data; the reused selected score is not an independent final test.

GroupKFold forms a complete partition of the groups, so every row can receive one group-held-out prediction. TimeSeriesSplit instead trains on earlier indices and tests on later blocks. With its default unbounded training window, successive training sets expand. Its initial training block receives no held-out prediction, and max_train_size can limit the training history. A manual temporal prediction array should leave unscored rows explicit; cross_val_predict requires every input row to appear in a test fold exactly once and therefore does not accept ordinary TimeSeriesSplit output.

Record the data snapshot, stable row identities, split configuration or exact fold assignments, and the fitted procedure. For a randomized splitter or estimator, also record the relevant randomness settings and dependency versions. A fixed seed does not protect against a changed row order or changed data. Save and apply indices to the same ordered dataset, or map them through stable row identifiers.

A few properties of the estimators above matter for how we interpret them.

CV fits use fewer development observations than a final fit on all development data. For equal row folds, each fit uses $(1-1/K)n$ rows; grouped folds can differ in size. The CV result therefore describes those training sizes and held-out tasks. It can be pessimistic when the learning procedure improves with additional data, but its bias and variance do not have a universal direction or monotonic relationship with $K$.

The variation of a held-out mean depends on the individual loss variances and their covariances. Under independent losses with a common finite variance $\sigma_L^2$, its variance is $\sigma_L^2/m$ for $m$ scored rows. The general variance formula includes pairwise covariances; dependent losses can still be uncorrelated. CV fits share data, so treating the $K$ fold scores as independent replicates and dividing their standard deviation by $\sqrt K$ does not give a general confidence interval.

For a chosen pair of evaluations, define a signed observed loss difference:

$$\Delta_\text{leak} \;=\; \hat{\mathcal{R}}^\text{honest} - \hat{\mathcal{R}}^\text{leaked}.$$

A positive difference means the evaluation labeled leaky reports a smaller loss in this pair; a negative value means it reports a larger loss. Its sign and magnitude are not guaranteed. If the two evaluations also change customers, training sizes, or fitted models, the gap does not isolate the causal effect of one information leak. The examples below diagnose the information boundary directly.

Loss units matter. Squared-error risk has squared-target units, whereas RMSE has target units. A gap between two RMSEs is therefore a separate descriptive quantity, not the same number as the squared-loss difference above. Neither gap is generally equal to a customer-effect variance.