Validation Strategies and Data Leakage

A model can predict another observation from a familiar customer much more accurately than an observation from a customer it has never seen. A random row split can mix those two tasks when each customer contributes several records. The resulting score may look useful while answering a different question from the one the model will face at deployment.

Validation starts by defining what the model will know when it makes a prediction. We then construct held-out data that reproduce that situation as closely as the available sample permits. New-customer predictions require holding out whole customers; future predictions require respecting time and label availability. Learned preprocessing belongs inside the training boundary of each evaluation.

Building on Statistical Modelling, Standardization, and Normalization, we examine the evaluation procedure around a fitted model. A 1-nearest-neighbor (1-NN) regressor keeps the examples simple: it predicts with the target of the closest training observation. Its behavior also shows why the numerical effect of a poor split depends on the data and learner.

The chapter develops five parts of an evaluation procedure:

• Training fits parameters, validation guides choices, and final testing assesses the frozen choice.
• Preprocessing is part of the model and must be fitted inside the same training fold as the estimator.
• The split should match the task: preserve useful class proportions, hold out new entities, and respect prediction-time information.
• Out-of-fold predictions provide held-out predictions for covered rows; retain the fold assignments to record which observations each fitted pipeline excluded.
• Reproducibility requires a specified data snapshot, procedure, and fold assignment; randomized procedures also need recorded randomness settings.

A deployment-relevant split makes model comparisons more useful. Holding out whole customers asks whether the fitted procedure transfers to new customers, while a chronological split asks how it behaves on later observations under the chosen information boundary. These designs address particular sources of dependence and information reuse; they do not by themselves guarantee independent groups, a representative sample, or unchanged deployment conditions.

Comparing a convenient row split with a task-relevant split can reveal a problem during development. A large discrepancy warrants checking entity overlap, available features, and time order. A small discrepancy is not proof of a sound evaluation: a learner may ignore the unavailable information, or other differences between the partitions may offset its numerical effect.

Designing the split also clarifies the feature contract. A spending summary based entirely on outcomes known before prediction can be useful history; a summary that includes the held-out outcome cannot be used for that prediction. Removing unavailable information makes feature computation consistent with that availability boundary, although its measured score may become worse.

Holding out complete entities can leave few independent units for fitting and evaluation. For example, five equal group folds over 100 customers train on 80 customers at a time. Cross-validation then assesses fits trained on fewer customers than a final fit using all 100. If additional training data improve that learning procedure, this can make CV pessimistic for the final fit; neither improvement nor the bias direction is universal.

Grouped splits require identifiers that represent the entity boundary of the task. A customer should retain the same identifier across relevant tables and visits. Inconsistent identifiers or joins can hide overlap, while assigning one identifier to different customers can create false overlap. Temporal evaluation also needs event times, feature availability times, and, when relevant, the time at which each target becomes known.

With few or heterogeneous groups, results can depend strongly on which groups are held out. Increasing the fold count changes the training sizes and evaluation partitions, but does not create new customers. Report the limited evidence, compare models on common partitions, and use uncertainty methods whose sampling assumptions fit the groups and time structure. Resampling a handful of observed groups cannot supply diversity that was never collected.

The formulas separate the population quantity we want from the sample quantity we can compute. We first define risk and held-out loss, then describe which observations may contribute to fitting. The grouped and temporal constraints express different deployment tasks; cross-validation repeats the corresponding fitting and evaluation procedure.

Let the collected observations be

Here $x_i \in \mathbb{R}^d$ is the feature vector, $y_i \in \mathbb{R}$ is the regression target, and $g_i$ identifies the customer or other group. The examples use the group column to construct splits, rather than as an input to 1-NN. We use squared loss, $L(\hat y,y)=(\hat y-y)^2$; classification uses a loss suited to its target.

Let $\mathcal P$ be the joint distribution of a fresh deployment observation $(x,y)$, and let $f$ be a fixed predictor. Assume the loss $L(f(x),y)$ has a finite expectation. Generalization risk is its expected loss:

We generally do not know this population expectation. Validation estimates it from a finite sample. The fitted model and all learned transformations must be evaluated under the information restrictions of the intended task, rather than on rows that contributed to their fitting or selection.

Let $I_\text{train}$, $I_\text{val}$, and $I_\text{test}$ be disjoint sets of row indices. The corresponding data subsets are denoted by $\mathcal D_\text{train}$, $\mathcal D_\text{val}$, and $\mathcal D_\text{test}$. Fit $\hat f$ on the training subset and let $I_\text{test}$ be nonempty. Its empirical test risk is the average loss on the held-out indices:

Two details in that expression carry all the weight:

• We divided by $|I_\text{test}|$, not by $n$: the estimator is an average over the specific rows in the test set, not over the whole data.
• The model is fixed before those rows are scored; nothing about them influenced the fit.

Sufficient conditions for this additive mean loss to be conditionally unbiased for $\mathcal R(\hat f)$, given the fitted model, are:

• each $(x_i, y_i)$ with $i \in I_\text{test}$ is drawn from the same distribution $\mathcal{P}$ as the future data, and
• each test observation is statistically independent of the fitted model $\hat f$.

Under these conditions, conditioning on the fitted model leaves each test loss with expectation $\mathcal R(\hat f)$, so their mean has that expectation too. Dependence among test rows can still affect its variance. Correlated training and test observations can make the model and test rows statistically dependent even without direct test-data reuse. Distribution shift and adaptive use of test information can invalidate this argument. The result concerns the mean loss; applying a nonlinear operation such as a square root need not preserve unbiasedness.

In practice we divide $\mathcal{D}$ into three disjoint sets with sizes $n_\text{train}$, $n_\text{val}$, and $n_\text{test}$ such that

• Training fits model parameters under a given hyperparameter setting.
• Validation compares candidate hyperparameters and chooses the best.
• Test assesses the chosen, frozen procedure on data that did not guide that choice.

The third role matters because validation outcomes guide model selection. Selecting the candidate with the best noisy validation result and reporting that same result can introduce optimism. Separate test data assess the choice without participating in it; another option is an outer resampling scheme that includes the entire selection procedure inside each outer training split.

A practical protocol is to inspect the final test result after choices are fixed. Recomputing the same frozen result does not itself change its statistical role. Changing the model, features, threshold, or evaluation protocol in response to that result uses it for validation; the revised procedure then needs fresh uninvolved evaluation data.

Allocations such as $60/20/20$ or $70/15/15$ are examples, not prescriptions. The needed sizes depend on the learning problem, event counts, group structure, and required precision. Cross-validation rotates validation folds within the development data after the final test portion has been reserved.

Preprocessing is part of the fitted procedure. Let $\phi$ be a scaler, imputer, encoder, or feature selector; fit its parameters on training data and fit the predictor on the transformed training rows. If the transform is instead learned from all collected rows, the resulting composite predictor is

The transform $\hat\phi_\mathcal D$ has used held-out information. A scaler or unsupervised imputer uses held-out feature values; a supervised selector or target encoder can also use held-out labels. This violates the standard inductive evaluation boundary even when the fitted predictor itself only receives training rows.

For example, a target encoder replaces category $c$ by its mean target. If $c_i$ is the category of row $i$ and $c$ occurs in the collected data, the full-data mean is

This mean includes the labels of held-out rows in category $c$. It therefore cannot be used to encode those rows in an inductive test. A training-only mean is

Apply this fitted mapping to held-out categories. If a category has no training observations, the denominator is zero, so an explicit fallback is needed; the implementation uses the overall training-target mean. Training rows encoded using their own labels can still overfit even when the outer held-out labels are excluded. Inner cross-fitting can construct training encodings from other training folds; scikit-learn documents this behavior for TargetEncoder.fit_transform, whereas fit followed by transform does not invoke that cross-fitting scheme.

Target leakage occurs when a feature supplies the outcome, or information unavailable when that prediction would have been made. A discharge field recorded after the prediction origin or a refund recorded only after the predicted return are examples when those events have not yet occurred. A historical purchase count can be valid. Audit the computation and availability of each feature at the prediction origin; neither a grouped split nor a modest score proves that this audit has passed.

For an i.i.d. prediction task, a random row split can be appropriate. Exchangeability means the joint distribution is unchanged by permuting row order; it does not mean the rows are independent. Repeated entities and time order require checking whether the available information and dependence match the intended task, rather than treating every row as a fresh independent case.

Stratified splits aim to preserve class proportions across folds, up to integer and allocation constraints. If $p_c(F_k)$ is the fraction of class $c$ among indices in fold $F_k$, the aim is $p_c(F_k)\approx p_c(\mathcal D)$. This reduces variation in class counts, but does not create independence or preserve all sources of uncertainty. A rare class with fewer observations than folds cannot appear in every fold. When groups or time impose other constraints, exact class balance may be infeasible or inappropriate.

Grouped splits hold out whole entities for a new-entity task. To see why this can matter, consider $y_i=\theta_{g_i}+\varepsilon_i$ with mutually independent, zero-mean residuals that are independent of the random group effects, all with finite second moments. Two distinct rows in the same group then have covariance $\operatorname{Var}(\theta_{g_i})$, which is positive when that effect varies. A learner that recognizes the group may use its training history. For an unseen-group test, impose

Every group is assigned wholly to one side. This removes direct entity overlap for the chosen grouping level. Five hundred rows from five customers still provide only five sampled customer histories; their statistical information also depends on within-customer loss dependence and group sizes, so it is not generally equivalent to exactly five independent rows. Group disjointness does not ensure that customers are otherwise independent or representative of future customers.

Temporal splits are appropriate when the task predicts later observations from information available earlier. For a simple chronological hold-out, let $t_i$ be the event time and choose cutoff $t_\text{split}$:

Training precedes the held-out event period in this construction, but event times alone do not establish label or feature availability. Delayed labels and overlapping outcome windows may require additional exclusions. A known future calendar value can be available at prediction time; an unobserved future outcome cannot. Future observations from existing patients call for chronology with eligible patient history. Future observations from previously unseen patients require both temporal and group hold-outs.

For the following cross-validation notation, use only development data after the final test set has been reserved, and let $n$ now denote the development sample size. Choose an integer $K$ with $2 \le K \le n$ and partition the development indices into $K$ nonempty disjoint folds $F_1,\ldots,F_K$. Fit $\hat f^{(-k)}$ on all development rows outside $F_k$, including fitting every learned transform there. The unweighted mean of the fold mean losses is

Here $\hat f^{(-k)}$ is the complete pipeline fitted without fold $k$. For a complete K-fold partition, each development observation appears in exactly one held-out fold. Its out-of-fold prediction is

The function $k(i)$ identifies the fold holding out row $i$. An OOF prediction excludes that row from fitting its prediction pipeline, provided preprocessing and any selection are also restricted correctly. OOF values support error analysis, stacking, calibration, and threshold selection. If they guide a choice, they serve as validation data; the reused selected score is not an independent final test.

GroupKFold forms a complete partition of the groups, so every row can receive one group-held-out prediction. TimeSeriesSplit instead trains on earlier indices and tests on later blocks. With its default unbounded training window, successive training sets expand. Its initial training block receives no held-out prediction, and max_train_size can limit the training history. A manual temporal prediction array should leave unscored rows explicit; cross_val_predict requires every input row to appear in a test fold exactly once and therefore does not accept ordinary TimeSeriesSplit output.

Record the data snapshot, stable row identities, split configuration or exact fold assignments, and the fitted procedure. For a randomized splitter or estimator, also record the relevant randomness settings and dependency versions. A fixed seed does not protect against a changed row order or changed data. Save and apply indices to the same ordered dataset, or map them through stable row identifiers.

A few properties of the estimators above matter for how we interpret them.

CV fits use fewer development observations than a final fit on all development data. For equal row folds, each fit uses $(1-1/K)n$ rows; grouped folds can differ in size. The CV result therefore describes those training sizes and held-out tasks. It can be pessimistic when the learning procedure improves with additional data, but its bias and variance do not have a universal direction or monotonic relationship with $K$.

The variation of a held-out mean depends on the individual loss variances and their covariances. Under independent losses with a common finite variance $\sigma_L^2$, its variance is $\sigma_L^2/m$ for $m$ scored rows. The general variance formula includes pairwise covariances; dependent losses can still be uncorrelated. CV fits share data, so treating the $K$ fold scores as independent replicates and dividing their standard deviation by $\sqrt K$ does not give a general confidence interval.

For a chosen pair of evaluations, define a signed observed loss difference:

A positive difference means the evaluation labeled leaky reports a smaller loss in this pair; a negative value means it reports a larger loss. Its sign and magnitude are not guaranteed. If the two evaluations also change customers, training sizes, or fitted models, the gap does not isolate the causal effect of one information leak. The examples below diagnose the information boundary directly.

Loss units matter. Squared-error risk has squared-target units, whereas RMSE has target units. A gap between two RMSEs is therefore a separate descriptive quantity, not the same number as the squared-loss difference above. Neither gap is generally equal to a customer-effect variance.

We first inspect row and group allocations, then examine a simulation where the numerical effect of customer overlap is small and nonmonotone. Together they separate a structural check from a score-based diagnostic.

The allocation diagrams contain six customers with four observations each. The row allocation alternates training and held-out observations within each customer; it is a deterministic illustration of overlap, rather than a random draw. The group allocation holds out customers E and F together.

The computed curve is not monotone: RMSE ranges from about 3.16 to 3.73 in this run and finishes above its zero-overlap value. Here the scalar feature is itself a noisy measurement of the customer baseline, so nearby observations from other customers can also predict the target. In addition, the function changes the selected customers and the number of test rows as overlap changes. This is a counterexample to treating overlap as a universal numerical penalty: the curve does not isolate a causal effect, and a small score gap does not establish that an evaluation matches deployment. For an unseen-customer task, group disjointness remains a design requirement regardless of this gap.

We now replay the same logic on a dataset small enough that every number can be verified by hand, which separates the mechanism from any library behavior. We use four customers, each contributing two rows. This is a smaller version of the setup in the first figure.

The four customers and their observations are:

• Customer A: $(x, y) = (10, 5)$ and $(11, 6)$
• Customer B: $(x, y) = (50, 25)$ and $(51, 26)$
• Customer C: $(x, y) = (20, 10)$ and $(21, 11)$
• Customer D: $(x, y) = (80, 40)$ and $(81, 41)$

For each customer, the second observation is one unit higher in feature space and one unit higher in the target, so a model that assigns each row its same-customer sibling will predict it almost exactly.

We use a 1-NN regressor on a single scalar feature $x$. The prediction for a test observation is the target of the training observation with the smallest $|x_\text{train} - x_\text{test}|$. For our data this rule is easy to evaluate by hand.

The leaking row split assigns the first row of every customer to training and the second to test. This deterministic allocation guarantees a training sibling for every test row. It illustrates overlap that a random row split can produce; a random split can also place both of a customer's rows in the test set.

Training set: $\{(10, 5), (50, 25), (20, 10), (80, 40)\}$, one row per customer.

Test set: $\{(11, 6), (51, 26), (21, 11), (81, 41)\}$, the sibling row of each customer.

For each test row, we find the nearest training $x$ and read off its $y$.

• Test row $(11, 6)$: distances $|11-10| = 1$, $|11-50| = 39$, $|11-20| = 9$, $|11-80| = 69$. Nearest is $x=10$ with $y=5$. Prediction $= 5$. Squared error $= (5-6)^2 = 1$.
• Test row $(51, 26)$: distances $|51-10| = 41$, $|51-50| = 1$, $|51-20| = 31$, $|51-80| = 29$. Nearest is $x=50$ with $y=25$. Prediction $= 25$. Squared error $= (25-26)^2 = 1$.
• Test row $(21, 11)$: distances $|21-10| = 11$, $|21-50| = 29$, $|21-20| = 1$, $|21-80| = 59$. Nearest is $x=20$ with $y=10$. Prediction $= 10$. Squared error $= (10-11)^2 = 1$.
• Test row $(81, 41)$: distances $|81-10| = 71$, $|81-50| = 31$, $|81-20| = 61$, $|81-80| = 1$. Nearest is $x=80$ with $y=40$. Prediction $= 40$. Squared error $= (40-41)^2 = 1$.

Mean squared error $= \frac{1}{4}(1 + 1 + 1 + 1) = 1$. RMSE $= \sqrt{1} = 1.000$.

Now we assign whole customers to one side, enforcing the group-disjointness condition. We place customers A and B in training and customers C and D in test. All four test rows belong to unseen customers.

Training set: $\{(10, 5), (11, 6), (50, 25), (51, 26)\}$.

Test set: $\{(20, 10), (21, 11), (80, 40), (81, 41)\}$.

• Test row $(20, 10)$: distances $|20-10| = 10$, $|20-11| = 9$, $|20-50| = 30$, $|20-51| = 31$. Nearest is $x=11$ with $y=6$. Prediction $= 6$. Squared error $= (6-10)^2 = 16$.
• Test row $(21, 11)$: distances $|21-10| = 11$, $|21-11| = 10$, $|21-50| = 29$, $|21-51| = 30$. Nearest is $x=11$ with $y=6$. Prediction $= 6$. Squared error $= (6-11)^2 = 25$.
• Test row $(80, 40)$: distances $|80-10| = 70$, $|80-11| = 69$, $|80-50| = 30$, $|80-51| = 29$. Nearest is $x=51$ with $y=26$. Prediction $= 26$. Squared error $= (26-40)^2 = 196$.
• Test row $(81, 41)$: distances $|81-10| = 71$, $|81-11| = 70$, $|81-50| = 31$, $|81-51| = 30$. Nearest is $x=51$ with $y=26$. Prediction $= 26$. Squared error $= (26-41)^2 = 225$.

Mean squared error $= \frac{1}{4}(16 + 25 + 196 + 225) = \frac{462}{4} = 115.5$. RMSE $= \sqrt{115.5} \approx 10.747$.

The row-split RMSE is 1.000 and the group-split RMSE is about 10.747, a difference of about 9.747 target units. The row split tests siblings of training observations; the group split tests different customers on a different set of target values. This illustrates how task and partition choices can change the error. It does not estimate a universal leakage penalty or imply that every learner would make the same errors.

In this tiny row allocation, every nearest training observation is the held-out row's same-customer sibling. In the grouped allocation, C and D have no training siblings, and the stated 1-NN rule selects rows from A or B. These are properties of this constructed dataset, not guarantees for all repeated-customer data.

We next use six synthetic customers with 60 observations each. This deliberately small example makes the limited group count visible and remains inexpensive to execute. The hand-written 1-NN rule above supplies the primitive prediction mechanism; scikit-learn supplies the splitters and the fold-fitted preprocessing below.

We will cover five things in sequence:

• generate data with repeated customers and a customer-level target component,
• compute cross-validated RMSE with a leaking row split and with a group split,
• compute an honest held-out RMSE on unseen customers as a reference,
• inspect preprocessing information flow even when the observed score gap is zero or has the opposite sign,
• demonstrate the API for stratified and time-series splitters.

The earlier scores are development demonstrations. The grouped hold-out is an illustrative reference, not an untouched external-test claim. For the following OOF calculation, use only its training-side customers and keep its held-out indices outside this computation. Three group folds over those four development customers give every development row one held-out prediction.

We implement standardization and 1-NN from primitive array operations, then compare the resulting predictions with a scikit-learn pipeline on exactly the same folds. The primitive predictor requires finite two-dimensional feature arrays and a one-dimensional target; it rejects inconsistent shapes and an empty training set, and raises an error if distance arithmetic overflows. It chooses the first training row on an exact distance tie. That tie rule is explicit because reference implementations can choose differently when equally distant neighbors have different targets.

The predictions are stored at the original development-row indices, rather than concatenated in fold order. Each fold fits its own mean and population standard deviation; a constant training feature uses scale 1. The numerical assertion checks agreement with the reference on this dataset, whose nearest-neighbor comparisons have no exact ties. It does not assert that all tie choices or near-constant-feature conventions match across implementations.

Pooling and averaging fold scores answer different aggregation questions. Let $n_k$ be the number of held-out rows and $r_k$ the RMSE in fold $k$. Squaring $r_k$ recovers its mean squared error, so pooled RMSE for those same predictions is

This weights individual squared errors equally, then takes one square root. It generally differs from the arithmetic mean of $r_k$, even with equal fold sizes. Pooled $R^2$ also differs in general from mean fold $R^2$, because its baseline uses the pooled target mean. State the metric, weighting, and aggregation used. OOF predictions used to select a procedure still require fresh uninvolved evaluation of that selected procedure.

KFold(n_splits=5, shuffle=True, random_state=42) partitions row indices into five folds. Shuffling can be useful for an i.i.d. task when storage order is incidental. If order represents prediction-time availability, choose a temporal design instead. With shuffle=True, the recorded random_state controls the random fold assignment for the same input order.

GroupKFold(n_splits=5) keeps each customer wholly within one held-out fold. At least five distinct customers are needed. With default shuffle=False, the implementation attempts to balance sample counts; group sizes can prevent exact balance. shuffle=True, available in the installed scikit-learn 1.8 environment, randomizes groups and attempts to balance group counts instead. StratifiedGroupKFold additionally attempts to preserve class proportions, subject to group constraints; it is not a general cure for every group-size imbalance.

GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42) creates one group-disjoint hold-out. Its fractional test_size refers to groups, rounded up, rather than directly to rows. Here it holds out two of the six customers. A development hold-out used for choices has a validation role, while a final test must remain uninvolved in those choices.

StratifiedKFold(n_splits=4, shuffle=True, random_state=42) creates four folds with approximate class-proportion balance. In the current example we stratified on a binarized version of $y$. For actual classification, pass the class labels directly.

TimeSeriesSplit(n_splits=3) creates three later test blocks from ordered input indices. The default unbounded training window expands; max_train_size can cap it. gap excludes a specified number of rows from the end of each training window. Choose exclusions from label windows and availability times; a gap does not automatically establish independence or equal a forecast horizon measured in calendar time.

.split(X, y=None, groups=None) yields (train_indices, test_indices) pairs. For grouped splitters, groups is required and must be an array of group identifiers of length $n$. For StratifiedKFold, pass the class label as y.

sklearn.pipeline.make_pipeline(scaler, model) constructs a composite estimator whose .fit(X_train, y_train) fits each step on the training fold only, and whose .predict(X) applies the same transformations to new data. When fitted within each training fold, this keeps the pipeline's learned transforms inside that fold. Features constructed outside the pipeline still need an availability audit.

.get_n_splits(X=None, y=None, groups=None) returns the number of splits. An OOF array instead has one position per input row; a coverage array can check which rows received predictions.

Choose the evaluation task and protocol before inspecting candidate scores. A new-customer task calls for group hold-outs, a future-observation task calls for prediction-origin restrictions, and an i.i.d. classification comparison may benefit from stratification. Future predictions for previously unseen patients need both group and temporal restrictions. Future predictions for existing patients can use eligible past patient history; they do not automatically require patient-disjoint folds.

Reserve test data early and retain their identities. Fit and select using development data, then evaluate the frozen procedure on the intended held-out task. If test results guide further choices, regard those data as validation and use fresh uninvolved data for the revised procedure. This separation does not remove possible population shift or sampling uncertainty.

The following practices connect the split to feature construction, reporting, and repeated model comparisons.

Record the task, splitter, and data identities as part of the experimental design. Reuse common partitions for paired comparisons of candidate models, and inspect paired score differences rather than comparing only marginal score spreads. Keep the final test out of those decisions. Classification Metrics and Decision Thresholds applies the same separation to threshold choices.

Wrap every transformation in a pipeline so that the transformer is fitted inside the same training fold as the estimator. make_pipeline(StandardScaler(), KNeighborsRegressor()) fits the scaler on the training rows of each fold and applies the learned statistics to the held-out rows at predict time, which removes the leak by construction. Where a pipeline is impractical, fit the transformer with .fit(X_train) and apply it with .transform(X_test) by hand, using the same recorded row assignment for both calls. What the pipeline enforces is fit-predict consistency with respect to the split, not any property of the estimator itself.

Choose the fold count with the available group count, class counts, training sizes, and computation budget in mind. A randomized row splitter needs recorded randomness settings; a chronological splitter needs ordered, availability-consistent input. Choose any temporal exclusions from the task's label and feature windows. Five or ten folds are examples, not a reason to override those constraints.

Use stable identifiers at the level that is new at deployment. A session identifier cannot detect two sessions from the same customer; a customer identifier cannot detect shared households or sites when those are the units that must be held out. Temporal ordering should describe when observations and labels become available to fitting and prediction, rather than relying only on record insertion order.

For each feature, document how its value can be computed at the prediction origin. Past target summaries can be eligible history; a future calendar value can be known in advance. A held-out outcome or an unavailable future event cannot be used. Fit learned scalers, imputers, selectors, and encoders within the training boundary. A pipeline helps enforce this boundary for its steps, but cannot repair a target-derived or unavailable feature that was already built outside it.

Missingness and outliers may differ across groups or periods. Fit imputation statistics on training rows and inspect how performance varies on the held-out task. A fold aligned with a real deployment missingness pattern may be informative rather than erroneous. Neither such alignment nor held-out preprocessing implies a universal numerical direction or size of score bias.

A surprisingly good score, a large row-versus-group difference, or a disagreement with a manual hold-out warrants investigation. None supplies a universal numerical threshold for declaring leakage. A finite-sample RMSE can fall below a population noise standard deviation by chance; score differences can also reflect task difficulty, sampling, or distribution shift. Inspect the information flow and reproduce the feature construction.

Repeatedly changing the procedure after consulting a held-out score uses that score for validation. Unstable identifiers, incorrect joins, or transforms fitted before splitting can introduce additional information reuse. Address those mechanisms directly, then obtain an evaluation that did not inform the resulting revision.

With few groups, report the partition and its limitations. A large held-out group receives more weight in a pooled row loss, whereas an unweighted mean of fold scores assigns equal weight to folds. An equal-customer mean loss is another estimand. Choose and state the weighting that matches the task instead of interpreting every difference as a model improvement.

CV computation depends on the number of fits, the observations in each fit, and the learning algorithm. Grouped and row K-fold with similar training sizes are not inherently ordered in cost. Temporal training windows can grow over successive splits. Time representative fits for the actual procedure before choosing a resampling budget.

Save the split assignment to reproduce the evaluation. Index storage grows with the number of stored indices and repeats, so it can matter for very large datasets. Sequentially fitting and scoring folds can avoid retaining all fitted estimators at once; an OOF array still stores one prediction per scored row.

Distinguish the cost of repeating the splitter from the cost of fitting models and constructing features. For many procedures the fits dominate, but this is an empirical property of the workload. Parallelizing folds can reduce elapsed time while increasing memory use, so keep the CPU-sized examples here sequential.

The independent customer count and within-customer dependence both affect the precision of a new-customer evaluation. In the current six-customer example, the grouped fold RMSE standard deviation is about 13.92, compared with about 0.24 for row CV. These are descriptive spreads over different folds and different evaluation questions; neither is automatically a standard error or a confidence interval. Use common partitions for paired model comparisons and inspect the paired score differences. A group bootstrap can be useful under appropriate sampling assumptions, but six observed groups give little information about the population of future customers.

Compare deployment errors with a development estimate for the same prediction task and metric, while accounting for sampling and label delays. A gap can reflect population shift, changed feature availability, implementation errors, or uncertainty. It does not uniquely identify a change in independence or prove that leakage occurred.

Collecting additional independent customers can improve evidence about new-customer performance under suitable sampling. Additional observations from existing customers can improve within-customer estimation, but do not supply previously unobserved customer types. Report both the group count and the number of observations when discussing precision.

Validation specifies which information the fitted procedure may use and which deployment task the held-out score assesses. A random row split can be appropriate for an i.i.d. task, while an unseen-customer task needs whole-customer hold-outs and future prediction needs prediction-origin restrictions. Repeated entities alone do not make every evaluation invalid.

Expected risk and empirical mean loss coincide in expectation under the stated distribution and fit-independence conditions; nonlinear scores need separate care. Training fits parameters, validation guides choices, and final testing evaluates the frozen choice. Learned preprocessing belongs inside each training fold, and target or future information must be available at the prediction origin. Group disjointness and pipelines address specific boundaries, rather than proving the entire evaluation valid.

We worked through a four-customer, eight-row example with row-split RMSE 1.00 and group-split RMSE about 10.75. The six-customer simulation produces mean random-fold RMSE about 2.30, grouped-fold RMSE about 9.98, and grouped hold-out RMSE about 23.06. The spread between the latter two is a warning about the limited group count and the particular partitions, rather than evidence that two estimates have converged. The overlap and target-encoding experiments also show that a known information leak need not produce a fixed or even favorable score change. Split design and prediction-time feature availability must be checked directly.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about validation strategies and data leakage.