Validation Strategies and Data Leakage › Implementation in scikit-learn
Machine Learning from ScratchAll chapters Validation Strategies and Data Leakage › Implementation in scikit-learnSection view · Sections in this chapter7 / 10 ↓
Reading tools Definitions, links & notes
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
We next use six synthetic customers with 60 observations each. This deliberately small example makes the limited group count visible and remains inexpensive to execute. The hand-written 1-NN rule above supplies the primitive prediction mechanism; scikit-learn supplies the splitters and the fold-fitted preprocessing below.
We will cover five things in sequence:
- generate data with repeated customers and a customer-level target component,
- compute cross-validated RMSE with a leaking row split and with a group split,
- compute an honest held-out RMSE on unseen customers as a reference,
- inspect preprocessing information flow even when the observed score gap is zero or has the opposite sign,
- demonstrate the API for stratified and time-series splitters.
PythonIn [11]Expand
rng_imp = np.random.default_rng(1234)
n_customers = 6
rows_per_cust = 60
cust_baseline = rng_imp.normal(50, 20, n_customers)
cust_ids = np.repeat(np.arange(n_customers), rows_per_cust)
X = (
np.repeat(cust_baseline, rows_per_cust)
+ rng_imp.normal(0, 2.0, n_customers * rows_per_cust)
).reshape(-1, 1)
y = np.repeat(cust_baseline, rows_per_cust) + rng_imp.normal(
0, 1.5, n_customers * rows_per_cust
)ConsoleOut [12]Expand
Observations: 360 Customers: 6 Baseline y range: [17.9, 108.3] Mean y: 60.57, std y: 26.85
Each customer has a latent baseline. The scalar feature is that baseline plus Gaussian noise with standard deviation 2.0, and the target is the same baseline plus independent noise with standard deviation 1.5. A nearby sibling may therefore be informative, but rows from other customers with similar features can also be useful. The seed fixes this particular synthetic draw; it does not make it representative of every deployment.
PythonIn [13]Expand
def cv_rmse(splitter, X, y, groups=None):
scores = []
for tr, te in splitter.split(X, y, groups):
pipeline = make_pipeline(
StandardScaler(), KNeighborsRegressor(n_neighbors=1)
)
pipeline.fit(X[tr], y[tr])
preds = pipeline.predict(X[te])
scores.append(np.sqrt(np.mean((preds - y[te]) ** 2)))
return np.array(scores)
random_kf = cv_rmse(KFold(n_splits=5, shuffle=True, random_state=42), X, y)
group_kf = cv_rmse(GroupKFold(n_splits=5), X, y, groups=cust_ids)
gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
tr_gss, te_gss = next(gss.split(X, y, groups=cust_ids))
pipeline = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=1))
pipeline.fit(X[tr_gss], y[tr_gss])
holdout_rmse = np.sqrt(np.mean((pipeline.predict(X[te_gss]) - y[te_gss]) ** 2))ConsoleOut [14]Expand
Random K-fold RMSE: 2.297 ± 0.244 (fold SD) Group K-fold RMSE: 9.979 ± 13.923 (fold SD) Group held-out RMSE: 23.060
For these six customers, the executed mean fold RMSE is about 2.30 for random K-fold and 9.98 for GroupKFold; the separate grouped hold-out has RMSE about 23.06. They use the same predictor but ask different questions about existing versus unseen customers. With only six groups and five folds, the grouped test folds contain either one or two customers, so both the fold scores and the hold-out are highly sensitive to which customers are held out. Their disagreement is a reason to report the actual partition and limited evidence, rather than to claim a precise deployment estimate. The printed ± values are descriptive population standard deviations of the five fold scores, not standard errors or confidence intervals.
The fold-level counts make entity overlap visible independently of the score. Every random fold in this draw shares all six customers with its training rows, whereas every grouped fold shares none. Overlap does not by itself establish that 1-NN selects a sibling; the nearest-neighbor distances determine that.
VisualizationOut [16]Expand
PythonIn [17]Expand
score_labels = ["Random row CV", "Grouped CV", "Grouped hold-out"]
score_values = [random_kf.mean(), group_kf.mean(), holdout_rmse]The bar chart shows these scores side by side.
VisualizationOut [18]Expand
The prediction scatters show the row and group hold-outs separately. The row split places many predictions near the identity line; the grouped hold-out has larger errors for its selected unseen customers. These figures compare particular samples, so their appearance does not establish a universal ordering of model performance.
VisualizationsOut [20]Expand
Now we turn to preprocessing while holding the group folds fixed. Fitting the scaler on all rows violates the training-only preprocessing boundary. In this one-dimensional 1-NN example, however, applying a common positive affine transformation to training and test distances preserves their nearest-neighbor ordering, apart from numerical tie effects. We should therefore expect the two scores to agree here, rather than expect a guaranteed score benefit from leakage.
PythonIn [21]Expand
def cv_rmse_scaling_leak(X, y, groups):
clean_scores, leaky_scores = [], []
for tr, te in GroupKFold(n_splits=5).split(X, y, groups):
scaler_clean = StandardScaler().fit(X[tr])
model = KNeighborsRegressor(n_neighbors=1)
model.fit(scaler_clean.transform(X[tr]), y[tr])
clean_scores.append(
np.sqrt(
np.mean(
(model.predict(scaler_clean.transform(X[te])) - y[te]) ** 2
)
)
)
scaler_leaky = StandardScaler().fit(X)
model_leaky = KNeighborsRegressor(n_neighbors=1)
model_leaky.fit(scaler_leaky.transform(X[tr]), y[tr])
leaky_scores.append(
np.sqrt(
np.mean(
(model_leaky.predict(scaler_leaky.transform(X[te])) - y[te])
** 2
)
)
)
return np.array(clean_scores), np.array(leaky_scores)
clean_scale, leaky_scale = cv_rmse_scaling_leak(X, y, cust_ids)ConsoleOut [22]Expand
Scaler fit per fold: 9.9789 Scaler fit on all data: 9.9789 Difference: 0.0000
The two scores are equal in the executed comparison. That equality follows from this predictor and transformation; it does not make globally fitted preprocessing a valid general evaluation procedure. Leakage must be identified by information flow, because its presence does not guarantee a visible improvement in every score.
PythonIn [23]Expand
rng_te = np.random.default_rng(7)
n_te = 800
cats_te = rng_te.integers(0, 30, n_te)
cat_effect = rng_te.normal(0, 4, 30)
y_te_target = cat_effect[cats_te] + rng_te.normal(0, 0.5, n_te)
tr_idx = rng_te.choice(n_te, size=int(0.8 * n_te), replace=False)
te_idx = np.setdiff1d(np.arange(n_te), tr_idx)
def encode_leaky(cats, y):
mapping = pd.Series(y).groupby(cats).mean()
return pd.Series(cats).map(mapping).values
def encode_clean(cats_tr, y_tr, cats_te):
mapping = pd.Series(y_tr).groupby(cats_tr).mean()
return pd.Series(cats_te).map(mapping).fillna(y_tr.mean()).values
feat_all = encode_leaky(cats_te, y_te_target).reshape(-1, 1)
model_leaky = KNeighborsRegressor(n_neighbors=1)
model_leaky.fit(feat_all[tr_idx], y_te_target[tr_idx])
rmse_target_leak = np.sqrt(
np.mean((model_leaky.predict(feat_all[te_idx]) - y_te_target[te_idx]) ** 2)
)
feat_train_clean = encode_clean(
cats_te[tr_idx], y_te_target[tr_idx], cats_te[tr_idx]
).reshape(-1, 1)
feat_test_clean = encode_clean(
cats_te[tr_idx], y_te_target[tr_idx], cats_te[te_idx]
).reshape(-1, 1)
model_clean = KNeighborsRegressor(n_neighbors=1)
model_clean.fit(feat_train_clean, y_te_target[tr_idx])
rmse_target_clean = np.sqrt(
np.mean((model_clean.predict(feat_test_clean) - y_te_target[te_idx]) ** 2)
)ConsoleOut [24]Expand
Target encoder fit on all data: 0.6937 Target encoder fit on train: 0.6766
The full-data target encoder uses held-out labels in its category means and is therefore leaky even though its executed RMSE (about 0.694) is slightly worse than the training-only version (about 0.677) in this draw. Both exceed the simulated noise standard deviation of 0.5. With 1-NN, repeated category encodings create ties, and predictions can depend on which training observation is selected among tied neighbors. This experiment demonstrates the information-flow violation, not a guaranteed direction or magnitude of score inflation. A finite-sample RMSE below a noise standard deviation would also not, by itself, prove leakage.
The category-level encodings show where the leak enters. A full-data mean includes the target values of rows that later appear in the test split, so it can shift away from the training-only estimate.
VisualizationOut [26]Expand
Finally, we demonstrate the splitter API for stratified and temporal data. The output shows how each splitter partitions the row indices.
ConsoleOut [28]Expand
StratifiedKFold fold sizes and class balance (using a coarse binary label): Fold 1: n_test= 90, balanced positive rate=0.500 Fold 2: n_test= 90, balanced positive rate=0.500 Fold 3: n_test= 90, balanced positive rate=0.500 Fold 4: n_test= 90, balanced positive rate=0.500 TimeSeriesSplit train and test ranges (illustrative ordered indices): Fold 1: train rows 0 to 8, test rows 9 to 15 Fold 2: train rows 0 to 15, test rows 16 to 22 Fold 3: train rows 0 to 22, test rows 23 to 29
The four stratified folds have positive fraction 0.500 in this constructed label example. This row stratification does not protect customer disjointness and is shown only to explain the API. The temporal example uses 30 illustrative, equally spaced time indices; it does not claim that the synthetic customer records form a real chronological dataset. Its default training windows expand before each held-out block.
The printed ranges are easier to interpret as windows. The figure places each fold's training and test intervals on the row index axis, so the past-then-future constraint is visible at a glance.
VisualizationOut [30]Expand
Computing and checking out-of-fold predictions
The earlier scores are development demonstrations. The grouped hold-out is an illustrative reference, not an untouched external-test claim. For the following OOF calculation, use only its training-side customers and keep its held-out indices outside this computation. Three group folds over those four development customers give every development row one held-out prediction.
We implement standardization and 1-NN from primitive array operations, then compare the resulting predictions with a scikit-learn pipeline on exactly the same folds. The primitive predictor requires finite two-dimensional feature arrays and a one-dimensional target; it rejects inconsistent shapes and an empty training set, and raises an error if distance arithmetic overflows. It chooses the first training row on an exact distance tie. That tie rule is explicit because reference implementations can choose differently when equally distant neighbors have different targets.
PythonIn [31]Expand
def nearest_predict(X_train, y_train, X_query):
X_train = np.asarray(X_train, dtype=float)
y_train = np.asarray(y_train, dtype=float)
X_query = np.asarray(X_query, dtype=float)
if X_train.ndim != 2 or X_query.ndim != 2 or y_train.ndim != 1:
raise ValueError("Expected two feature matrices and one target vector")
if len(X_train) == 0 or X_train.shape[1] == 0:
raise ValueError("Training requires observations and features")
if len(y_train) != len(X_train) or X_query.shape[1] != X_train.shape[1]:
raise ValueError("Inconsistent training or query dimensions")
if not all(np.isfinite(a).all() for a in (X_train, y_train, X_query)):
raise ValueError("Inputs must be finite")
if len(X_query) == 0:
return np.empty(0, dtype=float)
with np.errstate(over="raise", invalid="raise"):
differences = X_query[:, None, :] - X_train[None, :, :]
squared_distances = np.sum(differences**2, axis=2)
return y_train[np.argmin(squared_distances, axis=1)]
X_development = X[tr_gss].copy()
y_development = y[tr_gss].copy()
groups_development = cust_ids[tr_gss].copy()
development_folds = list(
GroupKFold(n_splits=3).split(
X_development, y_development, groups_development
)
)
manual_oof = np.full(len(y_development), np.nan)
heldout_counts = np.zeros(len(y_development), dtype=int)
oof_fold_rmse = []
oof_fold_sizes = []
for train_index, heldout_index in development_folds:
assert len(np.intersect1d(train_index, heldout_index)) == 0
assert set(groups_development[train_index]).isdisjoint(
groups_development[heldout_index]
)
# Learn these statistics from this fold's training rows only.
fold_mean = X_development[train_index].mean(axis=0)
fold_sd = X_development[train_index].std(axis=0, ddof=0)
fold_scale = np.where(fold_sd > 0, fold_sd, 1.0)
train_scaled = (X_development[train_index] - fold_mean) / fold_scale
heldout_scaled = (X_development[heldout_index] - fold_mean) / fold_scale
prediction = nearest_predict(
train_scaled, y_development[train_index], heldout_scaled
)
manual_oof[heldout_index] = prediction
heldout_counts[heldout_index] += 1
oof_fold_sizes.append(len(heldout_index))
oof_fold_rmse.append(
np.sqrt(np.mean((prediction - y_development[heldout_index]) ** 2))
)
assert np.all(heldout_counts == 1)
assert np.isfinite(manual_oof).all()
from sklearn.model_selection import cross_val_predict
reference_oof = cross_val_predict(
make_pipeline(
StandardScaler(),
KNeighborsRegressor(n_neighbors=1, algorithm="brute"),
),
X_development,
y_development,
cv=development_folds,
)
assert np.allclose(manual_oof, reference_oof, rtol=0, atol=1e-12)
pooled_oof_rmse = np.sqrt(np.mean((manual_oof - y_development) ** 2))
reference_oof_rmse = np.sqrt(np.mean((reference_oof - y_development) ** 2))
oof_fold_rmse = np.asarray(oof_fold_rmse)
oof_fold_sizes = np.asarray(oof_fold_sizes)
weighted_fold_rmse = np.sqrt(
np.average(oof_fold_rmse**2, weights=oof_fold_sizes)
)
assert np.isclose(pooled_oof_rmse, weighted_fold_rmse)ConsoleOut [32]Expand
Development customers: 4 Development rows: 240 Every row held out once: True Reference predictions agree: True Pooled OOF RMSE: 22.300 Mean fold RMSE: 15.613
The predictions are stored at the original development-row indices, rather than concatenated in fold order. Each fold fits its own mean and population standard deviation; a constant training feature uses scale 1. The numerical assertion checks agreement with the reference on this dataset, whose nearest-neighbor comparisons have no exact ties. It does not assert that all tie choices or near-constant-feature conventions match across implementations.
Pooling and averaging fold scores answer different aggregation questions. Let be the number of held-out rows and the RMSE in fold . Squaring recovers its mean squared error, so pooled RMSE for those same predictions is
This weights individual squared errors equally, then takes one square root. It generally differs from the arithmetic mean of , even with equal fold sizes. Pooled also differs in general from mean fold , because its baseline uses the pooled target mean. State the metric, weighting, and aggregation used. OOF predictions used to select a procedure still require fresh uninvolved evaluation of that selected procedure.
Key Parameters
KFold(n_splits=5, shuffle=True, random_state=42) partitions row indices into five folds. Shuffling can be useful for an i.i.d. task when storage order is incidental. If order represents prediction-time availability, choose a temporal design instead. With shuffle=True, the recorded random_state controls the random fold assignment for the same input order.
GroupKFold(n_splits=5) keeps each customer wholly within one held-out fold. At least five distinct customers are needed. With default shuffle=False, the implementation attempts to balance sample counts; group sizes can prevent exact balance. shuffle=True, available in the installed scikit-learn 1.8 environment, randomizes groups and attempts to balance group counts instead. StratifiedGroupKFold additionally attempts to preserve class proportions, subject to group constraints; it is not a general cure for every group-size imbalance.
GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42) creates one group-disjoint hold-out. Its fractional test_size refers to groups, rounded up, rather than directly to rows. Here it holds out two of the six customers. A development hold-out used for choices has a validation role, while a final test must remain uninvolved in those choices.
StratifiedKFold(n_splits=4, shuffle=True, random_state=42) creates four folds with approximate class-proportion balance. In the current example we stratified on a binarized version of . For actual classification, pass the class labels directly.
TimeSeriesSplit(n_splits=3) creates three later test blocks from ordered input indices. The default unbounded training window expands; max_train_size can cap it. gap excludes a specified number of rows from the end of each training window. Choose exclusions from label windows and availability times; a gap does not automatically establish independence or equal a forecast horizon measured in calendar time.
Key Methods
.split(X, y=None, groups=None) yields (train_indices, test_indices) pairs. For grouped splitters, groups is required and must be an array of group identifiers of length . For StratifiedKFold, pass the class label as y.
sklearn.pipeline.make_pipeline(scaler, model) constructs a composite estimator whose .fit(X_train, y_train) fits each step on the training fold only, and whose .predict(X) applies the same transformations to new data. When fitted within each training fold, this keeps the pipeline's learned transforms inside that fold. Features constructed outside the pipeline still need an availability audit.
.get_n_splits(X=None, y=None, groups=None) returns the number of splits. An OOF array instead has one position per input row; a coverage array can check which rows received predictions.
Comments
Add to the conversation
A question, a correction, or an example from your own work.
No comments yet. Be the first to share your thoughts!