Hyperparameter Search and Nested Cross-Validation › Implementation in Scikit-learn
Machine Learning from ScratchAll chapters Hyperparameter Search and Nested Cross-Validation › Implementation in Scikit-learnSection view · Sections in this chapter6 / 10 ↓
Reading tools Definitions, links & notes
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
With the bookkeeping clear, we can implement the complete procedure on a real synthetic dataset. We generate 180 observations from a cubic polynomial with Gaussian noise, then fit a polynomial ridge regression model. The pipeline is
Standardization is inside the pipeline, so it is fit only on the training portion of each split. This is not a cosmetic detail: if we standardized the full dataset before splitting, the validation folds would influence the training preprocessing through the mean and scale. Keeping the scaler inside the pipeline makes the entire preprocessing step part of the model that is refit within every fold.
The hyperparameters are the polynomial degree and the ridge penalty . The grid therefore has candidates. We use 5 outer folds, 4 inner folds, and 10 repeated outer splits. The functions below implement the search from primitive loops so that every selection step is visible: candidate generation, inner cross-validation, selection, outer refitting, and outer scoring.
We use a SeedSequence hierarchy to give each repetition its own child sequence, then separate outer splits, inner splits, primitive candidate draws, and library candidate draws within that repetition. NumPy designs these child streams to be independent with very high probability; they are reproducible pseudorandom streams, not a proof of physical independence. We rebuild the same hierarchy when comparing methods, so grid and random search share outer and inner folds. Candidate samplers have separate streams. We pass derived 32-bit integer seeds to the scikit-learn splitters and random search. The finite seed protocol is checked for collisions; distinct hierarchy paths alone would not guarantee distinct 32-bit integers.
PythonIn [11]Expand
def make_polynomial_data(n=180, noise=0.35, seed=7):
rng = np.random.default_rng(seed)
x = rng.uniform(-2.2, 2.2, size=n)
y = 0.7 * x**3 - 1.1 * x**2 + 0.6 * x + 0.8
y = y + rng.normal(0.0, noise, size=n)
return x.reshape(-1, 1), y
DEGREES = list(range(1, 9))
ALPHAS = [1e-3, 1e-2, 1e-1, 1.0]
BUDGET = len(DEGREES) * len(ALPHAS)
def grid_candidates():
return [{"degree": d, "alpha": a} for d in DEGREES for a in ALPHAS]
def random_candidates(rng, budget=BUDGET):
candidates = []
for _ in range(budget):
degree = int(rng.integers(1, 9))
alpha = float(np.exp(rng.uniform(np.log(1e-3), np.log(1.0))))
candidates.append({"degree": degree, "alpha": alpha})
return candidates
def make_model(candidate):
return make_pipeline(
PolynomialFeatures(degree=candidate["degree"], include_bias=False),
StandardScaler(),
Ridge(alpha=candidate["alpha"]),
)
def inner_cv_mse(candidate, X_train, y_train, inner_splits):
fold_scores = []
for tr, va in inner_splits:
model = make_model(candidate)
model.fit(X_train[tr], y_train[tr])
pred = model.predict(X_train[va])
fold_scores.append(mean_squared_error(y_train[va], pred))
return float(np.mean(fold_scores))
def best_so_far(scores):
return np.minimum.accumulate(np.asarray(scores))
def splitter_seed(sequence):
return int(sequence.generate_state(1, dtype=np.uint32)[0])
def nested_streams(seed):
# Rebuild, rather than mutate, a supplied repetition's spawn counter.
if isinstance(seed, np.random.SeedSequence):
root = np.random.SeedSequence(
seed.entropy, spawn_key=seed.spawn_key, pool_size=seed.pool_size
)
else:
root = np.random.SeedSequence(seed)
outer, inner, candidates, library = root.spawn(4)
return {
"outer": splitter_seed(outer),
"inner": [splitter_seed(s) for s in inner.spawn(5)],
"candidates": candidates.spawn(5),
"library": [splitter_seed(s) for s in library.spawn(5)],
}
def run_nested_cv(method, X, y, seed, budget=BUDGET):
if method not in {"grid", "random"}:
raise ValueError("method must be 'grid' or 'random'")
if not isinstance(budget, (int, np.integer)) or budget < 1:
raise ValueError("budget must be a positive integer")
if method == "grid" and budget != BUDGET:
raise ValueError("the defined grid contains exactly BUDGET candidates")
streams = nested_streams(seed)
outer = KFold(n_splits=5, shuffle=True, random_state=streams["outer"])
outer_scores = []
inner_scores = []
chosen = []
for fold, (tr, te) in enumerate(outer.split(X)):
if method == "grid":
candidates = grid_candidates()
else:
rng = np.random.default_rng(streams["candidates"][fold])
candidates = random_candidates(rng, budget)
inner = KFold(
n_splits=4, shuffle=True, random_state=streams["inner"][fold]
)
inner_splits = list(inner.split(X[tr]))
scores = [
inner_cv_mse(c, X[tr], y[tr], inner_splits) for c in candidates
]
best_idx = int(np.argmin(scores))
best = candidates[best_idx]
model = make_model(best)
model.fit(X[tr], y[tr])
pred = model.predict(X[te])
outer_scores.append(mean_squared_error(y[te], pred))
inner_scores.append(scores[best_idx])
chosen.append(best)
return np.array(outer_scores), np.array(inner_scores), chosenPythonIn [12]Expand
X, y = make_polynomial_data(seed=7)
experiment_root = np.random.SeedSequence(1000)
repetition_seeds = experiment_root.spawn(10)
final_seed, trace_seed, jitter_seed = experiment_root.spawn(3)
# Check the finite integer interfaces, including the demonstration hierarchy.
integer_seeds = [splitter_seed(final_seed)]
for sequence in repetition_seeds + [trace_seed]:
streams = nested_streams(sequence)
integer_seeds.extend(
[streams["outer"], *streams["inner"], *streams["library"]]
)
assert len(integer_seeds) == len(set(integer_seeds))
rows = []
fold_rows = []
for rep, seed in enumerate(repetition_seeds):
for method in ["grid", "random"]:
outer_scores, inner_scores, chosen = run_nested_cv(method, X, y, seed)
for fold, (outer_loss, inner_loss, candidate) in enumerate(
zip(outer_scores, inner_scores, chosen), start=1
):
fold_rows.append(
{
"method": method,
"rep": rep,
"outer_fold": fold,
"degree": candidate["degree"],
"alpha": candidate["alpha"],
"inner_mse": float(inner_loss),
"outer_mse": float(outer_loss),
}
)
rows.append(
{
"method": method,
"rep": rep,
"outer_mse": float(np.mean(outer_scores)),
"inner_mse": float(np.mean(inner_scores)),
"inner_outer_gap": float(
np.mean(outer_scores) - np.mean(inner_scores)
),
"outer_sd": float(np.std(outer_scores, ddof=1)),
}
)
nested_results = pd.DataFrame(rows)
fold_results = pd.DataFrame(fold_rows)PythonIn [13]Expand
summary = (
nested_results.groupby("method")
.agg(
outer_mean=("outer_mse", "mean"),
outer_split_sd=("outer_mse", lambda s: s.std(ddof=1)),
outer_mcse=("outer_mse", lambda s: s.std(ddof=1) / np.sqrt(len(s))),
inner_mean=("inner_mse", "mean"),
gap_mean=("inner_outer_gap", "mean"),
outer_sd_mean=("outer_sd", "mean"),
)
.reset_index()
)ConsoleOut [14]Expand
Nested CV summary across repeated outer splits method outer_mean outer_split_sd outer_mcse inner_mean gap_mean outer_sd_mean grid 0.1129 0.0023 0.0007 0.1127 0.0003 0.0296 random 0.1130 0.0026 0.0008 0.1127 0.0004 0.0296
The first repetition's fold-level table shows the configurations selected independently inside each outer training set. Different selections need not imply a large predictive difference: nearby candidates can have nearly equal validation losses.
ConsoleOut [15]Expand
method rep outer_fold degree alpha inner_mse outer_mse grid 0 1 3 0.100000 0.108641 0.125123 grid 0 2 3 0.001000 0.105756 0.147204 grid 0 3 3 0.001000 0.115480 0.100014 grid 0 4 3 0.100000 0.119183 0.083471 grid 0 5 3 0.001000 0.113433 0.090897 random 0 1 3 0.067673 0.108666 0.125193 random 0 2 3 0.001858 0.105757 0.147204 random 0 3 3 0.001584 0.115481 0.100015 random 0 4 3 0.206168 0.119137 0.083856 random 0 5 3 0.004695 0.113434 0.090889
Because both methods use the same outer partitions within each repetition, we also inspect their paired MSE differences. Positive differences below mean random search had higher outer MSE on that repetition.
PythonIn [16]Expand
paired_outer = nested_results.pivot(
index="rep", columns="method", values="outer_mse"
)
paired_differences = paired_outer["random"] - paired_outer["grid"]
paired_mean = float(paired_differences.mean())
paired_sd = float(paired_differences.std(ddof=1))ConsoleOut [17]Expand
mean paired random-minus-grid MSE: 0.00010156 SD of paired differences: 0.00039750
These paired summaries describe partition and search randomization on one fixed dataset. A small difference relative to the marginal spread of either method is not, by itself, a significance test; pairing can make differences much less variable than individual scores.
The summary table separates three quantities. outer_split_sd describes repetition-to-repetition variation on this fixed dataset; outer_mcse is that spread divided by the square root of the repetition count, describing Monte Carlo precision of the mean under the independent-stream randomization model, conditional on this dataset. Neither is uncertainty over new datasets. inner_mean averages selected inner scores, gap_mean is outer minus inner, and outer_sd_mean summarizes the within-run fold spread. Here the grid outer mean is 0.1129 and its selected inner mean is 0.1127; the corresponding random-search means are 0.1130 and 0.1127. The realized outer-minus-inner gaps are 0.00029 for grid search and 0.00037 for random search. These executed results do not contradict expected selection optimism under its stated assumptions; inner fits use fewer rows, and realized validation differences also contain sample variation.
Scikit-learn provides GridSearchCV and RandomizedSearchCV, which automate the inner search. We can use them inside the same outer loop. The grid search should match our primitive implementation closely, because both use the same folds, candidates, and scoring rule; any residual difference comes from tie-breaking order or numerical round-off in the fitted coefficients. A truly exact match requires the same regularization convention and identical preprocessing. The random search uses a different sampler, so we compare budgets and distributions rather than expecting identical candidate draws.
PythonIn [18]Expand
param_grid = {
"polynomialfeatures__degree": DEGREES,
"ridge__alpha": ALPHAS,
}
param_distributions = {
"polynomialfeatures__degree": DEGREES,
"ridge__alpha": loguniform(1e-3, 1.0),
}
def sklearn_nested_once(method, X, y, seed):
streams = nested_streams(seed)
outer = KFold(n_splits=5, shuffle=True, random_state=streams["outer"])
outer_scores = []
chosen = []
for fold, (tr, te) in enumerate(outer.split(X)):
inner = KFold(
n_splits=4, shuffle=True, random_state=streams["inner"][fold]
)
base = make_pipeline(
PolynomialFeatures(include_bias=False),
StandardScaler(),
Ridge(),
)
if method == "grid":
search = GridSearchCV(
base,
param_grid,
cv=inner,
scoring="neg_mean_squared_error",
n_jobs=1,
)
else:
search = RandomizedSearchCV(
base,
param_distributions,
n_iter=BUDGET,
cv=inner,
scoring="neg_mean_squared_error",
n_jobs=1,
random_state=streams["library"][fold],
)
search.fit(X[tr], y[tr])
pred = search.predict(X[te])
outer_scores.append(mean_squared_error(y[te], pred))
chosen.append(search.best_params_)
return np.array(outer_scores), chosenPythonIn [19]Expand
seed_compare = repetition_seeds[0]
sk_grid_scores, sk_grid_chosen = sklearn_nested_once("grid", X, y, seed_compare)
sk_rand_scores, sk_rand_chosen = sklearn_nested_once(
"random", X, y, seed_compare
)
our_grid_scores, our_grid_inner, our_grid_chosen = run_nested_cv(
"grid", X, y, seed_compare
)
our_rand_scores, our_rand_inner, our_rand_chosen = run_nested_cv(
"random", X, y, seed_compare
)
assert np.allclose(our_grid_scores, sk_grid_scores)
assert int(np.argmin([1.0, 1.0, 2.0])) == 0PythonIn [20]Expand
comparison = pd.DataFrame(
{
"implementation": [
"ours-grid",
"sklearn-grid",
"ours-random",
"sklearn-random",
],
"mean_outer_mse": [
our_grid_scores.mean(),
sk_grid_scores.mean(),
our_rand_scores.mean(),
sk_rand_scores.mean(),
],
}
)ConsoleOut [21]Expand
implementation mean_outer_mse
ours-grid 0.1093
sklearn-grid 0.1093
ours-random 0.1094
sklearn-random 0.1095
Grid outer scores match: TrueThe comparison serves two purposes. First, it checks that the primitive grid loop and GridSearchCV agree when they receive the same folds and candidate set. Second, it places the random-search results beside the grid results without pretending that the candidate draws are identical. Matching grid scores are a correctness check on the implementation, not evidence that grid search is generally better. The random-search comparison is about the behavior of a stochastic procedure under a fixed budget.
After assessment, we run the chosen grid-search procedure once more on all development observations. refit=True fits the selected configuration on all 180 rows after its inner search. We retain that estimator for future predictions; its training fit is not a new held-out performance estimate.
PythonIn [22]Expand
final_search = GridSearchCV(
make_model({"degree": 1, "alpha": 1e-3}),
param_grid,
cv=KFold(n_splits=4, shuffle=True, random_state=splitter_seed(final_seed)),
scoring="neg_mean_squared_error",
n_jobs=1,
refit=True,
)
final_search.fit(X, y)
final_model = final_search.best_estimator_ConsoleOut [23]Expand
Final selected configuration: {'polynomialfeatures__degree': 4, 'ridge__alpha': 0.001}
Final refit observations: 180The outer assessment used 144 training observations per fold, whereas this final refit uses 180. The nested score evaluates the pre-specified procedure at the smaller training size, not the conditional risk of this exact final model. We chose the grid procedure in advance for this demonstration; choosing between procedures after inspecting outer results would itself need independent assessment.
Key Parameters
Below are the main configuration choices that control the nested search shown above:
degree: Polynomial degree in the feature map, searched over1through8in the primitive loops. Higher degrees expand the feature family; variance also depends on regularization and data. Specify the degree range before outer evaluation and choose within it using inner folds. If outer results lead you to redesign the search, assess that redesigned procedure with new independent evaluation data.alpha: Ridge penalty strength, searched over1e-3through1. Larger values increase shrinkage; a log-spaced range covers several orders of magnitude efficiently.param_grid: Defines the grid search space. Keys use the pipeline step name, a double underscore, and the parameter name, such aspolynomialfeatures__degreeandridge__alpha. If the pipeline step names change, the search keys must change with them.param_distributions: Defines the random search space. Continuous distributions such asloguniform(1e-3, 1.0)are sampled on a logarithmic scale.n_iter: Sets the random-search budget. To compare grid and random search fairly, it should equal the number of grid candidates, with both searches using the same inner folds and scoring rule.cv: Controls the inner validation splitter. The explicitly seededKFoldobjects used here give reproducible partitions for unchanged row ordering.scoring: Set to"neg_mean_squared_error"because scikit-learn maximizes scores by convention; negative MSE makes a larger value correspond to a smaller error.random_state: Controls the random candidate draws inRandomizedSearchCV.n_jobs: Controls fit parallelism. We use1to keep this CPU example modest;-1requests all available processors. Reproducibility also depends on the estimator, splitters, and random seeds.refit: With the single-metric searches here,Truerefits the selected configuration on the full input passed tofit, after inner cross-validation.best_estimator_holds that fitted estimator andpredictuses it;best_params_,best_score_, andcv_results_expose the configuration, selected inner score, and candidate results.
Comments
Add to the conversation
A question, a correction, or an example from your own work.
No comments yet. Be the first to share your thoughts!