Hyperparameter Search and Nested Cross-Validation › Visualizing Hyperparameter Search and Nested Cross-Validation
Machine Learning from ScratchAll chapters Hyperparameter Search and Nested Cross-Validation › Visualizing Hyperparameter Search and Nested Cross-ValidationSection view · Sections in this chapter7 / 10 ↓
Reading tools Definitions, links & notes
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
For each candidate in evaluation order, we compute the inner cross-validation MSE and then take the running minimum. That running minimum is the best score the search would report after that many evaluations. Each trace changes only when a newly evaluated candidate improves the best score so far. Grid search uses a fixed Cartesian order. Random search draws candidates from the same degree and log-alpha space, so its curve can drop earlier or later depending on the draw. The figure is not a claim that one method always wins; it shows how budget and candidate order affect the internal score.
PythonIn [24]Expand
demo_streams = nested_streams(trace_seed)
outer_demo = KFold(n_splits=5, shuffle=True, random_state=demo_streams["outer"])
tr_demo, te_demo = next(iter(outer_demo.split(X)))
inner_demo = KFold(
n_splits=4, shuffle=True, random_state=demo_streams["inner"][0]
)
inner_splits_demo = list(inner_demo.split(X[tr_demo]))
grid_scores = [
inner_cv_mse(c, X[tr_demo], y[tr_demo], inner_splits_demo)
for c in grid_candidates()
]
rng_demo = np.random.default_rng(demo_streams["candidates"][0])
rand_scores = [
inner_cv_mse(c, X[tr_demo], y[tr_demo], inner_splits_demo)
for c in random_candidates(rng_demo)
]
trace_grid = best_so_far(grid_scores)
trace_random = best_so_far(rand_scores)
budget_axis = np.arange(1, BUDGET + 1)VisualizationOut [25]Expand
The second figure summarizes the repeated nested cross-validation runs. Each point is the mean outer-fold MSE from one repetition. The horizontal line marks the mean across repetitions for that search method. The vertical spread comes from different outer splits and, for random search, from different candidate draws. The two point clouds overlap in this realized example. The important comparison is not which point is lowest in one repetition, but whether the distributions differ systematically and how large the split-to-split variability is.
PythonIn [26]Expand
rng_jitter = np.random.default_rng(jitter_seed)
distribution_data = {}
for method in ["grid", "random"]:
values = nested_results.loc[
nested_results["method"] == method, "outer_mse"
].to_numpy()
distribution_data[method] = {
"values": values,
"jitter": rng_jitter.uniform(-0.08, 0.08, size=len(values)),
"mean": float(values.mean()),
}VisualizationOut [27]Expand
On this generated dataset, the grid search has a positive outer-minus-inner gap and random search has a positive gap. The gap combines selection, different training sizes, and random variation; it cannot identify selection bias by itself. The two outer means differ by much less than the observed marginal split-to-split spread; the paired summaries above describe the within-repetition differences. This is a descriptive comparison on one dataset, with paired outer partitions, and does not establish a general ranking of search methods. The outer scores evaluate the entire pre-specified procedure on data excluded from each fold's search.
Comments
Add to the conversation
A question, a correction, or an example from your own work.
No comments yet. Be the first to share your thoughts!