Hyperparameter Search and Nested Cross-Validation › Practical Implications

Grid search is a reasonable default when the search space is small, mostly discrete, and needs to be audited by reviewers; exhaustive enumeration makes the candidate set easy to explain and reproduce. Random search is often preferable when the space contains continuous hyperparameters, when several dimensions are expected to be irrelevant, or when a fixed budget must cover a broad range. In either case, the search space, budget, metric, and inner-fold rule should be fixed before results are inspected, and every candidate should be scored on the same inner folds. When the budget is large relative to the amount of validation data, the inner search can fit fold-specific noise; reducing unnecessary candidates or using more validation data can reduce this problem. Changing the search method alone does not supply independent evaluation.

Nested cross-validation is warranted when no untouched test set is available and the quantity of interest is the performance of the complete selection-and-fit procedure, rather than the performance of one fixed configuration. It is unnecessary when a clean holdout can answer the evaluation question directly, and it is not a substitute for leakage control. Scaling, imputation, feature selection, target encoding, and other data-dependent transformations must be fitted inside each training fold. For grouped, clustered, or time-dependent observations, the outer and inner splitters should respect that structure; ordinary random folds can share information across splits and make the nested estimate optimistic.

The useful diagnostics are the inner-outer gap, the variability of outer scores, and the stability of the selected hyperparameters across outer folds. A large gap can arise from selection over many noisy candidates, from different fitting sample sizes, or from finite-sample variation; it should be interpreted as a warning about the selection procedure, not as a precise decomposition of bias. Compare methods using their paired fold or repetition outcomes when the protocol pairs them. Marginal outer-score standard deviations alone do not determine the uncertainty of a difference, and repeated partitions of one dataset do not establish reproducibility over new datasets. For expensive models, reduce the candidate budget, parallelize folds, cache preprocessing, or use a cheaper surrogate during exploration, but preserve the same protocol for the final comparison. Reporting should include the search space, budget, inner and outer fold counts, metric, seeds, outer mean with a measure of spread, and the configurations selected across outer folds; the best inner score should not be presented as final generalization performance.