Hyperparameter Search and Nested Cross-Validation › Summary

Hyperparameter search is part of the modeling procedure. Grid search enumerates a finite Cartesian product and is easy to audit, while random search samples candidates from a distribution and often covers important dimensions more efficiently under a fixed budget. For a fixed candidate set with unbiased validation estimates, selecting the minimum produces optimistic validation loss in expectation. Actual inner-outer gaps also reflect training-size differences and sample variation. Nested cross-validation separates selection from assessment: inner folds choose hyperparameters, and outer folds estimate the risk of the entire selection-and-fit procedure.

We built the procedure from primitive loops, compared it with scikit-learn's search objects, and used repeated outer splits to report variability. The practical recipe is to pre-specify the search space and budget, keep preprocessing inside the pipeline, report the outer estimate with its spread, and refit the selected configuration on all development data. When an untouched test set exists, it can answer the final evaluation question directly; nested cross-validation is the tool for the situations where it does not.