Hyperparameter Search and Nested Cross-Validation

A search over 32 configurations returns one winner and a tempting number: its best validation score. That number helped choose the winner, so reporting it as though the configuration had been fixed in advance ignores selection. We need evaluation data that the search has not inspected.

A fitted model contains parameters learned from data, such as regression coefficients or neural-network weights. Many important choices are made before that fitting begins: the degree of a polynomial, the strength of a regularization penalty, the number of neighbors in a nearest-neighbor model, or the learning rate of an optimizer. These choices are hyperparameters. They define the model family, the fitting procedure, or both, and they are not updated by the inner optimization that fits the parameters.

A validation score is a noisy function of the hyperparameters, so choosing the best score is itself a learning procedure. If we use the same validation folds both to select hyperparameters and to report performance, the winner's score is usually too optimistic. The search has inspected many candidates and picked the one that happened to look best on those particular folds. Nested cross-validation separates the two jobs: an inner loop selects hyperparameters, and an outer loop estimates how well the entire selection-and-fit procedure generalizes.

In this chapter we compare grid search and random search under equal evaluation budgets, explain why selection bias appears, and build nested cross-validation from primitive loops. We then apply the ideas to a polynomial ridge regression model, assess it with outer folds, and compare equal-budget searches across repeated seeds. The chapter builds on the previous discussion of validation strategies and data leakage and on the classification metrics from the preceding chapter, where we saw that a score is only meaningful when we know exactly which data produced it.