Hyperparameter Search and Nested Cross-Validation › Disadvantages

The computational cost of search is multiplicative. If we evaluate $B$ candidates with $K_{\mathrm{inner}}$ inner folds across $J$ outer folds, we fit roughly $J \times B \times K_{\mathrm{inner}}$ models before any final refit. Grid search grows exponentially with the number of tuned dimensions, and random search can waste draws in uninformative regions. Grid search finds the smallest observed validation loss on its finite grid, but that need not identify the lowest true risk over the wider search space. Random search may miss that configuration entirely. Neither method removes validation noise.

Nested cross-validation offers a way to assess selection when no untouched test set is available. If we already have a clean test set that was never used for development, a single outer holdout is often enough and is cheaper. Nested cross-validation is most useful when we need to use the same data for both selection and assessment, or when we want to compare complete model-selection procedures rather than one fixed model. Ordinary random-fold evaluation assumes independent observations representative of the target population. For time series, grouped data, or clustered observations, ordinary random folds can leak information and produce misleadingly optimistic estimates.

Search can overfit the inner validation folds, especially when the budget is large relative to the amount of validation data. A large search space also invites researcher degrees of freedom: if we keep expanding the space after seeing results, the reported score is no longer a clean estimate of a pre-specified procedure. Finally, preprocessing must be inside the cross-validation loop. Scaling, imputation, and data-driven feature selection learn from data and must be fitted within each training fold. A fixed polynomial feature map does not estimate population quantities, but its degree is selected inside the search, and the full pipeline keeps that selection and learned scaling within the correct splits.