Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Hyperparameter Search and Nested Cross-Validation › Example

Michael BrenndoerferOctober 25, 20265 min read

Machine Learning from ScratchAll chapters

Part 4 of 15Model Evaluation and Selection

Chapter 33Hyperparameter Search and Nested Cross-Validation › Example

Reading section 5 of 10 · Example · 5 minSection 5 of 10 · 5 min

Section 5 of 10 · 5 min read

Hyperparameter Search and Nested Cross-Validation › ExampleSection view · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

We first import the libraries used for the arithmetic and the later search implementation. The hidden setup seeds the shared plotting environment once.

PythonIn [7]
CollapseExpandimport numpy as np
import pandas as pd
from scipy.stats import loguniform
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import GridSearchCV, KFold, RandomizedSearchCV
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler

To see the nested-cross-validation accounting without the distraction of model fitting, strip the problem down to a table of scores. Imagine five outer folds and three candidate polynomial degrees. For each outer fold, inner cross-validation gives one score per degree. The inner loop selects the degree with the smallest score. Then the outer loop measures that selected degree on the held-out outer fold. The code below records this bookkeeping, performs the selection, and computes the summary. The numbers are a small arithmetic illustration, not a claim about the synthetic dataset used later.

PythonIn [8]
CollapseExpandexample_scores = pd.DataFrame(
    {
        "outer_fold": [1, 2, 3, 4, 5],
        "inner_degree_1": [0.48, 0.51, 0.50, 0.47, 0.49],
        "inner_degree_2": [0.42, 0.36, 0.46, 0.39, 0.45],
        "inner_degree_3": [0.41, 0.39, 0.44, 0.40, 0.43],
        "outer_mse": [0.52, 0.49, 0.60, 0.46, 0.57],
    }
)

inner_cols = ["inner_degree_1", "inner_degree_2", "inner_degree_3"]
example_scores["selected_degree"] = (
    example_scores[inner_cols].to_numpy().argmin(axis=1) + 1
)
example_scores["best_inner_mse"] = (
    example_scores[inner_cols].to_numpy().min(axis=1)
)

PythonIn [9]
CollapseExpandinner_mean = example_scores["best_inner_mse"].mean()
outer_mean = example_scores["outer_mse"].mean()
example_gap = outer_mean - inner_mean
outer_sd = example_scores["outer_mse"].std(ddof=1)
outer_se = outer_sd / np.sqrt(len(example_scores))

ConsoleOut [10]
CollapseExpand outer_fold  inner_degree_1  inner_degree_2  inner_degree_3  outer_mse  selected_degree  best_inner_mse
          1           0.480           0.420           0.410      0.520                3           0.410
          2           0.510           0.360           0.390      0.490                2           0.360
          3           0.500           0.460           0.440      0.600                3           0.440
          4           0.470           0.390           0.400      0.460                2           0.390
          5           0.490           0.450           0.430      0.570                3           0.430
mean best inner MSE = 0.406
mean outer MSE      = 0.528
inner-outer gap     = 0.122
outer SD            = 0.057
outer SE            = 0.026

The first block builds one row per outer fold. Each row contains the three inner scores, the selected degree, the best inner score, and the outer score. The selection step is an argmin: in each row, we compare the three inner scores and keep the smallest. For example, if the inner scores are $0.48$ , $0.42$ , and $0.41$ , the selected degree is the third one and the best inner score is $0.41$ . That is the choice the inner loop would make.

The second step records, for each outer fold, the outer MSE of the selected configuration refitted on the outer training set and scored on the outer test fold; those values are supplied in the outer_mse column. The key distinction is summarized here:

The inner score answers: "Among these candidates, which one looks best on the inner validation folds?"
The outer score answers: "After choosing that candidate, how well does the fitted model perform on data that were not used for the choice?"

The printed table lets us compare those two answers fold by fold. For each outer fold $j$ , let $\hat R_{\mathrm{CV},j}(\lambda)$ be the inner cross-validation score of candidate $\lambda$ computed on the training data for outer fold $j$ , and let $\hat R_{\mathrm{outer},j}$ be the model's score on outer test fold $j$ . The inner mean is the average of the best inner scores, where the minimum is over the evaluated candidate set $\Lambda_B$ and the inner cross-validation is computed only on the training portion of outer fold $j$ :

\bar R_{\mathrm{inner}} = \frac{1}{J}\sum_{j=1}^{J} \min_{\lambda\in\Lambda_B} \hat R_{\mathrm{CV},j}(\lambda)

where:

$\bar R_{\mathrm{inner}}$ : the mean of the best inner cross-validation scores across outer folds
$J$ : the number of outer folds
$\Lambda_B$ : the finite set of evaluated candidate hyperparameters
$\hat R_{\mathrm{CV},j}(\lambda)$ : the inner cross-validation score for candidate $\lambda$ on the training portion of outer fold $j$

The outer mean is the average of the outer test scores:

\bar R_{\mathrm{outer}} = \frac{1}{J}\sum_{j=1}^{J} \hat R_{\mathrm{outer},j}.

where:

$\bar R_{\mathrm{outer}}$ : the mean of the outer test-fold scores across outer folds
$J$ : the number of outer folds
$\hat R_{\mathrm{outer},j}$ : the model's score on outer test fold $j$

The difference between the outer and inner means estimates the inner-outer gap for this table:

\widehat{\mathrm{gap}} = \bar R_{\mathrm{outer}}-\bar R_{\mathrm{inner}},

where the outer mean uses each fold's held-out score while the inner mean uses that fold's best-selected candidate, so their difference is a descriptive inner-outer gap:

$\widehat{\mathrm{gap}}$ : the estimated gap between the outer and inner means
$\bar R_{\mathrm{inner}}$ : the mean of the best inner-fold scores across outer folds
$\bar R_{\mathrm{outer}}$ : the mean of the outer-fold scores across outer folds
$J$ : the number of outer folds
$\hat R_{\mathrm{CV},j}(\lambda)$ : the inner cross-validation score for candidate $\lambda$ on outer fold $j$ , computed only on the training portion of outer fold $j$
$\hat R_{\mathrm{outer},j}$ : the model's score on outer test fold $j$

The difference is positive when the outer mean exceeds the selected inner mean. Selection, different fitting sample sizes, and finite-sample variation all contribute; this gap does not isolate selection bias. The output reports this gap. It also reports the outer standard deviation and the standard error

\widehat{\mathrm{SE}}=\frac{s_{\mathrm{outer}}}{\sqrt{J}}

where:

$\widehat{\mathrm{SE}}$ : the independence-based standard-error benchmark for this illustrative table
$s_{\mathrm{outer}}$ : the sample standard deviation of the outer-fold scores
$J$ : the number of outer folds

The fold standard deviation describes spread among these five values. Dividing it by $\sqrt{J}$ would be a standard-error estimate for independent identically distributed fold scores, but overlapping training sets invalidate that justification for ordinary nested CV. Treat the printed outer SE as an independence benchmark, not a calibrated uncertainty interval.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Comments

Add to the conversation

Stay up to date