Statistical Modelling › Cross-Validation: Evaluating Models on Held-Out Data

Simply splitting data into training and test sets provides a basic evaluation framework, but this approach has limitations. The performance estimate depends heavily on which specific observations end up in each set, leading to high variance in our assessment. Cross-validation addresses this problem by systematically using different portions of the data for training and testing, producing more stable and reliable performance estimates.

The most common cross-validation approach is k-fold cross-validation, which divides the dataset into k equally sized folds. The algorithm then trains k different models, each time using k-1 folds for training and the remaining fold for validation. By averaging performance across all k trials, we obtain a robust estimate that reduces the influence of any particular train-test split.

Typical choices for k range from 5 to 10, with k=5 or k=10 being most common. Larger values of k provide more training data for each model and evaluate more splits, but increase computational cost. The extreme case, leave-one-out cross-validation where k equals the number of observations, provides maximum data efficiency but can be prohibitively expensive for large datasets.

The process begins by randomly shuffling the data and partitioning it into k folds. In the first iteration, we train our model on folds 2 through k and evaluate it on fold 1. In the second iteration, we train on folds 1 and 3 through k while evaluating on fold 2. This continues until each fold has served as the validation set exactly once. The final performance metric averages the results across all k iterations, often reported with a standard error to capture variability.

This systematic rotation through the data ensures that every observation is used for both training and validation, maximizing the information we extract from limited data. It also provides insight into model stability. If performance varies dramatically across folds, this suggests the model is sensitive to the specific training examples it sees, which may indicate overfitting or insufficient training data.

For classification problems, particularly those with imbalanced classes, stratified cross-validation offers an important refinement. Standard k-fold cross-validation might by chance create folds where rare classes are underrepresented or missing entirely. Stratified cross-validation ensures that each fold maintains approximately the same class distribution as the complete dataset, leading to more representative performance estimates and preventing situations where the validation set contains no examples of certain classes.

Beyond evaluating a single model's performance, cross-validation helps with comparing different models and selecting hyperparameters. When we need to choose between different model architectures or tune parameters like regularization strength, cross-validation gives an honest assessment of how each configuration will perform on new data. The key is to avoid using the test set for these decisions. Instead, we use cross-validation on the training data to select the best model, then evaluate that final model on a held-out test set that has never been seen during the model development process.

This example illustrates how 5-fold cross-validation systematically rotates through the data, with each fold serving as validation data once while the remaining folds provide training data.