Calibration and Parameter Estimation in Models › Overfitting and Model Selection
Quantitative FinanceAll chapters Calibration and Parameter Estimation in Models › Overfitting and Model SelectionOne section · Sections in this chapter13 / 17 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
Overfitting means fitting sample-specific noise or patterns that do not transfer to the intended prediction task. An overfit model can have nonzero training residuals, and exact interpolation alone does not imply poor generalization. Compare calibration fit with suitable held-out data rather than identifying overfitting from training error alone.
The tendency of flexible models to fit noise in calibration data, leading to unstable parameters and poor out-of-sample performance. Overfit models appear excellent on calibration instruments but fail on new data.
Symptoms of Overfitting
Overfitting manifests in several characteristic ways that you learn to recognize:
-
Parameter instability is a warning to investigate: small data changes can expose weak identification, poor conditioning, optimizer problems or changing market conditions, as well as overfitting. Erratic fitted parameters alone do not establish that the model is fitting noise.
-
Extreme or boundary parameter values warrant checking constraints, data, model assumptions and the optimizer. They may reflect underlying dynamics and do not alone prove that the model is fitting noise. Economic intuition can help assess whether the fitted values are plausible.
-
Poor out-of-sample performance tests generalization. A precise calibration fit followed by large errors on held-out instruments is a warning, but does not isolate overfitting from misspecification or distribution shift. Reserve separate data for testing and investigate the source of the failure.
Sources of Overfitting
The primary sources of overfitting include:
-
Too many parameters relative to information. Extra free parameters can fit noise when the data weakly constrain them. Information depends on parameter sensitivities and quote-error covariance; redundant sensitivities or correlated errors can reduce it. Adjacent strikes alone do not establish a universal information ranking, and effective independent observations differ from fitted model degrees of freedom.
-
Noisy data. Measurement error, staleness and temporary dislocations can influence fitted parameters. Such artifacts may persist or be correlated, but they need not transfer to the intended pricing or forecasting task. Assess their data-quality origin and dependence rather than assuming they vanish immediately.
-
Model misspecification. A restricted model can use compensating parameters to approximate dynamics it cannot represent. This is distinct from fitting sample noise, though the two can interact. Extra flexibility can also reduce genuine systematic bias; assess held-out behavior and the economic meaning of the fitted parameters.
Preventing Overfitting
Several strategies mitigate overfitting:
Regularization adds penalties for complex solutions:
where:
- : penalty function (e.g., parameter magnitude or parameter variability)
- : regularization parameter controlling the trade-off between fit and complexity
Regularization encourages the chosen measure of simplicity or stability. It can improve generalization when the penalty and its strength suit the problem, but it also adds bias. Specify the parameter scaling and choose the penalty using an appropriate validation procedure.
Cross-validation fits on training subsets and evaluates held-out data. It estimates predictive error for the chosen data and split design; randomly held-out strikes do not establish performance in future market regimes. Use date-based or other deployment-relevant holdouts when that is the intended task.
Information criteria like AIC and BIC provide principled model selection that balances fit quality against model complexity:
where:
- : Akaike Information Criterion
- : Bayesian Information Criterion
- : number of parameters
- : sample size
- : maximized likelihood value
Compare these criteria using compatible maximized likelihoods on the same observations and a count of all fitted free parameters, including nuisance parameters such as an estimated error variance. An arbitrary SSE is not itself . Lower values favor a model by the selected criterion under its assumptions, not universally better predictions. BIC's penalty exceeds AIC's when ; their usual asymptotic interpretations require suitable regularity and identification.
Justified parameter bounds can exclude implausible values and reduce some extreme noise-fitting, but they do not prevent overfitting within the interval. Historical parameter ranges should not be transferred to a different pricing measure without a model of that relationship.
Let's evaluate held-out fit for the fixed-beta SABR specification:
from sklearn.model_selection import KFold
def cross_validate_sabr(strikes, market_vols, F, T, beta, n_folds=5):
"""
Cross-validation for SABR calibration.
Returns the unweighted mean of held-out-fold RMSEs.
"""
strikes, market_vols = validate_sabr_quotes(strikes, market_vols, F, T, beta)
if not isinstance(n_folds, (int, np.integer)) or not 2 <= n_folds <= len(strikes):
raise ValueError("Require 2 <= n_folds <= number of quotes")
kf = KFold(n_splits=n_folds, shuffle=True, random_state=42)
cv_errors = []
for train_idx, test_idx in kf.split(strikes):
# Training data
strikes_train = strikes[train_idx]
vols_train = market_vols[train_idx]
# Test data
strikes_test = strikes[test_idx]
vols_test = market_vols[test_idx]
# Calibrate on training set
x0 = [0.25, -0.20, 0.35]
bounds = [(0.01, 1.0), (-0.99, 0.99), (0.01, 2.0)]
result = minimize(
calibration_objective,
x0,
args=(F, T, strikes_train, vols_train, beta),
method="L-BFGS-B",
bounds=bounds,
)
require_successful_fit(result, "SABR cross-validation fold")
alpha_cv, rho_cv, nu_cv = result.x
# Evaluate on test set
test_errors = []
for K, vol_true in zip(strikes_test, vols_test):
vol_model = sabr_implied_vol(F, K, T, alpha_cv, beta, rho_cv, nu_cv)
if not np.isfinite(vol_model) or vol_model <= 0:
raise RuntimeError("Invalid held-out SABR volatility")
test_errors.append((vol_model - vol_true) ** 2)
cv_errors.append(np.sqrt(np.mean(test_errors)))
return np.mean(cv_errors)
# Generate more strikes for better CV demonstration
strikes_full = np.linspace(75, 125, 21)
market_vols_full = np.array(
[
sabr_implied_vol(
F_true, K, sabr_maturity, alpha_true, beta_true, rho_true, nu_true
)
for K in strikes_full
]
)
market_vols_full += np.random.normal(0, 0.008, len(strikes_full))
cv_rmse = cross_validate_sabr(
strikes_full, market_vols_full, F_true, sabr_maturity, beta_fixed, n_folds=5
)Cross-Validation RMSE: 0.73 vol points (73.4 bps)
This unweighted mean of fold RMSEs summarizes error on randomly held-out strikes for these training subsets and this synthetic dataset. A small value is evidence about that evaluation design, not proof of future-market performance or the absence of overfitting. Compare suitable baselines and reserve independent data for any subsequent model-selection claims.
Parameter Stability Analysis
Examining how calibrated parameters change with different data provides insight into identification and overfitting:
def bootstrap_parameter_stability(
strikes, market_vols, F, T, beta, n_bootstrap=50, rng=None
):
"""Paired resampling with explicit solver screening and failure counts."""
strikes, market_vols = validate_sabr_quotes(strikes, market_vols, F, T, beta)
if not isinstance(n_bootstrap, (int, np.integer)) or n_bootstrap <= 0:
raise ValueError("n_bootstrap must be a positive integer")
rng = np.random.default_rng(42) if rng is None else rng
params_samples = []
failed = 0
n = len(strikes)
for _ in range(n_bootstrap):
idx = rng.choice(n, size=n, replace=True)
result = minimize(
calibration_objective,
[0.25, -0.20, 0.35],
args=(F, T, strikes[idx], market_vols[idx], beta),
method="L-BFGS-B",
bounds=[(0.01, 1.0), (-0.99, 0.99), (0.01, 2.0)],
)
if result.success and np.isfinite(result.fun) and np.all(np.isfinite(result.x)):
params_samples.append(result.x.copy())
else:
failed += 1
print(
f"Bootstrap fits accepted: {len(params_samples)}/{n_bootstrap}; failed: {failed}"
)
if failed:
raise RuntimeError(
"Bootstrap contains failed fits; inspect them before inference"
)
return np.array(params_samples)
param_samples = bootstrap_parameter_stability(
strikes_full, market_vols_full, F_true, sabr_maturity, beta_fixed, n_bootstrap=100
)
Parameter Stability Summary: -------------------------------------------------- Parameter | Mean | Std Dev | Relative dispersion (%) | True -------------------------------------------------- alpha | 0.174 | 0.0318 | 18.3% | 0.200 rho | -0.398 | 0.1307 | 32.9% | -0.300 nu | 0.378 | 0.0702 | 18.6% | 0.400
These bootstrap samples describe how the fitted parameters respond to resampling this particular synthetic dataset. Their spread and any shift from the generating values are useful uncertainty diagnostics, not proof of structural or unique identification or of the absence of overfitting. The displayed relative dispersion is 100 times the standard deviation divided by the absolute sample mean. It becomes unstable near a zero mean; for signed parameters such as correlation, inspect absolute spread, scale and the sampled distribution rather than treating this ratio as a scale-free identification measure.
