Stochasticity and Uncertainty › Calibration
World Models HandbookAll chapters Stochasticity and Uncertainty › CalibrationOne section · Sections in this chapter6 / 11 ↓
Reading tools Definitions & inline links
Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.
Article links
Make inline references clickable
A predictive distribution can be sharply peaked and wrong, or wide and correct, or wide and wrong. Calibration ties claimed confidence to observed frequencies under an evaluation design, and downstream decisions often rely on that relationship. Miscalibrated probabilities can still carry information, but they do not match observed frequencies as claimed. Material undercoverage can cause a planner that trusts those intervals to underprice risk.
A predictive distribution is calibrated if outcomes are consistent with the probabilities it assigns under the evaluation design. For predictive intervals: among comparable cases assigned a 90 percent interval, about 90 percent of realized outcomes should fall inside. For continuous full distributions, probability integral transform values should be uniform under the required calibration conditions.
For continuous predictive distributions, outcomes transformed through the predicted CDF form the probability integral transform (PIT). Uniformity is necessary for probabilistic calibration under the usual setup. A U shape often suggests underdispersion, while a central hump often suggests overdispersion, but these shapes are not unique diagnoses and should be checked alongside bias and dependence.
The PIT diagnostic is easiest to trust once seen. The histogram below transforms held-out outcomes through the predicted CDFs of a deliberately overconfident model. Under the usual conditions, a calibrated model yields PIT values that are uniform in the population and an approximately flat held-out histogram up to sampling variation; the U shape here shows too much mass at the extremes, the signature of intervals that are too narrow.
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import norm
## Deterministic construction: true noise std 0.20, but the model claims 0.10
## (half the true standard deviation and one quarter of its variance).
## Seed fixed; test set size fixed at 2000.
rng_pit = np.random.default_rng(11)
n_pit = 2000
pred_mean_pit = rng_pit.normal(0.0, 0.5, n_pit) # well-specified means
true_out_pit = pred_mean_pit + 0.20 * rng_pit.standard_normal(n_pit)
claimed_std_pit = 0.10
pit_values = norm.cdf(true_out_pit, pred_mean_pit, claimed_std_pit)
The standard diagnostic is the reliability diagram. Choose nominal coverage levels, compute the empirical fraction of outcomes inside each corresponding predictive interval, and plot empirical against nominal. A curve below the diagonal indicates undercoverage. On the classification benchmarks studied by Guo et al. (2017), modern neural classifiers were often poorly calibrated and held-out logit temperature scaling was an effective post-hoc correction. That result does not establish out-of-distribution regression calibration or justify rescaling predictive variances without separate validation.
Two complementary metrics are worth tracking together:
- Calibration error: how far the selected reliability curve sits from the diagonal. For binned probability calibration, expected calibration error weights each bin's deviation by its frequency; a predictive-interval coverage curve needs its own stated summary across coverage levels.
- Sharpness: how narrow the predictive distributions are, averaged over inputs. Among calibrated models, sharper is better. Assess sharpness alongside calibration because a sharp forecast that materially undercovers can create false confidence.
The pairing matters because each metric is incomplete on its own. A model can achieve some calibration criteria with predictions too diffuse to be useful, and it can look sharp while under-covering outcomes. The practical goal is sharpness subject to the chosen calibration criterion; this is forecast-evaluation guidance, not a consequence of the bias-variance decomposition.
A model can be calibrated on held-out one-step transitions yet miscalibrated on model-generated or policy-induced states. One-step calibration does not transfer automatically to that population. When a model is deployed inside feedback, evaluate calibration on deployment-representative closed-loop rollouts or use sequential methods designed for adaptive data. Part XI on evaluation returns to this in depth.