Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Robustness, Calibration, and Safe Control › Calibrated Uncertainty and Abstention

Michael BrenndoerferAugust 7, 202610 min read

World Models HandbookAll chapters

Part 12 of 12Reliable World Models

Chapter 68Robustness, Calibration, and Safe Control › Calibrated Uncertainty and Abstention

Reading section 3 of 8 · Calibrated Uncertainty and Abstention · 10 minSection 3 of 8 · 10 min

Section 3 of 8 · 10 min read

Robustness, Calibration, and Safe Control › Calibrated Uncertainty and AbstentionOne section · Sections in this chapter3 / 8 ↓

Sections in this chapter8

Reading tools Definitions & inline links

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

A model is calibrated if its stated probabilities match observed frequencies. Concretely: among all the predictions where the model says "there is a 90% chance the true value lies in this interval," roughly 90% should contain the truth. Calibration is a property of the whole predictive distribution, not of any single prediction, so it can only be measured in aggregate. This is a subtle but important point: you cannot ask whether a single interval is calibrated, any more than you can ask whether a single coin flip is fair. You ask it about the set of predictions that share the same confidence level, and you compare the claimed frequency to the realized one.

Calibration vs. sharpness

Coverage and width answer different questions. An interval of $(-\infty,\infty)$ has coverage one, not calibrated 90% coverage. A useful interval should be narrow while achieving its stated coverage on the relevant population. Report both width and coverage; also inspect operationally important subgroups rather than relying only on an aggregate average.

In Bayesian Filtering and Belief States, a posterior represents state uncertainty; Stochasticity and Uncertainty distinguishes uncertainty sources. Here we additionally approximate predictions by $\mathcal N(\mu(x),\sigma^2(x))$ . Producing a mean and variance does not imply Gaussian shape. Quantile-based coverage can be tested without finite moments; standardized residuals using mean and standard deviation require finite moments and a positive scale.

Measuring coverageLink Copied

For a Gaussian predictive distribution, the natural diagnostic is the standardized residual. If the model is honest, each observation $y_i$ should look like a draw from $\mathcal{N}(\mu(x_i), \sigma^2(x_i))$ , so dividing the raw error by the claimed standard deviation puts every prediction on a common scale:

z_i = \frac{y_i - \mu(x_i)}{\sigma(x_i)}.

where:

$y_i$ : the observed target for the $i$ -th sample
$\mu(x_i)$ : the model's predicted mean at input $x_i$
$\sigma(x_i)$ : the model's predicted standard deviation at input $x_i$
$z_i$ : the standardized residual, measuring how many predicted standard deviations the observation deviates from the mean

For the Gaussian location-scale intervals used here, calibration at all central levels requires the absolute standardized residuals to have the corresponding Gaussian quantiles. Calibration alone does not imply that signed residuals are standard normal. The empirical coverage at nominal level $\alpha$ is:

\hat{c}(\alpha) = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\!\left[\,|z_i| \le \Phi^{-1}\!\left(\tfrac{1+\alpha}{2}\right) \right] \approx \alpha .

where:

$N$ : the number of evaluated samples
$\alpha \in (0,1)$ : the nominal coverage level, the fraction of probability mass the interval is supposed to contain
$\Phi^{-1}$ : the standard normal quantile function, so $\Phi^{-1}\!\left(\tfrac{1+\alpha}{2}\right)$ is the half-width of the central interval containing fraction $\alpha$ of a standard normal
$\mathbf{1}[\cdot]$ : the indicator, equal to $1$ when the inequality holds and $0$ otherwise
$\hat{c}(\alpha)$ : the empirical fraction of samples whose standardized residual falls inside that central interval

Plotting $\hat{c}(\alpha)$ against $\alpha$ gives a reliability diagram. The diagonal represents matching marginal coverage at the displayed levels; curves below it deliver less coverage than claimed, and curves above it deliver more. Sampling uncertainty remains, and the diagram does not establish conditional coverage at every state. Overcoverage may accompany unnecessarily wide intervals; undercoverage matters when decisions rely on margins that are too narrow. Neither direction alone measures controller safety.

In[13]:

Code

LEVELS = np.array([0.50, 0.68, 0.80, 0.90, 0.95])


def z_scores(X, Y, scale=1.0):
    mu_z, var_z, _ = predict(members, X)
    return ((Y - mu_z) / np.sqrt(var_z * scale**2)).ravel()


def coverage_curve(z, levels=LEVELS):
    return np.array(
        [(np.abs(z) <= norm.ppf(0.5 + lv / 2.0)).mean() for lv in levels]
    )


def fit_scale(X, Y):
    """One global multiplier that makes the average squared z-score equal to one."""
    return float(np.sqrt(np.mean(z_scores(X, Y) ** 2)))


global_scale = fit_scale(X_recal, Y_recal)

z_id = z_scores(X_test, Y_test)
z_dyn = z_scores(X_dyn, Y_dyn)
cover_id = coverage_curve(z_id)
cover_id_scaled = coverage_curve(z_scores(X_test, Y_test, global_scale))
cover_dyn = coverage_curve(z_dyn)
oracle_scale = float(np.sqrt(np.mean(z_dyn**2)))
cover_dyn_scaled = coverage_curve(z_scores(X_dyn, Y_dyn, oracle_scale))

Out[14]:

Console

global scale fitted on in-distribution calibration data: 1.000
scale that would be needed on the shifted regime:        1.065

  nominal   nominal cov   shifted cov
     0.50         0.492         0.493
     0.68         0.667         0.671
     0.80         0.789         0.775
     0.90         0.891         0.862
     0.95         0.947         0.929

The executed global scale is 1.000 and the shifted-set diagnostic is 1.065. At nominal 90% coverage, the measured frequencies are 89.1% on nominal test transitions and 86.2% on shifted transitions. This is a modest empirical coverage loss in this experiment, not evidence of an arbitrary catastrophic collapse.

Read the two coverage columns at the same nominal level. They are measured frequencies for the specified data splits, not guarantees. The shifted-data scale is an oracle diagnostic fitted and evaluated on that same shifted set; it is not a deployable recalibration result or a held-out validation estimate.

The fitted multiplier adjusts average squared standardized residuals, not every conditional probability or tail. It may help on some shifted populations, but coverage does not automatically transfer. Our calibration targets are correlated transitions from independent episodes, so the curve describes their empirical mixture; it is not a finite-sample confidence guarantee for an adaptive controller.

The code pools both coordinate residuals. In this simulator, $\Delta p=\Delta t\,(v_t+\Delta v)$ , not $\Delta t\,\Delta v$ . The feature map includes current velocity, so the fitted predictions satisfy $\widehat{\Delta p}=\Delta t\,(v_t+\widehat{\Delta v})$ , up to numerical roundoff. Subtracting these predictions from their targets leaves position residuals equal to $\Delta t$ times velocity residuals; the standardized coordinate residuals therefore coincide up to numerical roundoff. In a general multi-output model, pooled coverage can hide opposing coordinate errors; report coordinate-specific diagnostics as well.

Split conformal prediction provides a different route to marginal coverage: fit a model on separate training data, compute nonconformity scores on $n$ calibration examples, and use the appropriate order statistic, typically rank $\lceil(n+1)(1-\delta)\rceil$ , for target coverage $1-\delta$ . If the rank exceeds $n$ , an infinite threshold is the conservative convention. The finite-sample marginal result requires exchangeability of the calibration and test examples, conditional on the independently fitted predictor. Consecutive transitions from the same episode and adaptive control generally do not satisfy that assumption. Marginal coverage also does not guarantee coverage at every state or for an entire trajectory. This chapter implements Gaussian residual rescaling, not conformal intervals; see Angelopoulos and Bates for the distinction and conditions.

Out[15]:

Visualization

Nominal empirical coverage compared with nominal levels. — Nominal held-out transition coverage at five Gaussian interval levels, with and without a global scale fitted on a separate calibration split. Matching aggregate frequencies do not establish conditional or trajectory coverage.

Out[16]:

Visualization

Shifted empirical coverage with a diagnostic oracle rescaling. — Coverage under the damping shift with raw uncertainty and a shifted-data oracle scaling. The oracle uses the same outcomes for fitting and evaluation, so it is diagnostic rather than independently validated.

Abstention: turning uncertainty into a decisionLink Copied

Calibration describes predictions on a population. Abstention is a decision rule built on top of an uncertainty score. Here we rank predictions by predictive variance and evaluate only a retained fraction. An informative ranking may reduce retained-set error; it does not entitle a model to act safely. An actual abstention system must also specify what happens to the rejected cases and evaluate that alternative.

Possible designs include a planner that defers high-disagreement proposals, a manipulator that requests confirmation, or a controller that selects a predefined fallback. These are examples of design choices, not claims about every deployed system. A percentile trigger asks whether a score is unusual relative to a reference population; an absolute trigger asks whether a specified tolerance is exceeded. Either can be appropriate when its operating population and consequences have been validated.

One way to evaluate this ranking is the risk-coverage curve. Sweep the retained fraction $\rho$ , compute retained-set RMSE, and plot it against $\rho$ . The area summarizes this particular loss and population, so report the loss definition and coverage range. An uncertainty score independent of errors gives approximately unchanged expected error under random retention; finite samples fluctuate. Useful rankings often give lower error at lower coverage, but empirical curves need not be monotone.

In[17]:

Code

def risk_coverage(X, Y, fractions):
    """RMSE of the retained predictions as a function of the retained fraction."""
    mu_rc, var_rc, _ = predict(members, X)
    per_point_err = np.sqrt(np.mean((Y - mu_rc) ** 2, axis=1))
    per_point_sd = np.sqrt(var_rc.mean(axis=1))
    order = np.argsort(per_point_sd)  # most confident first
    curve = []
    for frac in fractions:
        keep = order[: max(1, int(round(frac * len(order))))]
        curve.append(float(np.sqrt(np.mean(per_point_err[keep] ** 2))))
    return np.array(curve)


FRACTIONS = np.linspace(0.15, 1.0, 18)
curve_id = risk_coverage(X_test, Y_test, FRACTIONS)
curve_in = risk_coverage(X_in, Y_in, FRACTIONS)
curve_dyn = risk_coverage(X_dyn, Y_dyn, FRACTIONS)

Out[18]:

Console

retain 0.25 | in-dist 0.0421 | input shift 0.0425 | dynamics shift 0.0424
retain 0.50 | in-dist 0.0430 | input shift 0.0424 | dynamics shift 0.0427
retain 1.00 | in-dist 0.0437 | input shift 0.0451 | dynamics shift 0.0459

At retained fraction 0.25, executed aggregate RMSEs are 0.0421, 0.0425, and 0.0424 for nominal, input-shifted, and damping-shifted data. At full coverage they are 0.0437, 0.0451, and 0.0459. The input-shift curve has 0.0424 at fraction 0.50, slightly below its 0.25 value: the empirical curve is not strictly monotone. These modest differences describe the chosen score and populations, not a universal benefit of abstention.

Three questions guide the reading. Does retaining fewer predictions reduce RMSE in the nominal population? Is that relationship preserved under input shift? Does it persist under dynamics shift? A favorable result in one regime cannot answer the other two. The score combines a residual variance floor with ensemble disagreement, while realized errors also contain unpredictable process noise. That noise can dominate errors and make a useful epistemic ranking difficult to see in aggregate RMSE.

Ranking and calibration are separate properties. A single positive rescaling preserves score rankings while potentially changing interval coverage. A percentile threshold targets a rejection fraction on its reference population, subject to finite-sample quantile and tie conventions, but a rolling window can normalize a sustained harmful shift into the new normal. An absolute threshold can encode a physically meaningful tolerance but needs validation under changed conditions. Neither form automatically survives distribution shift. Choose thresholds using retained-case risk, rejected-case outcomes, and the cost of deferral, not coverage alone.

Out[19]:

Visualization

Line chart of prediction error against retained fraction for three evaluation regimes. — Risk-coverage curves for the nominal, input-shifted, and damping-shifted test populations. Each point reports RMSE after retaining the lowest-variance fraction. Non-monotonic or small differences indicate that this uncertainty ranking is not a reliable selector of lower aggregate error in every regime.