Prediction Intervals and Split Conformal Prediction

A regression model hands us a single number: $\hat f(x)$. If we are forecasting tomorrow's demand, pricing an insurance contract, or deciding whether a patient needs a follow-up test, that number alone does not describe how wrong the prediction might be. A prediction of 1,000 units that is usually off by 20 is a different business situation from one that is usually off by 400, even though both point predictions are 1,000.

Part II introduced confidence intervals for population quantities. In regression, a confidence interval for the conditional mean $\mathbb{E}[Y\mid X=x]$ describes estimation uncertainty about that mean. A future observation $Y$ has additional randomness around it. A prediction interval targets that future observation. The familiar Gaussian linear-model construction uses distributional and variance assumptions; split conformal takes a different route.

Split conformal prediction takes a fitted predictor, calibrates a score on examples held out from fitting and model selection, and returns an interval or label set. Its finite-sample guarantee requires the calibration and fresh test scores to be exchangeable, conditional on the fitting procedure. It guarantees marginal coverage over calibration and test draws. It does not guarantee coverage for each individual, subgroup, fixed calibration sample, or observed test batch.

The mechanism is simple. We measure how badly the model misses on the calibration set, take a particular quantile of those misses, and add that quantile as a margin around every future prediction. Almost all of the subtlety lives in exactly which quantile to take, and in understanding precisely what the resulting guarantee does and does not promise. That is where we will spend our time.

This chapter builds directly on Probability Calibration, where we asked whether a model's stated probabilities match observed frequencies. Conformal prediction asks a related but distinct question for regression: not whether the model is right, but how wide a band we must draw around it before a stated fraction of future observations fall inside.