Prediction Intervals and Split Conformal Prediction › Summary

Confidence intervals target population quantities; prediction intervals target future observations. Split conformal constructs sets from a predictor and score fixed before calibration. Compute held-out scores, choose $k=\lceil(n+1)(1-\alpha)\rceil$, use the $k$th calibration score when $k\le n$, and use $+\infty$ otherwise. Absolute-residual scores turn that cutoff into a symmetric margin around the prediction.

The guarantee $P(Y_{n+1}\in\hat C(X_{n+1}))\ge1-\alpha$ averages over calibration and fresh test draws under score exchangeability. Almost surely distinct scores also give the upper slack bound $1-\alpha+1/(n+1)$; ties can be more conservative. Using an ordinary uncorrected empirical quantile does not provide the same general lower-bound argument.

This marginal guarantee does not ensure conditional or subgroup coverage, nor nominal coverage for every realized test batch. Training-only positive normalization can redistribute width, but our crude scale proxy still has local failures. Classification uses the same rank mechanism to threshold candidate label scores. An empty classification set necessarily misses the true label; a set containing every label in an exhaustive class space necessarily includes it. For proper nonempty subsets, cardinality alone does not calibrate individual inclusion.

Check score construction and data roles before interpreting the guarantee. Distribution shift, ordinary serial dependence and reuse of calibration outcomes for model selection can remove its justification. Keep fitting, calibration and evaluation separate, handle the infinite-threshold case explicitly, and monitor labeled deployment outcomes. The coverage theorem is a precise probability statement under assumptions, not a promise that every reported interval or observed batch succeeds.