Prediction Intervals and Split Conformal Prediction › Visualizing Split Conformal Prediction

The figures reuse the fitted models, calibration scores and test observations. Ordinary calculation cells prepare the additional cutoff, prediction grid, bin summaries and set-size counts; rendering cells do not refit the models or regenerate the data.

The histogram displays calibration absolute residuals and the cutoffs for two nominal levels. Scores to the right of a cutoff exceed that calibrated margin. The plot illustrates an empirical order statistic, not a promise that an identical fraction of every future batch will lie in the tail. It is a descriptive view of one synthetic calibration sample drawn once from the seeded generator, not a repeated-seed evaluation of variability in the cutoff.

The calibration residuals have a long right tail in this synthetic example. The 90% cutoff exceeds the 80% cutoff, illustrating the width cost of the higher target on these same scores. The size of that increase depends on the score distribution.

The second figure shows the intervals themselves in input space. The band is the model plus and minus the fixed margin, and the points are test observations coloured by whether they fall inside.

On this seeded test split, the uncovered points are more frequent at large absolute inputs, where the noise is larger. The band has the same vertical width everywhere, so it cannot adapt to that local noise. The binned coverage in the next figure measures the difference directly.

The third figure quantifies that gap. We split the test inputs into five bins by $|x|$ and compute coverage within each bin, for the constant-width and the normalized-score intervals. The horizontal line marks nominal coverage.

Because the horizontal axis is $|x|$, moving right means moving farther from the low-noise center. Constant-width coverage falls at larger $|x|$. Normalization changes that pattern, but its crude scale estimate also under-covers the innermost bin. Similar overall coverage can therefore coexist with quite different local failures; neither score has a conditional-coverage guarantee.

The fourth figure shows coverage against nominal level for the sweep we computed, with the shifted-distribution result plotted as a single failing point.

The exchangeable evaluation lies near the diagonal and can cross it because calibration and test samples are finite. The shifted sample has coverage of about 0.807 at the same nominal 0.90 level. The predictor and calibrated margin are unchanged in that comparison; the input distribution was deliberately shifted. These are evaluations of one calibration sample and one shifted test draw, illustrating a failure mode when the original sampling assumption no longer applies.

The final figure summarizes the classification side. The distribution of prediction-set sizes tells us how much work the conformal threshold is doing.

Set size counts the labels retained by this particular probability score and calibrated cutoff. A larger set leaves more labels unresolved.

• An empty set necessarily misses the true label.
• A set containing all $K$ labels necessarily includes the true label, provided the declared class space is exhaustive.
• The size of a proper nonempty subset, including a singleton when $K>1$, does not by itself determine an individual inclusion probability.

Set size alone does not identify an out-of-distribution input. Those conditional error-rate and out-of-distribution interpretations require separate evaluation. The displayed counts come from one seeded classification split, so the absence of empty sets is a conditional observation on that sample rather than a repeated-seed estimate of the empty-set rate.