Prediction Intervals and Split Conformal Prediction › Example

A small numerical example makes the quantile convention concrete. Let us work through a complete split conformal calculation by hand with ten calibration points and a simple regression model, so that every index in the formula is visible.

We have fitted $\hat f$ on a training set and evaluated it on a calibration set of $n = 10$ points, obtaining absolute residuals. Sorted, they are

$$0.05,\; 0.12,\; 0.19,\; 0.27,\; 0.31,\; 0.45,\; 0.53,\; 0.62,\; 0.74,\; 0.88 .$$

The largest residual is $0.88$ in the response units of this illustrative example. Suppose we want 90% coverage, so $\alpha=0.10$ and $1-\alpha=0.90$. The index is

$$k = \left\lceil (n+1)(1-\alpha) \right\rceil = \left\lceil (10+1)\times 0.90 \right\rceil = \lceil 9.9 \rceil = 10 .$$

where:

• $n = 10$: the calibration set size
• $\alpha = 0.10$: the miscoverage level, so $1-\alpha = 0.90$ is the nominal coverage
• $k$: the ceiling index, which selects the $k$-th smallest calibration score

Because $k = 10 = n$, the conformal quantile is the largest observed residual, $\hat q = 0.88$. A new input with $\hat f(x) = 5.00$ gets the interval

$$\hat C(x) = [\,5.00 - 0.88,\; 5.00 + 0.88\,] = [\,4.12,\; 5.88\,].$$

If the realized value turns out to be $y = 5.5$, it is inside the interval and this particular point is covered. If $y = 6.2$, it is not; the interval width of $1.76$ is simply not enough to catch a miss that large. The marginal guarantee averages over calibration and fresh test draws; it does not certify this particular point.

With almost surely distinct exchangeable scores, the repeated-sample coverage for this rule is $10/11\approx0.909$. With ties, that value is a lower bound. This is a statement about the procedure over calibration and test draws, not the conditional coverage of the particular ten residuals displayed above. At this level and calibration size, the rule selects the sample maximum.

Now compare an 80% interval, $\alpha = 0.20$. Then

$$k = \left\lceil (n+1)(1-\alpha) \right\rceil = \left\lceil 11 \times 0.80 \right\rceil = \lceil 8.8 \rceil = 9,$$

where:

• $\alpha = 0.20$: the miscoverage level, so $1-\alpha = 0.80$
• $k = 9$: the resulting index, so $\hat q = s_{(9)} = 0.74$
• $n = 10$: the calibration set size, giving the rank lower bound $9/(10+1) \approx 0.818$; equality holds for almost surely distinct exchangeable scores, while ties can, but need not, increase coverage

With $\hat q = 0.74$, the same prediction becomes $[4.26, 5.74]$, narrower as expected, with guarantee $9/11 \approx 0.818$.

The $(n+1)$ convention matters here. Suppose we had carelessly used $\lceil n(1-\alpha) \rceil$ instead. For $\alpha = 0.10$ that gives $\lceil 9.0 \rceil = 9$ and $\hat q = 0.74$, exactly the margin we should have used for the 80% interval. The reported interval would cover about $9/10 = 0.90$ of calibration points, matching the nominal level on the calibration data, yet its coverage of future points is only guaranteed at the lower level $k/(n+1) = 9/11 \approx 0.818$, not at the nominal $0.90$. The shortfall comes from replacing $(n+1)$ with $n$: the resulting index is too small by roughly one position, so at $\alpha = 0.10$ the effective guarantee falls below the nominal $0.90$. The wrong index can appear correct on the data used to compute it.

Now the classification case. Suppose the same ten calibration points give scores $1 - \hat p(y_i \mid x_i)$, sorted:

$$0.02,\; 0.05,\; 0.11,\; 0.18,\; 0.22,\; 0.30,\; 0.41,\; 0.55,\; 0.66,\; 0.80 .$$

At $\alpha = 0.10$ we again have

$$k = \left\lceil (n+1)(1-\alpha) \right\rceil = \left\lceil 11 \times 0.90 \right\rceil = 10 .$$

So $\hat q = s_{(10)} = 0.80$, giving the probability threshold $1 - 0.80 = 0.20$:

• $k = 10$: the resulting index, the tenth smallest calibration score
• $\hat q = 0.80$: the conformal quantile, equal to $s_{(10)}$
• $1-\hat q = 0.20$: the calibrated probability threshold applied to each class

A new input with class probabilities $(0.55, 0.30, 0.15)$ yields the set $\{A, B\}$: $A$ and $B$ clear the $0.20$ bar and $C$ does not. The set has size two. If the true label is $C$, this set misses it. The marginal bound does not assign an error probability to this particular vector or set.

Consider a classifier that spreads its probability mass across all three labels. Suppose the probabilities for some input are $(0.40, 0.33, 0.27)$. Since all three exceed $0.20$, the prediction set is all three labels. That input is effectively unclassified, and the set says so. Conversely, an input with probabilities $(0.90, 0.07, 0.03)$ yields the singleton $\{A\}$. Set size reports how many labels this score and calibrated threshold retain at that input. It does not supply a calibrated probability that a singleton is correct or prove that an empty set is out of distribution.