Prediction Intervals and Split Conformal Prediction

A regression model hands us a single number: $\hat f(x)$. If we are forecasting tomorrow's demand, pricing an insurance contract, or deciding whether a patient needs a follow-up test, that number alone does not describe how wrong the prediction might be. A prediction of 1,000 units that is usually off by 20 is a different business situation from one that is usually off by 400, even though both point predictions are 1,000.

Part II introduced confidence intervals for population quantities. In regression, a confidence interval for the conditional mean $\mathbb{E}[Y\mid X=x]$ describes estimation uncertainty about that mean. A future observation $Y$ has additional randomness around it. A prediction interval targets that future observation. The familiar Gaussian linear-model construction uses distributional and variance assumptions; split conformal takes a different route.

Split conformal prediction takes a fitted predictor, calibrates a score on examples held out from fitting and model selection, and returns an interval or label set. Its finite-sample guarantee requires the calibration and fresh test scores to be exchangeable, conditional on the fitting procedure. It guarantees marginal coverage over calibration and test draws. It does not guarantee coverage for each individual, subgroup, fixed calibration sample, or observed test batch.

The mechanism is simple. We measure how badly the model misses on the calibration set, take a particular quantile of those misses, and add that quantile as a margin around every future prediction. Almost all of the subtlety lives in exactly which quantile to take, and in understanding precisely what the resulting guarantee does and does not promise. That is where we will spend our time.

This chapter builds directly on Probability Calibration, where we asked whether a model's stated probabilities match observed frequencies. Conformal prediction asks a related but distinct question for regression: not whether the model is right, but how wide a band we must draw around it before a stated fraction of future observations fall inside.

The coverage bound holds in finite samples under exchangeability of the held-out scores. It does not require a Gaussian error distribution or a correctly specified predictor. A fixed Gaussian margin can under-cover when its assumptions fail. Split conformal instead calibrates the chosen score, including an infinite threshold when the calibration sample is too small for a finite cutoff. The resulting sets can be wide or uninformative while remaining marginally valid.

Conformal prediction wraps a fitted model. Linear regression, a boosted ensemble, or another predictor can supply the score, provided fitting and selection are settled using fitting data only. The residual-score version needs predictions on $n$ calibration examples and a sort of $n$ scores, conventionally $O(n\log n)$ time for sorting. This extra work depends on the predictor and calibration size; it is not universally negligible.

Changing the score can change the shape and adaptivity of the output. Absolute residuals give constant-width regression intervals; positive input-dependent scaling can redistribute width; class-probability scores give label sets. All use the same marginal rank argument when their scores are fixed before calibration. Better probability calibration or a different score may improve set efficiency, but neither guarantees smaller sets or better subgroup coverage in every problem.

Interval width and set size can inform operational rules, such as routing broad intervals to human review. Such a rule is an additional selection step: marginal coverage of the original procedure does not imply the same coverage among automatically accepted cases. Evaluate the selected cases separately and distinguish a useful triage signal from a calibrated conditional error rate.

The guarantee is marginal. It averages over calibration data and a fresh test point, rather than promising the target coverage at every input or within every subgroup. Constant-width intervals can over-cover low-noise regions and under-cover high-noise regions. A particular realized calibration set and finite test batch can also have coverage below nominal. None of these observations alone contradicts the marginal theorem.

Exchangeability can fail under distribution shift and ordinary serial dependence. A future regime can differ from the calibration period, and a sequence whose dependence changes with temporal lag is generally not invariant to arbitrary permutations. Dependence alone is not enough to rule out exchangeability: dependent exchangeable samples exist. A chronological split or rolling recalibration does not by itself restore the ordinary split-conformal guarantee.

Holding out calibration data leaves fewer observations for fitting. When data are scarce, related methods such as jackknife+ and CV+ reuse observations through additional model fits. Their constructions and finite-sample bounds differ from ordinary split conformal. Jackknife+ is the leave-one-out special case of CV+, obtained when each fold contains one observation. These methods should not be treated as interchangeable ways to preserve the split-conformal guarantee. This chapter keeps the separate fitting and calibration roles visible.

Conformal calibration does not improve the predictor itself. Large calibration errors can require a wide margin, and a constant margin can still leave a difficult subgroup under-covered. A finite calibration sample does not force the realized test fraction to meet nominal. Width is useful alongside coverage, but model choice, score choice and sampling variation all affect it.

Discrete scores and ties can make deterministic sets conservative. Classification sets can contain several labels or be empty. For the probability-threshold score, an empty set means that no class probability reaches the calibrated threshold. It does not by itself establish that the input is out of distribution. A downstream rule can add a label to an empty set, which only enlarges the set and therefore cannot reduce inclusion coverage.

Exchangeability is the assumption that the joint distribution of the $n+1$ points (the $n$ calibration points plus the test point) is invariant to permutation of their indices. In symbols, for any permutation $\pi$ of $\{1,\dots,n+1\}$, the joint distribution of $(Z_1,\dots,Z_{n+1})$ equals that of $(Z_{\pi(1)},\dots,Z_{\pi(n+1)})$, where $Z_i = (X_i, Y_i)$. Independent and identically distributed data are exchangeable. Exchangeable data need not be independent: a uniformly ordered sample without replacement from a fixed population is exchangeable but dependent. A calibration/test distinction that systematically changes the joint distribution lacks this permutation symmetry.

Write $\hat C(x)$ for the prediction set returned at input $x$. Let $\alpha\in(0,1)$ be the target miscoverage level, so $1-\alpha$ is the target inclusion probability.

Marginal coverage is $P(Y_{n+1}\in\hat C(X_{n+1}))\ge1-\alpha$, with probability over calibration and test draws. Conditional coverage would require $P(Y_{n+1}\in\hat C(X_{n+1})\mid X_{n+1}=x)\ge1-\alpha$ at every relevant $x$. For regression over unrestricted distributions with continuously varying covariates, exact distribution-free conditional coverage has strong impossibility results: nontrivial finite-length intervals cannot be guaranteed uniformly without additional restrictions (Vovk, 2012; Barber, Candès, Ramdas and Tibshirani, 2021). Group-conditional guarantees and assumptions about the conditional distribution are different, more restricted targets.

These distinctions give practical failure checks. Compare calibration and deployment populations; check whether temporal or clustered structure defeats exchangeability; and keep model fitting, preprocessing and selection outside the calibration and test outcomes. Training residuals can be optimistic because fitting used their responses. Reusing calibration outcomes for model selection can also destroy the held-out symmetry argument. Either practice removes the ordinary proof; it does not imply an identical numerical shortfall in every dataset.

We now build the machinery. The aim is to turn a bare point prediction into a band, and to do it in a way that comes with a guarantee we can state precisely. Three questions have to be answered along the way:

1. What are we trying to cover? Not the regression surface, but the next observation. If the response retains nondegenerate noise, additional fitting data do not remove that noise.
2. How do we measure a miss? We need a number that grows when a candidate observation becomes implausible under the fitted model, a nonconformity score.
3. How do we convert finitely many measured misses into a margin that is valid for the next one? This is where exchangeability enters, and where a seemingly cosmetic $+1$ turns out to carry the entire guarantee.

Notation throughout: fit the predictor and score on $\mathcal D_{\mathrm{train}}$, then use a disjoint calibration set $\mathcal D_{\mathrm{cal}}=\{(X_i,Y_i)\}_{i=1}^n$. A fresh test pair is $(X_{n+1},Y_{n+1})$. Conditional on the fitted score function, require these $n+1$ calibration and test scores to be exchangeable. Independent observations from one distribution, with an independent fitting split, are a sufficient setting. The separate test sample in the example evaluates the completed procedure.

Before building the procedure, distinguish three targets: a population quantity, a future observation, and a class label. The target determines which inclusion statement we need to establish.

A confidence interval targets a population quantity, such as a mean, coefficient or quantile. For example, an interval for $\mu(x)=\mathbb E[Y\mid X=x]$ aims to contain that fixed conditional mean across repeated fitting samples at the specified input. Its width can shrink under regular consistent estimation; shrinking is not an unconditional property of every confidence procedure.

A prediction interval targets a future observation and aims for a specified inclusion probability. It accounts for response variation as well as uncertainty in the fitted predictor. With persistent nondegenerate noise it can remain wide even when mean estimation is precise. For a symmetric noise distribution, a symmetric band around its center uses the $(1-\alpha)$ quantile of the absolute noise as its half-width.

A prediction set is the classification analogue: a random subset $\hat C(x) \subseteq \{1,\dots,K\}$ with $P\big(Y_{n+1} \in \hat C(X_{n+1})\big) \ge 1-\alpha$. Whereas an interval is an ordered range on the real line, a set is an unordered collection of labels, and its natural size measure is cardinality rather than length.

For comparison, consider the correctly specified linear model with $n$ fitting observations and $p$ nonconstant predictors plus an intercept. In this comparison only, $n$ denotes fitting size rather than calibration size. Assume $n>p+1$, a fixed full-rank design, independent Gaussian errors with variance $\sigma^2$, and an independent future error with the same variance. At a fixed new design vector $x_0$, the Gaussian prediction interval is

where:

• $\hat y_0 = x_0^\top \hat\beta$: the fitted value at the new input
• $\hat\sigma^2$: the residual variance estimate (the sum of squared residuals divided by $n-p-1$)
• $X$: the full-rank $n\times(p+1)$ training design matrix, including an intercept column and $p$ nonconstant predictors
• $t_{n-p-1,1-\alpha/2}$: the $1-\alpha/2$ quantile of a $t$ distribution with $n-p-1$ degrees of freedom
• $x_0^\top (X^\top X)^{-1} x_0$: the new-point leverage, with $x_0$ containing the intercept and the same $p$ predictors as the design matrix

The "$1+$" inside the square root reflects the additional noise of the future observation. It is there because the error of prediction decomposes as

Under these Gaussian-model assumptions, conditional on the fixed design and new input, the new observation error is independent of the fitting-sample estimation error. Their variances therefore add:

where:

• $Y_{n+1}$: the future observation whose value is not yet known
• $\hat y_{n+1}$: the fitted model's prediction at the future input $x_{n+1}$
• $\mu(x_{n+1})$: the true conditional mean $\mathbb{E}[Y \mid X = x_{n+1}]$
• $\sigma^2$: the noise variance $\operatorname{Var}(Y_{n+1} - \mu(x_{n+1}))$

Under regular increasing-sample designs with a fixed number of predictors, mean-estimation uncertainty decreases. The mean confidence interval uses only the estimation term, while the prediction interval also contains the future-noise term. For a 95% Gaussian interval, its large-sample noise margin is approximately $1.96\sigma$. This comparison explains the extra $1$ in the displayed square root.

The weakness of this construction is the assumption stack: Gaussian errors, constant variance, a correctly specified linear model, and independent observations. Split conformal keeps the same target, the observation, but replaces every one of those assumptions with exchangeability alone.

The engine of conformal prediction is the nonconformity score. The idea is to condense each candidate pair $(x, y)$ and the fitted model $\hat f$ into a single number, $s(x,y)$, that grows as the pair becomes less plausible under the model. A score is doing its job when it is small for observations the model explains well and large for observations it explains poorly.

For regression with a scalar response the standard choice is the absolute residual

where:

• $s_i$: the nonconformity score for calibration point $i$, measuring how far the model's prediction misses the observed response
• $y_i$: the observed response for calibration point $i$
• $\hat f(x_i)$: the fitted model's prediction for calibration point $i$

For classification with fitted class probabilities $\hat p(y \mid x)$ the standard choice is

where:

• $s_i$: the nonconformity score for calibration point $i$, equal to one minus the probability of the observed label
• $\hat p(y_i \mid x_i)$: the fitted probability the model assigns to the observed label $y_i$ at input $x_i$
• $y_i$: the observed class label of calibration point $i$

This score is small when the model assigns high probability to the observed label and large when it assigns low probability. Many other scores are possible; we will meet several. Choose and freeze the score function without using the calibration or test outcomes for fitting or selection. At prediction time, evaluate it for candidate labels or responses; the unknown true future response is not supplied to the procedure.

Given the scores $s_1,\dots,s_n$, sort them and write the order statistics as

The conformal quantile at miscoverage level $\alpha$ is the order statistic at index

For $k\le n$, set $\hat q=s_{(k)}$. If $k=n+1$, set $\hat q=+\infty$ and return the full response space. Clipping $k$ to $n$ would lose the requested guarantee at small calibration sizes.

where:

• $n$: the calibration set size
• $\alpha$: the miscoverage level, so $1-\alpha$ is the nominal coverage
• $k$: the ceiling index $\lceil (n+1)(1-\alpha)\rceil$, which can equal $n+1$; that case uses an infinite threshold
• $s_{(k)}$: the $k$-th smallest calibration score
• $\hat q$: the conformal score threshold; for absolute-residual regression it is the interval half-width

This implementation selects an order statistic without interpolation. Other quantile conventions need to be checked against the stated finite-sample cutoff; an ordinary interpolating call is not generally equivalent.

The $\lceil \cdot \rceil$ symbol denotes the ceiling function: the smallest integer greater than or equal to its argument. For $n = 100$ and $\alpha = 0.10$ we get $(n+1)(1-\alpha) = 101 \times 0.90 = 90.9$, so $k = \lceil 90.9 \rceil = 91$ and $\hat q = s_{(91)}$.

To see the role of $n+1$, first assume the $n+1$ exchangeable scores are almost surely distinct. The test score then has a uniform rank in $\{1,\ldots,n+1\}$. For $k\le n$, it falls below the $k$th calibration score exactly when that rank is at most $k$. Thus

where:

• $s_{n+1}$: the nonconformity score of the future test point, an exchangeable draw from the same pool as the calibration scores
• $s_{(k)}$: the $k$-th smallest calibration score, used as the conformal quantile $\hat q$
• $k$: the ceiling index $\lceil (n+1)(1-\alpha)\rceil$, computed from the calibration size $n$ and miscoverage level $\alpha$

The equality holds because $s_{n+1} \le s_{(k)}$ happens exactly when the rank of $s_{n+1}$ is one of the $k$ smallest.

The rank calculation behind the guarantee can be inspected directly. With $n=10$ and $\alpha=0.10$, we have $(n+1)(1-\alpha)=11\times0.90=9.9$, so $k=\lceil9.9\rceil=10$. Under distinct exchangeable scores, the test score has eleven equally likely ranks, and ranks one through ten fall at or below the calibrated cutoff $s_{(10)}$; only the largest rank falls outside it. The cell below prepares the rank probabilities for the figure: it computes the ceiling index without clipping, builds uniform rank probabilities, and flags the ranks covered by the cutoff. This is an exact rank illustration, with no sampled data.

This bound arises because $k \le n$ requires $\lceil (n+1)(1-\alpha)\rceil \le n$, equivalently $(n+1)(1-\alpha) \le n$ since $n$ is an integer. Solving:

Putting the pieces together, split conformal regression is a four-step procedure, sometimes called inductive conformal prediction (Papadopoulos et al., 2002). Notice how cleanly the steps separate the modelling from the calibration. The model is fit and frozen first, and only then does the conformal machinery measure its mistakes.

1. Fit the point predictor $\hat f$ on $\mathcal{D}_{\text{train}}$. Nothing about this step is special; use whatever model you like and whatever model selection procedure you like, as long as it never touches $\mathcal{D}_{\text{cal}}$.
2. Compute calibration scores $s_i = |y_i - \hat f(x_i)|$ for $i = 1,\dots,n$.
3. Compute $k=\lceil(n+1)(1-\alpha)\rceil$. Set $\hat q=s_{(k)}$ if $k\le n$ and $\hat q=+\infty$ otherwise.
4. For a new input $x$, return $\hat C(x) = \big[\hat f(x) - \hat q,\; \hat f(x) + \hat q\big]$ when $\hat q$ is finite, and return the full response space when $\hat q = +\infty$.

The interval is the point prediction plus or minus a constant margin. When $\hat q$ is finite its width is exactly $2\hat q$ everywhere, independent of $x$; when $\hat q = +\infty$ the set is the full response space. Here $\hat q$ is the conformal quantile computed from calibration scores, and there is no guarantee that averaging a realized test batch reproduces the nominal coverage. That is both its simplicity and its main weakness, and we will return to the consequence shortly.

Step 4 is worth reading backwards, because that is how the guarantee is usually proved. The event $Y_{n+1} \in \hat C(X_{n+1})$ is the same as $|Y_{n+1} - \hat f(X_{n+1})| \le \hat q$, which is the same as $s_{n+1} \le \hat q$. Everything reduces to the rank argument in the previous subsection. The proof does not require a particular predictor family or parametric response law. Its sampling argument uses exchangeability of the held-out scores.

For classification, the same four steps apply with a different score, and step 4 becomes a set construction:

where:

• $y$: a candidate class label in $\{1,\dots,K\}$
• $\hat p(y \mid x)$: the fitted probability the model assigns to label $y$ at input $x$
• $\hat q$: the conformal quantile from the calibration scores, so $1-\hat q$ is the probability threshold

For the score $1-\hat p(y\mid x)$, classification keeps labels whose fitted probability is at or above $1-\hat q$. The calibration step chooses the threshold automatically, in such a way that the fraction of calibration points whose nonconformity score $1-\hat p(y_i\mid x_i)$ exceeds $\hat q$ is at most $\alpha$. This uses held-out outcomes to calibrate set inclusion. It is different from Probability Calibration, which evaluates or adjusts probability reliability. Accurate class probabilities are useful, but are not required for this marginal set-coverage proof.

The threshold applies to each candidate label, so set size varies with the probability vector and the calibrated cutoff. Several labels may pass, or none may pass. The marginal guarantee controls overall inclusion under its sampling assumptions. For a proper nonempty subset of the class space, set size alone does not determine its conditional inclusion probability. Set size alone also does not establish an out-of-distribution interpretation.

Several properties follow from the construction and are worth naming. Treat them as a checklist of what the formula does and does not give us.

• Coverage is finite-sample. Under the score-exchangeability condition, $P(Y_{n+1}\in\hat C(X_{n+1}))\ge1-\alpha$ holds for every $n\ge1$ and $0<\alpha<1$, using the infinite-threshold branch when necessary. No asymptotic approximation is used.
• The guarantee is distribution-free over score distributions satisfying exchangeability. The predictor and score construction must be fixed before calibration; no correctly specified parametric response model is required.
• Monotonicity in $\alpha$. With the same calibration scores, decreasing $\alpha$ cannot decrease $k$ or $\hat q$. The sets are nested, although the discrete index can stay unchanged over a range of levels.
• Invariance to monotone rescaling of scores. Replacing $s$ by any strictly increasing transform $g(s)$ yields the same prediction sets, since the ranking is unchanged. The transform must be well-defined and the same for every calibration and candidate score. The small-calibration branch still returns the full response space; it does not require evaluating $g(+\infty)$. Dividing by an input-dependent scale is a different score construction and can change the resulting sets.
• Width depends on the score distribution. With a fixed score and independent calibration samples growing in size, the empirical cutoff converges to its population score quantile at continuity and identifiability points. When the fitted score also changes, additional consistency assumptions are needed. Improving a predictor under one loss does not universally shrink every conformal cutoff.
• Ties can make deterministic inclusion conservative: $P(s_{n+1}\le\hat q)$ may exceed $k/(n+1)$. This removes the distinct-score upper slack bound; it does not by itself prove that an interval can be narrowed while keeping the same guarantee.

A small numerical example makes the quantile convention concrete. Let us work through a complete split conformal calculation by hand with ten calibration points and a simple regression model, so that every index in the formula is visible.

We have fitted $\hat f$ on a training set and evaluated it on a calibration set of $n = 10$ points, obtaining absolute residuals. Sorted, they are

The largest residual is $0.88$ in the response units of this illustrative example. Suppose we want 90% coverage, so $\alpha=0.10$ and $1-\alpha=0.90$. The index is

where:

• $n = 10$: the calibration set size
• $\alpha = 0.10$: the miscoverage level, so $1-\alpha = 0.90$ is the nominal coverage
• $k$: the ceiling index, which selects the $k$-th smallest calibration score

Because $k = 10 = n$, the conformal quantile is the largest observed residual, $\hat q = 0.88$. A new input with $\hat f(x) = 5.00$ gets the interval

If the realized value turns out to be $y = 5.5$, it is inside the interval and this particular point is covered. If $y = 6.2$, it is not; the interval width of $1.76$ is simply not enough to catch a miss that large. The marginal guarantee averages over calibration and fresh test draws; it does not certify this particular point.

With almost surely distinct exchangeable scores, the repeated-sample coverage for this rule is $10/11\approx0.909$. With ties, that value is a lower bound. This is a statement about the procedure over calibration and test draws, not the conditional coverage of the particular ten residuals displayed above. At this level and calibration size, the rule selects the sample maximum.

Now compare an 80% interval, $\alpha = 0.20$. Then

where:

• $\alpha = 0.20$: the miscoverage level, so $1-\alpha = 0.80$
• $k = 9$: the resulting index, so $\hat q = s_{(9)} = 0.74$
• $n = 10$: the calibration set size, giving the rank lower bound $9/(10+1) \approx 0.818$; equality holds for almost surely distinct exchangeable scores, while ties can, but need not, increase coverage

With $\hat q = 0.74$, the same prediction becomes $[4.26, 5.74]$, narrower as expected, with guarantee $9/11 \approx 0.818$.

The $(n+1)$ convention matters here. Suppose we had carelessly used $\lceil n(1-\alpha) \rceil$ instead. For $\alpha = 0.10$ that gives $\lceil 9.0 \rceil = 9$ and $\hat q = 0.74$, exactly the margin we should have used for the 80% interval. The reported interval would cover about $9/10 = 0.90$ of calibration points, matching the nominal level on the calibration data, yet its coverage of future points is only guaranteed at the lower level $k/(n+1) = 9/11 \approx 0.818$, not at the nominal $0.90$. The shortfall comes from replacing $(n+1)$ with $n$: the resulting index is too small by roughly one position, so at $\alpha = 0.10$ the effective guarantee falls below the nominal $0.90$. The wrong index can appear correct on the data used to compute it.

Now the classification case. Suppose the same ten calibration points give scores $1 - \hat p(y_i \mid x_i)$, sorted:

At $\alpha = 0.10$ we again have

So $\hat q = s_{(10)} = 0.80$, giving the probability threshold $1 - 0.80 = 0.20$:

• $k = 10$: the resulting index, the tenth smallest calibration score
• $\hat q = 0.80$: the conformal quantile, equal to $s_{(10)}$
• $1-\hat q = 0.20$: the calibrated probability threshold applied to each class

A new input with class probabilities $(0.55, 0.30, 0.15)$ yields the set $\{A, B\}$: $A$ and $B$ clear the $0.20$ bar and $C$ does not. The set has size two. If the true label is $C$, this set misses it. The marginal bound does not assign an error probability to this particular vector or set.

Consider a classifier that spreads its probability mass across all three labels. Suppose the probabilities for some input are $(0.40, 0.33, 0.27)$. Since all three exceed $0.20$, the prediction set is all three labels. That input is effectively unclassified, and the set says so. Conversely, an input with probabilities $(0.90, 0.07, 0.03)$ yields the singleton $\{A\}$. Set size reports how many labels this score and calibrated threshold retain at that input. It does not supply a calibrated probability that a singleton is correct or prove that an empty set is out of distribution.

We build the conformal quantile and interval arithmetic from NumPy primitives, using scikit-learn for the base predictors. Fitting and calibration data construct the intervals; untouched test data evaluate them. The example keeps all three roles disjoint and performs no model or score selection using calibration or test outcomes.

The data are one-dimensional and heteroscedastic, with the noise standard deviation growing quadratically away from the origin. This is a realistic situation where a constant-width interval will struggle, so it doubles as a stress test.

The interval width grows as nominal coverage rises. Empirical test coverage fluctuates around the target and can fall below it for a particular calibration sample and finite test set; the marginal theorem does not require every row of this table to exceed nominal.

Three key design choices are the miscoverage level $\alpha$, the calibration size $n$, and the score. Understanding their roles is what separates a working deployment from a mysterious one.

The level $\alpha$ specifies the tolerated marginal miscoverage. Its choice depends on the decision, the cost of misses and the usefulness of the resulting sets. Calibration size constrains whether that level permits a finite cutoff, and the experiment uses $\alpha=0.10$ as an illustrative choice.

The calibration size controls rank granularity. With distinct scores the coverage excess from rounding is less than $1/(n+1)$; this is a probability-scale bound, not a universal bound on excess interval width. For example, $n=19$ and $\alpha=0.10$ give $k=18$. Quantile variability also depends on the score distribution, especially its density near the cutoff, so there is no distribution-free sample count that guarantees a stable narrow interval.

The score is the main modelling lever. Absolute residuals give constant width. Normalized residuals give locally adaptive width. Signed residuals give one-sided bounds. We turn to the adaptive case next, because it is where the constant-width weakness becomes visible.

Absolute-residual scores use the same cutoff at every input. An input-dependent positive scale can redistribute width. Fix that scale using fitting data only, then calibrate $s_i=|y_i-\hat f(x_i)|/\hat\sigma(x_i)$. The same exchangeability proof applies, because the score is fixed before calibration. Here $\hat\sigma(x)>0$ is an error-scale proxy, not a validated conditional standard deviation; the construction does not guarantee improved conditional coverage.

We build $\hat\sigma(x)$ by regressing the training absolute residuals on $|x|$, then clip it away from zero to avoid division blow-ups. This is a crude but honest noise model, and it is fitted only on training data, consistent with the requirement that the score function be frozen before calibration.

The printed classification coverage can be slightly below nominal on this finite test sample. In this seeded run there are no empty sets: most inputs receive one label, some receive two, and one receives all three. A calibration-set check is different from a test-set guarantee: by construction, the fraction of calibration scores above the chosen cutoff is at most $\alpha$. None of these summaries gives conditional correctness for a specific singleton.

The main parameters that govern the conformal procedure are the miscoverage level $\alpha$, the calibration size $n$, and the choice of nonconformity score.

• alpha: The miscoverage level, so $1-\alpha$ is the nominal coverage. It specifies the target marginal miscoverage and determines the interval width through the quantile index. There is no universally correct value; this example uses $0.10$.
• n (calibration size): Controls the rank grid and the probability-scale rounding bound. A finite threshold requires $n\ge1/\alpha-1$; otherwise the full-space branch is valid. Stability and width also depend on the score distribution.
• scores (nonconformity score): The main modelling lever. Absolute residuals give constant-width intervals; scaled residuals (dividing by a training-only local noise estimate) give locally adaptive width; signed residuals give one-sided bounds. The same strictly increasing scalar transform applied to every score preserves the sets; an input-dependent normalization generally changes them.

The figures reuse the fitted models, calibration scores and test observations. Ordinary calculation cells prepare the additional cutoff, prediction grid, bin summaries and set-size counts; rendering cells do not refit the models or regenerate the data.

The histogram displays calibration absolute residuals and the cutoffs for two nominal levels. Scores to the right of a cutoff exceed that calibrated margin. The plot illustrates an empirical order statistic, not a promise that an identical fraction of every future batch will lie in the tail. It is a descriptive view of one synthetic calibration sample drawn once from the seeded generator, not a repeated-seed evaluation of variability in the cutoff.

Set size counts the labels retained by this particular probability score and calibrated cutoff. A larger set leaves more labels unresolved.

• An empty set necessarily misses the true label.
• A set containing all $K$ labels necessarily includes the true label, provided the declared class space is exhaustive.
• The size of a proper nonempty subset, including a singleton when $K>1$, does not by itself determine an individual inclusion probability.

Set size alone does not identify an out-of-distribution input. Those conditional error-rate and out-of-distribution interpretations require separate evaluation. The displayed counts come from one seeded classification split, so the absence of empty sets is a conditional observation on that sample rather than a repeated-seed estimate of the empty-set rate.

Split conformal is useful when a fitted predictor needs an additional marginal inclusion guarantee and an appropriate held-out calibration sample is available. Valuation or screening examples can use width to expose the limits of a point prediction. Classification sets can support routing or abstention rules, but those selected decisions need separate evaluation. Sequential demand forecasting additionally requires an argument for its dependence and deployment assumptions; ordinary split conformal is not automatically valid there.

Fit and select the predictor, preprocessing and score using fitting data only, then use held-out calibration outcomes to choose the cutoff. Keep evaluation outcomes outside those choices as well. Data-reuse alternatives such as jackknife+ and CV+ require their own refitting procedures and coverage results. Conformalized quantile regression, discussed in relation to Quantile Regression, calibrates a score built from fitted lower and upper quantile functions. It can redistribute width in heteroscedastic problems; neither the initial quantile fit nor its conformalized version automatically supplies distribution-free conditional coverage.

Plan calibration size against the desired level. A finite cutoff is available when $\alpha\ge1/(n+1)$; otherwise the infinite threshold still gives valid but uninformative sets. The distinct-score rounding excess is below $1/(n+1)$, while the sampling variability and width of the cutoff depend on the score distribution. Report observed coverage together with a width summary and the calibration and evaluation sample sizes.

Two implementation pitfalls are worth checking directly. An ordinary call to np.quantile(scores, 1-alpha) need not implement the required finite-sample index. Computing scores from data used to fit or select the predictor can also invalidate the held-out proof. Training errors can be optimistic, but the size and direction of a particular test-coverage change are not determined solely by model capacity. Use the specified cutoff and keep the data roles separate.

Monitor labeled deployment outcomes for changes in coverage and width. Refreshing calibration data can respond to a changing population, but it does not itself establish exchangeability. A chronological split, a blocked holdout trained on earlier observations, or a rolling-origin split supports honest temporal evaluation when fitting, preprocessing and feature construction use only information available at each forecast origin; it does not automatically provide conformal validity under serial dependence. Specialized online and dependent-data methods have their own assumptions and targets, such as long-run rather than per-step coverage. Their guarantees must be checked for the intended forecasting task.

When intervals are too wide for the intended decision, examine the predictor and the score as well as the data. Lowering $\alpha$ asks for higher coverage and cannot narrow intervals built from the same scores. Improving one predictive loss does not universally reduce every conformal quantile, and a narrow interval alone does not establish useful individual coverage. Evaluate the tradeoff on an appropriate held-out population.

Confidence intervals target population quantities; prediction intervals target future observations. Split conformal constructs sets from a predictor and score fixed before calibration. Compute held-out scores, choose $k=\lceil(n+1)(1-\alpha)\rceil$, use the $k$th calibration score when $k\le n$, and use $+\infty$ otherwise. Absolute-residual scores turn that cutoff into a symmetric margin around the prediction.

The guarantee $P(Y_{n+1}\in\hat C(X_{n+1}))\ge1-\alpha$ averages over calibration and fresh test draws under score exchangeability. Almost surely distinct scores also give the upper slack bound $1-\alpha+1/(n+1)$; ties can be more conservative. Using an ordinary uncorrected empirical quantile does not provide the same general lower-bound argument.

This marginal guarantee does not ensure conditional or subgroup coverage, nor nominal coverage for every realized test batch. Training-only positive normalization can redistribute width, but our crude scale proxy still has local failures. Classification uses the same rank mechanism to threshold candidate label scores. An empty classification set necessarily misses the true label; a set containing every label in an exhaustive class space necessarily includes it. For proper nonempty subsets, cardinality alone does not calibrate individual inclusion.

Check score construction and data roles before interpreting the guarantee. Distribution shift, ordinary serial dependence and reuse of calibration outcomes for model selection can remove its justification. Keep fitting, calibration and evaluation separate, handle the infinite-threshold case explicitly, and monitor labeled deployment outcomes. The coverage theorem is a precise probability statement under assumptions, not a promise that every reported interval or observed batch succeeds.

Ready to test your understanding? Take this quick quiz to reinforce what you've learned about prediction intervals and split conformal prediction.