Prediction Intervals and Split Conformal Prediction › Disadvantages

The guarantee is marginal. It averages over calibration data and a fresh test point, rather than promising the target coverage at every input or within every subgroup. Constant-width intervals can over-cover low-noise regions and under-cover high-noise regions. A particular realized calibration set and finite test batch can also have coverage below nominal. None of these observations alone contradicts the marginal theorem.

Exchangeability can fail under distribution shift and ordinary serial dependence. A future regime can differ from the calibration period, and a sequence whose dependence changes with temporal lag is generally not invariant to arbitrary permutations. Dependence alone is not enough to rule out exchangeability: dependent exchangeable samples exist. A chronological split or rolling recalibration does not by itself restore the ordinary split-conformal guarantee.

Holding out calibration data leaves fewer observations for fitting. When data are scarce, related methods such as jackknife+ and CV+ reuse observations through additional model fits. Their constructions and finite-sample bounds differ from ordinary split conformal. Jackknife+ is the leave-one-out special case of CV+, obtained when each fold contains one observation. These methods should not be treated as interchangeable ways to preserve the split-conformal guarantee. This chapter keeps the separate fitting and calibration roles visible.

Conformal calibration does not improve the predictor itself. Large calibration errors can require a wide margin, and a constant margin can still leave a difficult subgroup under-covered. A finite calibration sample does not force the realized test fraction to meet nominal. Width is useful alongside coverage, but model choice, score choice and sampling variation all affect it.

Discrete scores and ties can make deterministic sets conservative. Classification sets can contain several labels or be empty. For the probability-threshold score, an empty set means that no class probability reaches the calibrated threshold. It does not by itself establish that the input is out of distribution. A downstream rule can add a label to an empty set, which only enlarges the set and therefore cannot reduce inclusion coverage.

Exchangeability is the assumption that the joint distribution of the $n+1$ points (the $n$ calibration points plus the test point) is invariant to permutation of their indices. In symbols, for any permutation $\pi$ of $\{1,\dots,n+1\}$, the joint distribution of $(Z_1,\dots,Z_{n+1})$ equals that of $(Z_{\pi(1)},\dots,Z_{\pi(n+1)})$, where $Z_i = (X_i, Y_i)$. Independent and identically distributed data are exchangeable. Exchangeable data need not be independent: a uniformly ordered sample without replacement from a fixed population is exchangeable but dependent. A calibration/test distinction that systematically changes the joint distribution lacks this permutation symmetry.

Write $\hat C(x)$ for the prediction set returned at input $x$. Let $\alpha\in(0,1)$ be the target miscoverage level, so $1-\alpha$ is the target inclusion probability.

Marginal coverage is $P(Y_{n+1}\in\hat C(X_{n+1}))\ge1-\alpha$, with probability over calibration and test draws. Conditional coverage would require $P(Y_{n+1}\in\hat C(X_{n+1})\mid X_{n+1}=x)\ge1-\alpha$ at every relevant $x$. For regression over unrestricted distributions with continuously varying covariates, exact distribution-free conditional coverage has strong impossibility results: nontrivial finite-length intervals cannot be guaranteed uniformly without additional restrictions (Vovk, 2012; Barber, Candès, Ramdas and Tibshirani, 2021). Group-conditional guarantees and assumptions about the conditional distribution are different, more restricted targets.

These distinctions give practical failure checks. Compare calibration and deployment populations; check whether temporal or clustered structure defeats exchangeability; and keep model fitting, preprocessing and selection outside the calibration and test outcomes. Training residuals can be optimistic because fitting used their responses. Reusing calibration outcomes for model selection can also destroy the held-out symmetry argument. Either practice removes the ordinary proof; it does not imply an identical numerical shortfall in every dataset.