Evaluating Survival Predictions › Disadvantages

Concordance does not measure probability calibration. Two models with exactly the same ranking of units have exactly the same concordance index, even if one predicts a survival probability of 0.9 for a unit whose true survival probability is 0.2. A concordance of 0.75 therefore says nothing about whether the predicted probabilities can be used in a budget. Given the supplied scalar scores, it does not identify the full shape of the predicted curves. If those scores encode median ordering, errors in tail probabilities can remain invisible. A change to a curve can still change concordance when it changes the chosen score's ordering. If a decision needs probabilities, concordance is a screening tool, not the answer.

Harrell's concordance is also sensitive to the censoring distribution. The set of usable pairs is determined by who was censored when, so studies of the same population can target different comparable-pair mixes when follow-up differs. Cross-study comparisons therefore need their follow-up patterns and target populations stated. The IPCW-corrected version of concordance targets a censoring-independent estimand, but it buys that independence with a modelling assumption: the censoring model must match the required independence structure, and censoring survival must be positive over the specified target window. The pooled weights here assume censoring independent of failure time and prediction covariates.

IPCW metrics inherit the same fragility. Inverse weights are unstable by construction: if $\hat{G}(t)$ approaches zero, the weight $1/\hat{G}(t)$ grows without bound and a single observation can dominate the average. Positivity failures are not hypothetical. In the six-unit teaching cohort below, the last unit is censored at the final follow-up time, which drives the estimated censoring survival to exactly zero, so the strict Brier estimator used here rejects that horizon and later ones. A larger sample cannot remove a population support failure. Large inverse weights can increase variance, and the estimated Brier score can exceed one in small samples even though the population quantity is bounded in $[0,1]$. The reason is that the IPCW estimator is an average of weighted squared errors rather than a convex combination: a single small $\hat{G}$ inflates one weight, so one unit's weighted squared error can exceed one and push the average above the $[0,1]$ bound that holds for the unweighted population quantity.

A Brier score at one horizon reduces event timing to a binary outcome. At 1,500 hours, a unit that fails at 1,600 hours and one observed running through 3,600 hours both have the survivor outcome; their errors can still differ if their predicted probabilities differ. If timing across a window matters, evaluate several horizons or integrate over a specified interval. This does not remove the censoring assumptions. Withdrawal related to impending failure can make the pooled censoring probabilities used here inappropriate, potentially biasing the weighted scores; a larger sample does not correct a misspecified observation model.