Evaluating Survival Predictions › Practical Implications

The evaluation metric should be chosen from the decision the model supports. A ranking decision, such as which unit to inspect first, which patient to prioritise, or which loan to review, calls for concordance, because Harrell's concordance averages ranking credit among its comparable observed pairs, with half credit for tied risk scores. Truncated IPCW concordance instead targets latent pairs whose earlier failure precedes its cutoff, under the censoring assumptions. A decision that needs a probability at a named horizon, such as how many spare units to hold, how much reserve to book, or how many transplant slots to plan for, calls for the Brier score at that horizon, because it is a proper scoring rule and penalises predicted probabilities that are wrong. Report both when ranking and probability forecasts matter: a model can rank well while its probabilities are poorly calibrated. Neither number diagnoses every failure, so inspect calibration directly as well.

The comparison in this chapter illustrates the separation. Kaplan–Meier scores exactly 0.5 on concordance, since a constant score produces only ties. Yet its Brier score can be close to covariate models at some horizons. That is not a contradiction. It reflects two different questions: discrimination asks whether the order is right, while accuracy asks whether the numbers are usable. A selection process that optimises only concordance can accept a model whose predicted probabilities are unusable for planning, and a process that optimises only the Brier score can hide a weak ranking behind a favourable marginal fit.

The executed example scores 180 test units with the dense pairwise concordance implementation. A matrix operation still needs quadratic pair storage; the Brier calculation is a weighted mean at each horizon, and bootstrap resampling repeats these calculations. Choose the implementation and resample count for the available resources. Include the supported horizons and uncertainty calculation in the model-validation plan.

The evaluation horizons belong in the analysis plan. Choose them from the decision context, fix them before looking at test results, and record why each one was chosen. A maintenance horizon should match the inspection cycle, a warranty horizon should match the warranty term, and a clinical horizon should reflect a clinical decision within adequately supported study follow-up. For the strict IPCW Brier estimator, the estimated censoring survival must stay positive at the horizon. Separately, report observed failures by the horizon, follow-up strictly beyond it, and uncertainty; there is no universal minimum count that establishes precision for every forecast and decision. Here we score 800, 1,500, and 2,500 hours and integrate over 300–3,000 hours, within positive estimated censoring support. The strict Brier estimator implemented here is undefined when $\hat G(\tau)=0$; an inclusive boundary rule would require its own matched observation indicator and left-limit weight.

Calibration addresses a different question from discrimination and should be checked directly when probabilities are used. A calibration diagnostic at horizon $\tau$ groups units by predicted survival, estimates observed survival within each group, and compares the two. A Kaplan–Meier group estimate requires censoring to be independent within that group. An IPCW alternative needs a censoring model valid for the group and positive support; weighting does not itself solve small-bin uncertainty. Here the display places observed survival against mean predicted survival with a diagonal reference. Its mean absolute deviation is a binned diagnostic, not a complete conditional-calibration test. Whole-curve diagnostics assess additional aspects: with continuous correctly specified survival distributions, the predicted survival evaluated at each true event time is uniform. Distributional calibration examines that property with an appropriate treatment of censored outcomes. It complements fixed-horizon checks rather than guaranteeing better diagnosis or a universal sample-size requirement.

Two checks keep ranking and probability assessment separate. First, do not read a c-index as evidence of calibration. In the Cox and common-shape Weibull models used here, multiplying every predicted hazard by ten leaves the model risk ranking unchanged while changing survival probabilities from $\hat S(t)$ to $\hat S(t)^{10}$. The same ranking invariance holds for a fixed-horizon score $-\hat S(\tau)$, but need not hold for medians or other curve functionals when curves cross. The changed probabilities can destroy calibration even when the original forecasts were calibrated. Second, keep the validation design clean. Fit models on training, estimate the censoring distribution on training, use the validation partition for choices that could overfit, and reserve the test partition for the final, prespecified evaluation phase. When several models are compared on the same test set, their estimates can be correlated, and selecting the best of many noisy test estimates can make the apparent winner optimistic, just as with any other metric discussed in Multiple Comparisons. Report intervals alongside point estimates, and prefer a nested design when the same data would otherwise be used for tuning and evaluation, following the logic of Hyperparameter Search and Nested Cross-Validation.

Treating censored durations as observed failures can produce plausible numbers from unobserved outcomes. Dropping the event indicator and entering a censored time as a failure asserts a failure that was not observed. Scoring an early-censored unit as a survivor makes the opposite unsupported imputation. The direction and size of the resulting distortion depend on the predictions and censoring pattern. If conclusions change sharply when censoring weights are switched on, the naive computation was doing substantial work and should not be trusted.

The censoring distribution is a nuisance component of the evaluation estimator. Estimating it on an independent training partition, as we do here, conditions the evaluation on those fitted weights. Estimating it on the evaluation cohort can also be valid under the censoring assumptions; uncertainty calculations then need to include that estimation step. In either design, the censoring model must describe the population being scored.

The metric helpers should reject unsupported evaluation requests before calculating weights: require positive finite censoring survival at every contributing time and horizon, an explicit supported truncation horizon for Uno concordance, and at least one comparable pair for either concordance. Do not replace a zero censoring probability by a tiny positive number. Report horizon support and comparable-pair counts with the results.

Withdrawal related to impending failure can invalidate the independence assumption behind these pooled IPCW weights. The resulting Brier and concordance estimates can target the wrong quantities; a larger sample does not make the censoring assumption true. Investigate why units leave the study before relying on the weights. When censoring is informative, a sensitivity analysis or a model that explicitly represents the dependence is needed; changing the evaluation metric alone cannot solve the problem.