Evaluating Survival Predictions

We have spent the previous four chapters building survival models. We learned how censoring changes what a likelihood can see, how Kaplan–Meier and Nelson–Aalen estimate a marginal curve without covariates, how Cox proportional hazards turns covariates into hazard ratios, and how accelerated failure-time models describe the same data on the log-time scale. Once we have a fitted model, how do we know whether it is any good?

That question is harder than it looks in the survival setting, because the quantity we want to score is the thing censoring hides. Suppose we maintain a fleet of 900 industrial units. For each unit we record the hours it ran, a few operating covariates, and whether we saw it fail or merely stopped watching when the study ended. A model now returns two kinds of output. It may return a risk score, a single number per unit that orders units from "fails soonest" to "lasts longest". Or it may return a whole survival function, a curve $\hat{S}_i(t)$ giving the probability that unit $i$ survives past any horizon $t$ we care about. These are different products, and they support different decisions: a risk score tells a planner which units to inspect first, while survival probabilities can inform replacement planning over a two-year horizon. If each initial unit is replaced once on its first failure, summing the model's two-year failure probabilities estimates the expected replacement count. A replacement budget also needs unit costs and a policy for uncertainty.

Censoring blocks both evaluations. If a unit leaves the study at month 6, we cannot ask whether it survived past month 8, so we generally cannot determine its squared error for an ordinary mean-squared-error total. A prediction of one-half is the exception: its squared error is one-quarter for either outcome. If we simply discard it, we throw away the fact that it survived at least to month 6. If we pretend it survived past month 8, we have invented evidence. Manipulating censored durations is worse still: treating them as observed failures asserts that a unit died at the moment we stopped writing down its status. The metrics below use the observed event status and follow-up time without assigning an unobserved failure time.

Two censoring-aware tools carry most of the weight. The concordance index asks whether the model orders failure times correctly, scoring only the pairs of units for which the data alone determine who failed first. The time-dependent Brier score asks whether the model's predicted probabilities are close to the observed outcomes at a chosen horizon, reweighting the units we can fully observe so that they stand in for the units we cannot. We will derive both, discover exactly what each can and cannot see, and then apply them to a held-out equipment cohort where Kaplan–Meier, Cox, and Weibull AFT predictions are compared with censoring weights estimated only on the training partition. The third piece, calibration, is what closes the loop: a model can rank units perfectly and still be badly wrong about the numbers we plan with.