Evaluating Survival Predictions

We have spent the previous four chapters building survival models. We learned how censoring changes what a likelihood can see, how Kaplan–Meier and Nelson–Aalen estimate a marginal curve without covariates, how Cox proportional hazards turns covariates into hazard ratios, and how accelerated failure-time models describe the same data on the log-time scale. Once we have a fitted model, how do we know whether it is any good?

That question is harder than it looks in the survival setting, because the quantity we want to score is the thing censoring hides. Suppose we maintain a fleet of 900 industrial units. For each unit we record the hours it ran, a few operating covariates, and whether we saw it fail or merely stopped watching when the study ended. A model now returns two kinds of output. It may return a risk score, a single number per unit that orders units from "fails soonest" to "lasts longest". Or it may return a whole survival function, a curve $\hat{S}_i(t)$ giving the probability that unit $i$ survives past any horizon $t$ we care about. These are different products, and they support different decisions: a risk score tells a planner which units to inspect first, while survival probabilities can inform replacement planning over a two-year horizon. If each initial unit is replaced once on its first failure, summing the model's two-year failure probabilities estimates the expected replacement count. A replacement budget also needs unit costs and a policy for uncertainty.

Censoring blocks both evaluations. If a unit leaves the study at month 6, we cannot ask whether it survived past month 8, so we generally cannot determine its squared error for an ordinary mean-squared-error total. A prediction of one-half is the exception: its squared error is one-quarter for either outcome. If we simply discard it, we throw away the fact that it survived at least to month 6. If we pretend it survived past month 8, we have invented evidence. Manipulating censored durations is worse still: treating them as observed failures asserts that a unit died at the moment we stopped writing down its status. The metrics below use the observed event status and follow-up time without assigning an unobserved failure time.

Two censoring-aware tools carry most of the weight. The concordance index asks whether the model orders failure times correctly, scoring only the pairs of units for which the data alone determine who failed first. The time-dependent Brier score asks whether the model's predicted probabilities are close to the observed outcomes at a chosen horizon, reweighting the units we can fully observe so that they stand in for the units we cannot. We will derive both, discover exactly what each can and cannot see, and then apply them to a held-out equipment cohort where Kaplan–Meier, Cox, and Weibull AFT predictions are compared with censoring weights estimated only on the training partition. The third piece, calibration, is what closes the loop: a model can rank units perfectly and still be badly wrong about the numbers we plan with.

Evaluation metrics that account for censoring let us compare models using the recorded follow-up. A health system triaging transplant candidates, a lender scoring loan prepayment, a maintenance team scheduling overhauls, and a reliability engineer forecasting warranty claims can all face the same observation problem: the outcome of interest is a time, and the observation window ends before every event is observed. The censoring model must be justified for the population scored. Concordance uses observed pairwise orderings. Inverse-probability-of-censoring weighting (IPCW) instead reweights observed contributions under assumptions about censoring and support. Their interpretation depends on those assumptions and on the evaluation horizon; three years of follow-up alone does not establish model quality.

The two families of metrics measure different things, and that separation is a practical advantage. Concordance depends only on the ranking induced by a risk score. It is invariant to any strictly increasing transformation of the score, so it works for Cox linear predictors, for negated AFT log-scale terms, for negated restricted mean survival times, and for the risk output of a random survival forest that never produces an interpretable linear form. The Brier score, by contrast, depends on the predicted probabilities themselves and measures their squared error. Reporting both separates ranking from probability accuracy. Brier error also depends on the outcome uncertainty and resolution, so it does not isolate calibration; the calibration plots below compare predicted and observed probabilities directly.

The Brier score uses an explicit horizon, and truncated IPCW concordance uses a cutoff for the earlier failure; Harrell's concordance here has no horizon parameter. These choices connect evaluation directly to decisions. The prediction we need at a 1,500-hour maintenance window is not the same object as the prediction we need for a five-year warranty reserve, and a horizon-indexed metric can be computed at exactly the horizon the decision uses. Report the scored cohort and evaluation window alongside each metric so that a reader can see which units and period the number describes.

Both metrics accept predictions from different model families. The dense concordance implementation below compares pairs, with quadratic time and memory in the evaluation sample size; the Brier calculation is a mean of per-subject contributions at each horizon. Estimating the censoring distribution adds a Kaplan–Meier fit. Scoring fixed predictions does not refit the survival model, while bootstrapping repeats the scoring calculation. Runtime therefore depends on sample size, horizon count, and the number of resamples.

Concordance does not measure probability calibration. Two models with exactly the same ranking of units have exactly the same concordance index, even if one predicts a survival probability of 0.9 for a unit whose true survival probability is 0.2. A concordance of 0.75 therefore says nothing about whether the predicted probabilities can be used in a budget. Given the supplied scalar scores, it does not identify the full shape of the predicted curves. If those scores encode median ordering, errors in tail probabilities can remain invisible. A change to a curve can still change concordance when it changes the chosen score's ordering. If a decision needs probabilities, concordance is a screening tool, not the answer.

Harrell's concordance is also sensitive to the censoring distribution. The set of usable pairs is determined by who was censored when, so studies of the same population can target different comparable-pair mixes when follow-up differs. Cross-study comparisons therefore need their follow-up patterns and target populations stated. The IPCW-corrected version of concordance targets a censoring-independent estimand, but it buys that independence with a modelling assumption: the censoring model must match the required independence structure, and censoring survival must be positive over the specified target window. The pooled weights here assume censoring independent of failure time and prediction covariates.

IPCW metrics inherit the same fragility. Inverse weights are unstable by construction: if $\hat{G}(t)$ approaches zero, the weight $1/\hat{G}(t)$ grows without bound and a single observation can dominate the average. Positivity failures are not hypothetical. In the six-unit teaching cohort below, the last unit is censored at the final follow-up time, which drives the estimated censoring survival to exactly zero, so the strict Brier estimator used here rejects that horizon and later ones. A larger sample cannot remove a population support failure. Large inverse weights can increase variance, and the estimated Brier score can exceed one in small samples even though the population quantity is bounded in $[0,1]$. The reason is that the IPCW estimator is an average of weighted squared errors rather than a convex combination: a single small $\hat{G}$ inflates one weight, so one unit's weighted squared error can exceed one and push the average above the $[0,1]$ bound that holds for the unweighted population quantity.

A Brier score at one horizon reduces event timing to a binary outcome. At 1,500 hours, a unit that fails at 1,600 hours and one observed running through 3,600 hours both have the survivor outcome; their errors can still differ if their predicted probabilities differ. If timing across a window matters, evaluate several horizons or integrate over a specified interval. This does not remove the censoring assumptions. Withdrawal related to impending failure can make the pooled censoring probabilities used here inappropriate, potentially biasing the weighted scores; a larger sample does not correct a misspecified observation model.

Evaluation starts by fixing exactly what we observe, because censoring determines which comparisons are even possible. For units $i = 1, \dots, n$ we observe a triple $(t_i, \delta_i, x_i)$, where:

• $T_i$ is the true failure time of unit $i$, unobserved when the unit is censored
• $C_i$ is the censoring time, the moment we stop observing the unit
• $t_i = \min(T_i, C_i)$ is the recorded follow-up time
• $\delta_i = \mathbf{1}\{T_i \le C_i\}$ is the event indicator, equal to 1 when we saw the failure and 0 when the unit was censored
• $x_i \in \mathbb{R}^p$ is a vector of covariates

The pair $(t_i, \delta_i)$ is the part we see for every unit. The true failure time $T_i$ is only partly known: it is pinned down when $\delta_i = 1$ and bounded below when $\delta_i = 0$.

Collecting the rows into a matrix $X \in \mathbb{R}^{n \times p}$ and the follow-up times into a vector $t \in \mathbb{R}^n$, an entire survival dataset is the pair $(t, \delta)$ plus $X$. Evaluation also needs the model's risk scores or survival probability forecasts, the chosen horizons or cutoff, and a censoring model where IPCW is used. The observed outcomes alone do not determine a model's evaluation score.

A survival model can return two distinct products, and the difference between them is the key to choosing an evaluation metric. A risk score is a single number $r_i \in \mathbb{R}$ attached to unit $i$, with the convention that larger means "fails sooner". A survival prediction is a function $\hat{S}_i(t) = \hat{P}(T_i > t \mid x_i)$ defined for every horizon $t \ge 0$. A risk score is a ranking device; a survival curve is a probability forecast.

The Cox proportional hazards model produces both. Its hazard specification is

$$h(t \mid x) = h_0(t)\, e^{x^\top \beta},$$

where:

• $h(t \mid x)$: the hazard, the instantaneous failure rate at time $t$ for a unit with covariates $x$
• $h_0(t)$: the unspecified baseline hazard, shared by all units
• $x \in \mathbb{R}^p$: the covariate vector for a unit
• $\beta \in \mathbb{R}^p$: the coefficient vector, one log hazard ratio per covariate

To turn a hazard into a survival function, we use the identity $S(t) = \exp\{-H(t)\}$, where $H(t) = \int_0^t h(s)\,ds$ is the cumulative hazard (the integral of the hazard from $0$ to $t$). That identity holds because $h(t) = -\frac{d}{dt}\log S(t)$, so integrating both sides from 0 to $t$ and exponentiating recovers $S$ (using $S(0)=1$). Applying it here, the covariate term factors out of the integral because it does not depend on time:

$$H(t \mid x) = \int_0^t h_0(s)\, e^{x^\top \beta}\, ds = e^{x^\top \beta} \int_0^t h_0(s)\,ds = e^{x^\top \beta} H_0(t).$$ $$S(t \mid x) = \exp\{-H_0(t) e^{x^\top \beta}\} = \Big(e^{-H_0(t)}\Big)^{e^{x^\top \beta}} = S_0(t)^{\,e^{x^\top \beta}}.$$

The linear predictor $\eta_i = x_i^\top \beta$ is the natural risk score for a Cox model, and in matrix form we write the whole vector of predictors at once as

$$\eta = X\beta \in \mathbb{R}^{n},$$

where:

• $\eta \in \mathbb{R}^n$: the vector of linear predictors, one per unit
• $X \in \mathbb{R}^{n \times p}$: the covariate matrix
• $\beta \in \mathbb{R}^p$: the coefficient vector
• $n$: the number of units
• $p$: the number of covariates

For finite linear predictors, $e^{\eta_i}$ multiplies the same baseline hazard for every unit, so a larger $\eta_i$ gives a larger hazard multiplier. Its survival curve is no higher and, in exact arithmetic, is strictly lower wherever $0<H_0(t)<\infty$. Curves can coincide before the first baseline-hazard jump or at an endpoint where the cumulative hazard diverges and survival reaches zero. The linear predictor itself supplies a horizon-independent risk ordering. For crossing curves, a chosen horizon or another specified curve functional determines the ranking.

Accelerated failure-time models are specified on the log-time scale instead:

$$\log T = \mu + x^\top \gamma + \sigma W,$$

where:

• $T$: the true failure time of a unit
• $\mu$: the intercept on the log-time scale
• $x \in \mathbb{R}^p$: the covariate vector for a unit
• $\gamma \in \mathbb{R}^p$: the coefficient vector on the log-time scale
• $\sigma > 0$: a scale parameter
• $W$: a standard Gumbel random variable for minima, independent of $x$, with $P(W > w) = \exp(-e^{w})$

Writing $\lambda(x) = e^{\mu + x^\top \gamma}$ and $\rho = 1/\sigma$, we can derive the survival function in a few steps. Starting from the definition and substituting,

$$P(T > t \mid x) = P\!\left(W > \frac{\log t - \mu - x^\top \gamma}{\sigma}\right) = \exp\!\left\{-\exp\!\left(\frac{\log t - \mu - x^\top \gamma}{\sigma}\right)\right\}.$$

The inner exponential simplifies because $\exp\{(\log t - \mu - x^\top\gamma)/\sigma\} = (t/\lambda(x))^{\rho}$, using $\rho = 1/\sigma$ and $\lambda(x) = e^{\mu + x^\top\gamma}$. Hence

$$S(t \mid x) = \exp\!\left\{-\left(\frac{t}{\lambda(x)}\right)^{\rho}\right\}.$$

For an AFT model the linear predictor $x^\top\gamma$ is a log-scale term, so a larger value means longer life and lower risk. The risk score must therefore be oriented in the opposite direction, for example $r_i = -x_i^\top \hat\gamma$, or equivalently $r_i = -\log \widehat{\text{median}}(T_i \mid x_i)$. This sign convention is not cosmetic: reversing it turns a model with concordance above 0.5 into one with concordance below 0.5 (specifically, $1 - C$ in the absence of ties), so a useful ranking is turned into a harmful one.

A risk score can be formed from a chosen functional of the survival curve, such as $-\hat S_i(\tau)$, a finite negative predicted median, or a negative restricted mean survival time. For a Cox model, $-\hat S_i(\tau)$ preserves the linear-predictor risk ranking in exact arithmetic when $0<\hat H_0(\tau)<\infty$; lower survival means greater risk. Numerical underflow can collapse distinct survival predictions to zero, so use $\eta_i$ directly for Cox concordance when it is available. Step-function medians can introduce ties or remain undefined, so these choices need not give identical concordance in finite samples. For crossing curves, the chosen horizon can also change the ordering.

The concordance index asks a question about pairs of units: when the data tell us that unit $i$ failed before unit $j$, does the model give $i$ the higher risk score? The whole subtlety is in the first clause, because censoring means the data often tell us nothing. The formula is therefore built around the idea of a comparable pair, a pair of units whose failure order is established by the observations themselves.

Consider a pair $(i, j)$ with $t_i < t_j$. If $\delta_i = 1$, then $T_i = t_i$ exactly, and since $t_j = \min(T_j, C_j) \le T_j$, we can chain the inequalities to get $T_i = t_i < t_j \le T_j$, establishing $T_i < T_j$. The ordering $T_i < T_j$ is therefore established by the data, and we call the pair comparable. Now suppose instead that $\delta_i = 0$. All we know is $T_i > t_i$. It is entirely possible that $T_i$ lies between $t_i$ and $t_j$, or beyond $t_j$; the data do not decide. The pair is not comparable and must be dropped. The same logic applies when $\delta_j = 1$ and $t_j < t_i$: then $j$ plays the role of the earlier, uncensored member and the pair is comparable with the roles reversed. Pairs where the earlier member is censored, and pairs where both members are censored, contribute nothing. The displayed rule deliberately excludes pairs with equal recorded times. Under our event-before-censoring convention, a failure and a censoring at the same recorded time can still reveal a strict failure order; that tied case is outside this rule.

Harrell's concordance index is a weighted fraction of comparable pairs that the model orders correctly, with a tie in the risk score counted as half credit:

$$\hat{C} = \frac{\displaystyle\sum_{i}\sum_{j} \mathbf{1}\{t_i < t_j,\ \delta_i = 1\}\Big(\mathbf{1}\{r_i > r_j\} + \tfrac{1}{2}\mathbf{1}\{r_i = r_j\}\Big)}{\displaystyle\sum_{i}\sum_{j} \mathbf{1}\{t_i < t_j,\ \delta_i = 1\}}.$$

where:

• $\hat{C}$: the estimated Harrell concordance index
• $r_i$: the risk score for unit $i$, with larger values meaning shorter predicted survival
• $t_i$: the recorded follow-up time of unit $i$
• $\delta_i$: the event indicator for unit $i$, equal to 1 when the failure was observed
• $\mathbf{1}\{\cdot\}$: the indicator function, equal to 1 when its argument is true and 0 otherwise

Reading the pieces: the outer sum runs over all ordered pairs; the indicator $\mathbf{1}\{t_i < t_j, \delta_i = 1\}$ selects usable pairs and acts as a switch that zeroes out everything else; the two inner indicators give one full point for a correct ordering and half a point for a tie; and the denominator counts how many usable pairs there were, which turns the sum into an average. The numerator is credit, the denominator is opportunity, and Harrell's index is the fraction of available credit relative to opportunity that the model converts. The ratio is defined only when at least one comparable pair exists; otherwise its denominator is zero and the helper raises an error.

Without censoring, the estimator targets the population concordance with half credit for tied risk scores. Under censoring, Harrell's statistic instead targets the mix of comparable pairs selected by this study's follow-up pattern.

$$C = P\big(r_I > r_J \mid T_I < T_J\big) + \tfrac12 P\big(r_I = r_J \mid T_I < T_J\big).$$

This target averages ranking credit over two independent units whose true failure times satisfy $T_I<T_J$. Censored observations reveal only some such pairs, which is why Harrell's estimate can depend on follow-up.

When no observation is censored and failure times are distinct, every pair contributes to survival concordance. This is related to ROC AUC through time-dependent discrimination measures, but it is not generally the ordinary binary AUC at a fixed horizon. For example, times $(1,2,3)$ and risk scores $(2,3,1)$ give concordance $2/3$ and a binary AUC of $1/2$ at horizon $1.5$.

Harrell's comparable-pair mix depends on censoring. Truncated IPCW concordance instead targets the ranking credit among true failure pairs with $T_I<T_J$ and $T_I<\tau^*$. With continuous censoring, Uno's estimator uses inverse-square censoring weights. The formula below includes the tie-aware product needed under our event-before-censoring convention:

$$\hat{C}_{\text{Uno}}(\tau^*) = \frac{\displaystyle\sum_i\sum_j \frac{\mathbf{1}\{t_i<t_j,\ t_i<\tau^*,\ \delta_i=1\}}{\hat{G}(t_i-)\hat{G}(t_i)}\Big(\mathbf{1}\{r_i>r_j\}+\tfrac12\mathbf{1}\{r_i=r_j\}\Big)}{\displaystyle\sum_i\sum_j \frac{\mathbf{1}\{t_i<t_j,\ t_i<\tau^*,\ \delta_i=1\}}{\hat{G}(t_i-)\hat{G}(t_i)}}.$$

where:

• $\hat{C}_{\text{Uno}}(\tau^*)$: truncated IPCW concordance, using Uno's inverse-square form when censoring is continuous
• $\tau^*$: a fixed upper limit on the earlier failure time, inside positive censoring support
• $\hat G(t_i-)$: the censoring survival immediately before the failure time
• $r_i$: the risk score for unit $i$, with larger values meaning shorter predicted survival
• $t_i$: the recorded follow-up time of unit $i$
• $\delta_i$: the event indicator for unit $i$, equal to 1 when the failure was observed
• $\hat{G}(u) = \hat{P}(C > u)$: the censoring survival estimated by reverse Kaplan–Meier; at a tied recorded time, original failures leave its risk set before the censoring jump
• $C$: the censoring time, the moment observation of a unit stops

Choose a fixed truncation horizon $\tau^*$ with positive censoring survival throughout the evaluation window, and include only earlier failures with $t_i<\tau^*$. For continuous censoring times, conditional on a qualifying latent pair with earlier failure time $T_i=u<T_j$, the probability of observing that pair is $G(u)^2$, which explains the inverse-square weight. Under our convention $\delta_i=\mathbf{1}\{T_i\le C_i\}$, censoring ties require the more precise product $G(t_i-)G(t_i)$: the failing unit needs $C_i\ge t_i$, while the comparison unit needs $C_j>t_i$. The pooled weights used here assume that $C$ is independent of the joint pair $(T,x)$.

Concordance only needs a ranking, so it can ignore the actual probability values. The Brier score cannot. At horizon $\tau$ it measures squared error on the probability scale:

$$\text{BS}(\tau) = \frac{1}{n}\sum_{i=1}^{n}\Big(Y_i(\tau) - \hat{S}_i(\tau)\Big)^2, \qquad Y_i(\tau) = \mathbf{1}\{T_i > \tau\},$$

where:

• $\text{BS}(\tau)$: the Brier score at horizon $\tau$, the mean squared error of the survival probability forecast
• $n$: the number of units in the evaluation sample
• $Y_i(\tau)$: the realised outcome for unit $i$, equal to 1 if unit $i$ survives past $\tau$ and 0 if it fails by then
• $\hat{S}_i(\tau)$: the model's predicted probability that unit $i$ survives past $\tau$
• $T_i$: the true failure time of unit $i$, which is only partly observed under censoring

$Y_i(\tau)$ is the realised outcome: 1 if unit $i$ is still running at $\tau$, 0 if it has failed by then. With fully observed outcomes this is just the mean squared error of a probability forecast, and it is minimised in expectation by the true conditional survival probability. This makes the Brier score a proper probability score. Its aggregate value does not identify calibration alone, so we also compare observed and predicted probabilities within groups.

Censoring hides $Y_i(\tau)$ for part of the sample. Under our event-before-censoring convention, four cases describe what is known:

• If $t_i\le\tau$ and $\delta_i=1$, the unit failed by $\tau$, so $Y_i(\tau)=0$ is known.
• If $t_i>\tau$, the unit was observed past $\tau$, so $Y_i(\tau)=1$ is known.
• If $t_i=\tau$ and $\delta_i=0$, then $C_i=\tau<T_i$, so $Y_i(\tau)=1$ is also known.
• If $t_i<\tau$ and $\delta_i=0$, the unit was censored before the horizon, so $Y_i(\tau)$ is unknown.

The naive fixes are to drop those units, score them as survivors, or score their censoring times as failures. These operations make different unsupported assumptions about outcomes that were not observed.

Inverse probability of censoring weighting divides each observed outcome contribution by its observation probability. Under censoring independent of failure time and prediction covariates, $P(\delta=1\mid T=u,x)=P(C\ge u)=G(u-)$ with $G(u)=P(C>u)$. The left limit matters when censoring and failure times can tie; with continuous censoring times it equals $G(u)$.

$$\mathbb{E}\!\left[\frac{\delta\,\mathbf{1}\{T\le\tau\}}{G(T-)}\right] = \mathbb{E}\!\left[\frac{\mathbf{1}\{T\le\tau\}}{G(T-)}\,\mathbb{E}[\delta\mid T]\right] = P(T\le\tau).$$

The $G(T-)$ in the denominator cancels the probability of observing a failure at $T$, so each observed failure represents $1/G(T-)$ units. For the survivor term, we deliberately select units observed strictly past $\tau$: $\{t>\tau\}=\{T>\tau,C>\tau\}$. Under the stated independence assumptions and $G(\tau)>0$, $\mathbb E[\mathbf1\{t>\tau\}/G(\tau)]=P(T>\tau)$ even if censoring has an atom at $\tau$. A unit censored exactly at $\tau$ has a known survivor outcome, but is excluded by this selection rule. An inclusive rule would instead select $\{t>\tau\}\cup\{t=\tau,\delta=0\}=\{T>\tau,C\ge\tau\}$ and divide by $G(\tau-)$; that alternative is not the estimator implemented below.

Applying both identities term by term gives the Graf estimator, the standard IPCW Brier score:

$$\widehat{\text{BS}}(\tau) = \frac{1}{n}\sum_{i=1}^{n}\left[\frac{\hat{S}_i(\tau)^2\;\mathbf{1}\{t_i \le \tau,\ \delta_i = 1\}}{\hat{G}(t_i-)} + \frac{\big(1 - \hat{S}_i(\tau)\big)^2\;\mathbf{1}\{t_i > \tau\}}{\hat{G}(\tau)}\right].$$

where:

• $\widehat{\text{BS}}(\tau)$: the estimated Brier score at horizon $\tau$
• $n$: the number of units in the evaluation sample
• $\hat{S}_i(\tau)$: the model's predicted survival probability for unit $i$ at horizon $\tau$
• $t_i$: the recorded follow-up time of unit $i$
• $\delta_i$: the event indicator for unit $i$, equal to 1 when the failure was observed
• $\hat{G}(u)$: the estimated survival function of the censoring distribution, evaluated at time $u$

The first term collects observed failures by $\tau$: their squared error is $\hat S_i(\tau)^2$, with weight $1/\hat G(t_i-)$. The second term selects units observed strictly past $\tau$: their squared error is $(1-\hat S_i(\tau))^2$, with weight $1/\hat G(\tau)$. Units censored at or before $\tau$ contribute zero to this particular estimator, although censoring exactly at $\tau$ reveals a survivor outcome. With true censoring probabilities, positive support, and the stated independence assumptions, these selected weighted contributions recover the population Brier error in expectation. For a fixed probability forecast and independent, identically distributed evaluation units, an oracle law of large numbers makes the true-weight average converge to that expectation. One sufficient condition for estimated weights to reach the same target is uniform consistency of the required censoring probabilities, including their left limits, with the true probabilities bounded away from zero over the evaluation window. Uniform control keeps the reciprocals controlled; pointwise consistency alone does not guarantee this. For a forecast fitted on independent training data, condition on the fit and target that fitted forecast's population Brier error. The identity is not a promise that a finite sample uses every known outcome or has no information loss.

For this strict Brier estimator, require $G(\tau)>0$ and $G(u-)>0$ at contributing failure times $u\le\tau$; the implementation also checks their estimated counterparts. Very small values can make inference unstable. For general covariate-dependent losses, the pooled Kaplan–Meier censoring weights used here are justified under the sufficient condition that $C$ is independent of the joint pair $(T,x)$. If failure and censoring are only conditionally independent given $x$, use a correctly specified conditional censoring model $\hat G(u\mid x_i)$ in each unit's weights instead. A training-based censoring model also needs to describe the evaluation population. When the horizon is not fixed in advance, the integrated Brier score summarises the path:

$$\widehat{\text{IBS}} = \frac{1}{\tau_{\max}}\int_0^{\tau_{\max}} \widehat{\text{BS}}(s)\,ds \;\approx\; \frac{1}{\tau_{\max}}\sum_{k=1}^{K-1}\frac{\widehat{\text{BS}}(s_k) + \widehat{\text{BS}}(s_{k+1})}{2}\,(s_{k+1} - s_k),$$

where:

• $\widehat{\text{IBS}}$: the estimated integrated Brier score over the full follow-up window
• $\tau_{\max}$: the largest horizon included in the integration, the upper end of the evaluation window
• $\widehat{\text{BS}}(s)$: the estimated IPCW Brier score at horizon $s$
• $s_k$: the $k$-th point of the integration grid, with $s_1=0<\dots<s_K=\tau_{\max}$; for a window $[a,b]$ such as the implementation's $[300,3000]$, integrate over that window and divide by $b-a$
• $K$: the number of grid points

This is the trapezoid rule applied to the grid. Reading the right-hand side: each trapezoid's area is its average height times its width, and dividing the total by $\tau_{\max}$ normalises the integral to the scale of a single Brier score. The factor $1/\tau_{\max}$ in front plays the role of the $1/\tau_{\max}$ in the integral definition $\widehat{\text{IBS}} = \frac{1}{\tau_{\max}}\int_0^{\tau_{\max}} \widehat{\text{BS}}(s)\,ds$, so the trapezoid approximation lands on the same scale.

• With finite nonnegative pair weights and a positive comparable-pair denominator, concordance lies in $[0,1]$. Conditional on those pairs and weights, an independent random ranking with exchangeable score pairs gives $0.5$ in expectation, including when tied scores receive half credit. In exact arithmetic, a strictly increasing transformation on the score's domain preserves its pairwise rankings and ties. Examples include positive rescaling, exponentiation over the real numbers, and a logarithm when every score is positive; nonpositive rescaling or a transformation that merges distinct scores need not preserve concordance.
• When at least one comparable pair contributes positive weight, a constant risk score gives concordance $0.5$ because every contributing pair receives half credit. This includes a marginal Kaplan–Meier baseline. With no contributing pair, Harrell or truncated IPCW concordance is undefined rather than $0.5$.
• Harrell's concordance depends on the censoring distribution through the set of comparable pairs. Uno's concordance targets a censoring-independent quantity but requires independent censoring and a positive censoring survival function.
• With the true censoring probabilities and the stated independence assumptions, the IPCW Brier score has the same expectation as the complete-outcome Brier score, minimised by the true conditional survival probability. Under the consistency and reciprocal-control conditions stated above, estimated weights recover this target asymptotically; they do not give an exact finite-sample unbiasedness guarantee.
• For a binary horizon outcome $Y\in\{0,1\}$ and probability forecast $F\in[0,1]$, let $m(F)=P(Y=1\mid F)$ and $p=P(Y=1)$. Here $Y=1$ means survival past the horizon, so $p$ is the marginal survival probability. The expected Brier score is $\mathbb E[(F-m(F))^2]-\operatorname{Var}(m(F))+p(1-p)$. The reliability term measures the gap between each forecast and the conditional probability at that forecast level; the resolution term measures how those conditional probabilities vary. The uncertainty term $p(1-p)$ is the marginal outcome variance. It is not a lower bound on prediction error, because informative forecasts can offset it through resolution. Exact forecast levels, or groups with a constant forecast, give this identity; coarse bins with varying forecasts require additional terms. This probability-score decomposition does not turn concordance into a calibration measure: concordance uses pairwise ranking credit rather than squared probability errors.
• In finite samples the IPCW Brier estimator can fall outside $[0,1]$, because a small $\hat{G}$ can make a single term large; small censoring survival can increase its variance, with the effect depending on the forecasts and event distribution.

We start with a cohort small enough to inspect by hand. Six units, followed for up to twelve months, with a risk score and a predicted survival probability at an eight-month horizon attached to each. The pairwise and weighted-error examples use this table and can be checked line by line. A separate illustration uses two explicitly specified Weibull curves to show how a survival probability becomes a ranking score and a squared error.

Each row of the cohort is a follow-up window that opens at time zero and closes either with a failure or with censoring. For the strictly unequal recorded times used by our comparable-pair rule, the pair is usable only if the earlier row closes with a failure: then its failure time is known to precede the other unit's true failure time. This six-unit table has no recorded-time ties.

A predicted survival curve plays two roles. Read vertically at a fixed horizon, its height orders units by risk. Read as a function, it supplies the probability that enters the Brier score. The next figure shows both readings on two illustrative Weibull curves with the same shape but different scales.

The Brier score needs more. If the high-risk unit failed before 1,500 hours, the model's squared error is $\hat{S}(\tau)^2$; if it survived, the error is $(1 - \hat{S}(\tau))^2$. The same prediction is excellent or terrible depending on an outcome we may never observe, which is exactly the situation IPCW weights are designed to handle.

Now we compute the per-unit contributions at $\tau=8$ months using a naive survivor rule and IPCW. For these untied teaching data, swapping event and censoring indicators in Kaplan–Meier estimates $G(u)=P(C>u)$. With failure/censoring ties and the convention $T\le C$ for an observed failure, reverse Kaplan–Meier must remove original failures from the censoring risk set before the censoring jump at that time, and failure weights use the left limit $\hat G(t_i-)$.

Unit C, censored at month 6, receives a naive contribution of $(1 - 0.70)^2 = 0.09$ because the naive rule assumes it survived past month 8. The model had predicted a comfortable 0.70 for a unit whose fate we never learn, and the naive score rewards it. Under IPCW, C contributes exactly nothing, and the units that carry information are upweighted: D's contribution rises from $0.5625$ to $0.75$ because $1/\hat{G}(8) = 1/0.75 \approx 1.3333$, and E and F rise by the same factor. Here the IPCW score is about 8.4 percent larger than the survivor-imputed score. The size and direction of the difference depend on the predictions and the censoring pattern.

We now work the same six units through the arithmetic by hand. The purpose is not to replace the code but to make every number in the code auditable. Once the formulas have been evaluated on a table this small, the vectorised implementation has a reference point we can trust.

List the units with their observed time, event indicator, and risk score: A$(3, 1, 0.90)$, B$(5, 1, 0.60)$, C$(6, 0, 0.70)$, D$(8, 1, 0.30)$, E$(9, 0, 0.40)$, F$(12, 0, 0.10)$. There are $\binom{6}{2} = 15$ unordered pairs, and we classify each one by asking whether the earlier member is an observed failure.

Begin with A. A failed at month 3, so every pair in which A is the earlier member is usable: A with B, C, D, E, and F. That is five pairs, and in every one of them $r_A = 0.90$ exceeds the other score. Five concordant pairs.

B fails at month 5, so the four pairs B vs C, B vs D, B vs E, B vs F are usable. Against C the model is wrong, since $0.60 < 0.70$, giving one discordant pair. Against D, E, F the scores are $0.30, 0.40, 0.10$, all below $0.60$, giving three concordant pairs. Running total: eight concordant, one discordant.

C is censored, so the three pairs C vs D, C vs E, C vs F are excluded. D fails at month 8, so D vs E and D vs F are usable. Against E, $0.30 < 0.40$ is discordant. Against F, $0.30 > 0.10$ is concordant. Running total: nine concordant, two discordant. The remaining pair, E vs F, has both members censored and is excluded.

That gives eleven usable pairs, of which nine are concordant and two discordant, so

$$\hat{C} = \frac{9 + \tfrac{1}{2}\cdot 0}{11} = \frac{9}{11} \approx 0.818.$$

Thus $\hat{C} \approx 0.818$: roughly four times in five, when the data tell us which of two units failed first, the model agrees. Note that the four excluded pairs are not a random sample of the fifteen: they all involve C or E, two of the three censored units, so any claim about the model's accuracy in the late-failure region rests on very little evidence.

For the teaching calculation we fix $\tau^*=10$ months, within positive censoring support. All eleven usable pairs have earlier failures before this cutoff. Their failure times do not tie censorings, so $\hat G(t_i-)=\hat G(t_i)$ and the product weight reduces to $\hat G(t_i)^{-2}$. The censoring survival function comes from the same Kaplan–Meier machinery applied to the censoring indicator:

$$\hat{G}(3) = 1, \quad \hat{G}(5) = 1, \quad \hat{G}(6) = \tfrac{3}{4}, \quad \hat{G}(8) = \tfrac{3}{4}, \quad \hat{G}(9) = \tfrac{3}{8}, \quad \hat{G}(12) = 0.$$

The step at month 6 comes from one censoring event among the four units still under observation, so the curve drops from 1 to $1\cdot(1 - 1/4) = 0.75$. At month 9 one of the two remaining units is censored, so it drops further to $0.75\cdot(1 - 1/2) = 0.375$. At month 12 the last unit is censored and the estimate reaches zero, so the strict Brier implementation used here rejects horizons at or beyond month 12. Censoring exactly at month 12 still establishes survival past that boundary; using that outcome would require the inclusive survivor selection and its matching left-limit censoring weight.

The pair weights are $\hat{G}(t_i)^{-2}$ applied to the earlier member of each pair. A's pairs get weight $1/1^2 = 1$ each, contributing $5$ to the numerator and $5$ to the denominator. B's pairs get $1/1^2 = 1$ each, contributing $3$ to the numerator (three concordant) and $4$ to the denominator (four usable pairs). D's pairs get $1/0.75^2 = 1.7778$ each, contributing $1.7778$ and $3.5556$. Therefore

$$\hat{C}_{\text{Uno}} = \frac{5 + 3 + 1.7778}{5 + 4 + 3.5556} = \frac{9.7778}{12.5556} \approx 0.779.$$

The IPCW version is lower because D's pairs, one of which is discordant, now carry extra weight relative to the others. The two numbers answer slightly different questions: Harrell targets the mix of pairs this study's censoring pattern allowed us to resolve, while truncated IPCW targets true failure pairs whose earlier failure precedes the specified cutoff, under the censoring assumptions.

Compute the Graf contributions in code so every displayed value comes from the data, then check them against the formulas.

Scoring the censored unit as a survivor gives the model unearned credit for an outcome nobody observed, and dropping the censored unit conditions the sample on its horizon outcome being observable, which changes the question being asked. Under the stated censoring assumptions, IPCW targets the complete-outcome Brier error without imputing the unknown outcomes; finite-sample estimation error remains.

The same two calculations, written as explicit loops over pairs, confirm the hand arithmetic and give us a reference to check the vectorised implementation against.

Three hand calculations, three confirmations. With the definitions validated on a cohort we can inspect, we can now scale up to a realistic sample where the models themselves must be fitted.

The six-unit cohort validates the arithmetic, but it cannot tell us how the evaluation behaves as a statistical estimator. We therefore simulate a realistic equipment cohort with a known generating mechanism, fit three models on a training partition, estimate the censoring weights on that same training partition, and score the fitted models on a test partition they never saw. The generating Weibull AFT model also has proportional hazards, so both fitted model families have appropriate population forms. Their finite-sample estimates and held-out metrics can still differ.

Roughly 45 percent of this cohort is censored. That overall fraction alone does not determine the weight stability or the bias of a naive rule; inspect the censoring survival at the scored horizons.

The split is part of the evaluation design, not an afterthought. We keep a training partition for fitting, a validation partition for any model-choice step, and a test partition reserved for a final, prespecified evaluation phase.