Evaluating Survival Predictions › Formula

Evaluation starts by fixing exactly what we observe, because censoring determines which comparisons are even possible. For units $i = 1, \dots, n$ we observe a triple $(t_i, \delta_i, x_i)$, where:

• $T_i$ is the true failure time of unit $i$, unobserved when the unit is censored
• $C_i$ is the censoring time, the moment we stop observing the unit
• $t_i = \min(T_i, C_i)$ is the recorded follow-up time
• $\delta_i = \mathbf{1}\{T_i \le C_i\}$ is the event indicator, equal to 1 when we saw the failure and 0 when the unit was censored
• $x_i \in \mathbb{R}^p$ is a vector of covariates

The pair $(t_i, \delta_i)$ is the part we see for every unit. The true failure time $T_i$ is only partly known: it is pinned down when $\delta_i = 1$ and bounded below when $\delta_i = 0$.

Collecting the rows into a matrix $X \in \mathbb{R}^{n \times p}$ and the follow-up times into a vector $t \in \mathbb{R}^n$, an entire survival dataset is the pair $(t, \delta)$ plus $X$. Evaluation also needs the model's risk scores or survival probability forecasts, the chosen horizons or cutoff, and a censoring model where IPCW is used. The observed outcomes alone do not determine a model's evaluation score.

A survival model can return two distinct products, and the difference between them is the key to choosing an evaluation metric. A risk score is a single number $r_i \in \mathbb{R}$ attached to unit $i$, with the convention that larger means "fails sooner". A survival prediction is a function $\hat{S}_i(t) = \hat{P}(T_i > t \mid x_i)$ defined for every horizon $t \ge 0$. A risk score is a ranking device; a survival curve is a probability forecast.

The Cox proportional hazards model produces both. Its hazard specification is

$$h(t \mid x) = h_0(t)\, e^{x^\top \beta},$$

where:

• $h(t \mid x)$: the hazard, the instantaneous failure rate at time $t$ for a unit with covariates $x$
• $h_0(t)$: the unspecified baseline hazard, shared by all units
• $x \in \mathbb{R}^p$: the covariate vector for a unit
• $\beta \in \mathbb{R}^p$: the coefficient vector, one log hazard ratio per covariate

To turn a hazard into a survival function, we use the identity $S(t) = \exp\{-H(t)\}$, where $H(t) = \int_0^t h(s)\,ds$ is the cumulative hazard (the integral of the hazard from $0$ to $t$). That identity holds because $h(t) = -\frac{d}{dt}\log S(t)$, so integrating both sides from 0 to $t$ and exponentiating recovers $S$ (using $S(0)=1$). Applying it here, the covariate term factors out of the integral because it does not depend on time:

$$H(t \mid x) = \int_0^t h_0(s)\, e^{x^\top \beta}\, ds = e^{x^\top \beta} \int_0^t h_0(s)\,ds = e^{x^\top \beta} H_0(t).$$ $$S(t \mid x) = \exp\{-H_0(t) e^{x^\top \beta}\} = \Big(e^{-H_0(t)}\Big)^{e^{x^\top \beta}} = S_0(t)^{\,e^{x^\top \beta}}.$$

The linear predictor $\eta_i = x_i^\top \beta$ is the natural risk score for a Cox model, and in matrix form we write the whole vector of predictors at once as

$$\eta = X\beta \in \mathbb{R}^{n},$$

where:

• $\eta \in \mathbb{R}^n$: the vector of linear predictors, one per unit
• $X \in \mathbb{R}^{n \times p}$: the covariate matrix
• $\beta \in \mathbb{R}^p$: the coefficient vector
• $n$: the number of units
• $p$: the number of covariates

For finite linear predictors, $e^{\eta_i}$ multiplies the same baseline hazard for every unit, so a larger $\eta_i$ gives a larger hazard multiplier. Its survival curve is no higher and, in exact arithmetic, is strictly lower wherever $0<H_0(t)<\infty$. Curves can coincide before the first baseline-hazard jump or at an endpoint where the cumulative hazard diverges and survival reaches zero. The linear predictor itself supplies a horizon-independent risk ordering. For crossing curves, a chosen horizon or another specified curve functional determines the ranking.

Accelerated failure-time models are specified on the log-time scale instead:

$$\log T = \mu + x^\top \gamma + \sigma W,$$

where:

• $T$: the true failure time of a unit
• $\mu$: the intercept on the log-time scale
• $x \in \mathbb{R}^p$: the covariate vector for a unit
• $\gamma \in \mathbb{R}^p$: the coefficient vector on the log-time scale
• $\sigma > 0$: a scale parameter
• $W$: a standard Gumbel random variable for minima, independent of $x$, with $P(W > w) = \exp(-e^{w})$

Writing $\lambda(x) = e^{\mu + x^\top \gamma}$ and $\rho = 1/\sigma$, we can derive the survival function in a few steps. Starting from the definition and substituting,

$$P(T > t \mid x) = P\!\left(W > \frac{\log t - \mu - x^\top \gamma}{\sigma}\right) = \exp\!\left\{-\exp\!\left(\frac{\log t - \mu - x^\top \gamma}{\sigma}\right)\right\}.$$

The inner exponential simplifies because $\exp\{(\log t - \mu - x^\top\gamma)/\sigma\} = (t/\lambda(x))^{\rho}$, using $\rho = 1/\sigma$ and $\lambda(x) = e^{\mu + x^\top\gamma}$. Hence

$$S(t \mid x) = \exp\!\left\{-\left(\frac{t}{\lambda(x)}\right)^{\rho}\right\}.$$

For an AFT model the linear predictor $x^\top\gamma$ is a log-scale term, so a larger value means longer life and lower risk. The risk score must therefore be oriented in the opposite direction, for example $r_i = -x_i^\top \hat\gamma$, or equivalently $r_i = -\log \widehat{\text{median}}(T_i \mid x_i)$. This sign convention is not cosmetic: reversing it turns a model with concordance above 0.5 into one with concordance below 0.5 (specifically, $1 - C$ in the absence of ties), so a useful ranking is turned into a harmful one.

A risk score can be formed from a chosen functional of the survival curve, such as $-\hat S_i(\tau)$, a finite negative predicted median, or a negative restricted mean survival time. For a Cox model, $-\hat S_i(\tau)$ preserves the linear-predictor risk ranking in exact arithmetic when $0<\hat H_0(\tau)<\infty$; lower survival means greater risk. Numerical underflow can collapse distinct survival predictions to zero, so use $\eta_i$ directly for Cox concordance when it is available. Step-function medians can introduce ties or remain undefined, so these choices need not give identical concordance in finite samples. For crossing curves, the chosen horizon can also change the ordering.

The concordance index asks a question about pairs of units: when the data tell us that unit $i$ failed before unit $j$, does the model give $i$ the higher risk score? The whole subtlety is in the first clause, because censoring means the data often tell us nothing. The formula is therefore built around the idea of a comparable pair, a pair of units whose failure order is established by the observations themselves.

Consider a pair $(i, j)$ with $t_i < t_j$. If $\delta_i = 1$, then $T_i = t_i$ exactly, and since $t_j = \min(T_j, C_j) \le T_j$, we can chain the inequalities to get $T_i = t_i < t_j \le T_j$, establishing $T_i < T_j$. The ordering $T_i < T_j$ is therefore established by the data, and we call the pair comparable. Now suppose instead that $\delta_i = 0$. All we know is $T_i > t_i$. It is entirely possible that $T_i$ lies between $t_i$ and $t_j$, or beyond $t_j$; the data do not decide. The pair is not comparable and must be dropped. The same logic applies when $\delta_j = 1$ and $t_j < t_i$: then $j$ plays the role of the earlier, uncensored member and the pair is comparable with the roles reversed. Pairs where the earlier member is censored, and pairs where both members are censored, contribute nothing. The displayed rule deliberately excludes pairs with equal recorded times. Under our event-before-censoring convention, a failure and a censoring at the same recorded time can still reveal a strict failure order; that tied case is outside this rule.

Harrell's concordance index is a weighted fraction of comparable pairs that the model orders correctly, with a tie in the risk score counted as half credit:

$$\hat{C} = \frac{\displaystyle\sum_{i}\sum_{j} \mathbf{1}\{t_i < t_j,\ \delta_i = 1\}\Big(\mathbf{1}\{r_i > r_j\} + \tfrac{1}{2}\mathbf{1}\{r_i = r_j\}\Big)}{\displaystyle\sum_{i}\sum_{j} \mathbf{1}\{t_i < t_j,\ \delta_i = 1\}}.$$

where:

• $\hat{C}$: the estimated Harrell concordance index
• $r_i$: the risk score for unit $i$, with larger values meaning shorter predicted survival
• $t_i$: the recorded follow-up time of unit $i$
• $\delta_i$: the event indicator for unit $i$, equal to 1 when the failure was observed
• $\mathbf{1}\{\cdot\}$: the indicator function, equal to 1 when its argument is true and 0 otherwise

Reading the pieces: the outer sum runs over all ordered pairs; the indicator $\mathbf{1}\{t_i < t_j, \delta_i = 1\}$ selects usable pairs and acts as a switch that zeroes out everything else; the two inner indicators give one full point for a correct ordering and half a point for a tie; and the denominator counts how many usable pairs there were, which turns the sum into an average. The numerator is credit, the denominator is opportunity, and Harrell's index is the fraction of available credit relative to opportunity that the model converts. The ratio is defined only when at least one comparable pair exists; otherwise its denominator is zero and the helper raises an error.

Without censoring, the estimator targets the population concordance with half credit for tied risk scores. Under censoring, Harrell's statistic instead targets the mix of comparable pairs selected by this study's follow-up pattern.

$$C = P\big(r_I > r_J \mid T_I < T_J\big) + \tfrac12 P\big(r_I = r_J \mid T_I < T_J\big).$$

This target averages ranking credit over two independent units whose true failure times satisfy $T_I<T_J$. Censored observations reveal only some such pairs, which is why Harrell's estimate can depend on follow-up.

When no observation is censored and failure times are distinct, every pair contributes to survival concordance. This is related to ROC AUC through time-dependent discrimination measures, but it is not generally the ordinary binary AUC at a fixed horizon. For example, times $(1,2,3)$ and risk scores $(2,3,1)$ give concordance $2/3$ and a binary AUC of $1/2$ at horizon $1.5$.

Harrell's comparable-pair mix depends on censoring. Truncated IPCW concordance instead targets the ranking credit among true failure pairs with $T_I<T_J$ and $T_I<\tau^*$. With continuous censoring, Uno's estimator uses inverse-square censoring weights. The formula below includes the tie-aware product needed under our event-before-censoring convention:

$$\hat{C}_{\text{Uno}}(\tau^*) = \frac{\displaystyle\sum_i\sum_j \frac{\mathbf{1}\{t_i<t_j,\ t_i<\tau^*,\ \delta_i=1\}}{\hat{G}(t_i-)\hat{G}(t_i)}\Big(\mathbf{1}\{r_i>r_j\}+\tfrac12\mathbf{1}\{r_i=r_j\}\Big)}{\displaystyle\sum_i\sum_j \frac{\mathbf{1}\{t_i<t_j,\ t_i<\tau^*,\ \delta_i=1\}}{\hat{G}(t_i-)\hat{G}(t_i)}}.$$

where:

• $\hat{C}_{\text{Uno}}(\tau^*)$: truncated IPCW concordance, using Uno's inverse-square form when censoring is continuous
• $\tau^*$: a fixed upper limit on the earlier failure time, inside positive censoring support
• $\hat G(t_i-)$: the censoring survival immediately before the failure time
• $r_i$: the risk score for unit $i$, with larger values meaning shorter predicted survival
• $t_i$: the recorded follow-up time of unit $i$
• $\delta_i$: the event indicator for unit $i$, equal to 1 when the failure was observed
• $\hat{G}(u) = \hat{P}(C > u)$: the censoring survival estimated by reverse Kaplan–Meier; at a tied recorded time, original failures leave its risk set before the censoring jump
• $C$: the censoring time, the moment observation of a unit stops

Choose a fixed truncation horizon $\tau^*$ with positive censoring survival throughout the evaluation window, and include only earlier failures with $t_i<\tau^*$. For continuous censoring times, conditional on a qualifying latent pair with earlier failure time $T_i=u<T_j$, the probability of observing that pair is $G(u)^2$, which explains the inverse-square weight. Under our convention $\delta_i=\mathbf{1}\{T_i\le C_i\}$, censoring ties require the more precise product $G(t_i-)G(t_i)$: the failing unit needs $C_i\ge t_i$, while the comparison unit needs $C_j>t_i$. The pooled weights used here assume that $C$ is independent of the joint pair $(T,x)$.

Concordance only needs a ranking, so it can ignore the actual probability values. The Brier score cannot. At horizon $\tau$ it measures squared error on the probability scale:

$$\text{BS}(\tau) = \frac{1}{n}\sum_{i=1}^{n}\Big(Y_i(\tau) - \hat{S}_i(\tau)\Big)^2, \qquad Y_i(\tau) = \mathbf{1}\{T_i > \tau\},$$

where:

• $\text{BS}(\tau)$: the Brier score at horizon $\tau$, the mean squared error of the survival probability forecast
• $n$: the number of units in the evaluation sample
• $Y_i(\tau)$: the realised outcome for unit $i$, equal to 1 if unit $i$ survives past $\tau$ and 0 if it fails by then
• $\hat{S}_i(\tau)$: the model's predicted probability that unit $i$ survives past $\tau$
• $T_i$: the true failure time of unit $i$, which is only partly observed under censoring

$Y_i(\tau)$ is the realised outcome: 1 if unit $i$ is still running at $\tau$, 0 if it has failed by then. With fully observed outcomes this is just the mean squared error of a probability forecast, and it is minimised in expectation by the true conditional survival probability. This makes the Brier score a proper probability score. Its aggregate value does not identify calibration alone, so we also compare observed and predicted probabilities within groups.

Censoring hides $Y_i(\tau)$ for part of the sample. Under our event-before-censoring convention, four cases describe what is known:

• If $t_i\le\tau$ and $\delta_i=1$, the unit failed by $\tau$, so $Y_i(\tau)=0$ is known.
• If $t_i>\tau$, the unit was observed past $\tau$, so $Y_i(\tau)=1$ is known.
• If $t_i=\tau$ and $\delta_i=0$, then $C_i=\tau<T_i$, so $Y_i(\tau)=1$ is also known.
• If $t_i<\tau$ and $\delta_i=0$, the unit was censored before the horizon, so $Y_i(\tau)$ is unknown.

The naive fixes are to drop those units, score them as survivors, or score their censoring times as failures. These operations make different unsupported assumptions about outcomes that were not observed.

Inverse probability of censoring weighting divides each observed outcome contribution by its observation probability. Under censoring independent of failure time and prediction covariates, $P(\delta=1\mid T=u,x)=P(C\ge u)=G(u-)$ with $G(u)=P(C>u)$. The left limit matters when censoring and failure times can tie; with continuous censoring times it equals $G(u)$.

$$\mathbb{E}\!\left[\frac{\delta\,\mathbf{1}\{T\le\tau\}}{G(T-)}\right] = \mathbb{E}\!\left[\frac{\mathbf{1}\{T\le\tau\}}{G(T-)}\,\mathbb{E}[\delta\mid T]\right] = P(T\le\tau).$$

The $G(T-)$ in the denominator cancels the probability of observing a failure at $T$, so each observed failure represents $1/G(T-)$ units. For the survivor term, we deliberately select units observed strictly past $\tau$: $\{t>\tau\}=\{T>\tau,C>\tau\}$. Under the stated independence assumptions and $G(\tau)>0$, $\mathbb E[\mathbf1\{t>\tau\}/G(\tau)]=P(T>\tau)$ even if censoring has an atom at $\tau$. A unit censored exactly at $\tau$ has a known survivor outcome, but is excluded by this selection rule. An inclusive rule would instead select $\{t>\tau\}\cup\{t=\tau,\delta=0\}=\{T>\tau,C\ge\tau\}$ and divide by $G(\tau-)$; that alternative is not the estimator implemented below.

Applying both identities term by term gives the Graf estimator, the standard IPCW Brier score:

$$\widehat{\text{BS}}(\tau) = \frac{1}{n}\sum_{i=1}^{n}\left[\frac{\hat{S}_i(\tau)^2\;\mathbf{1}\{t_i \le \tau,\ \delta_i = 1\}}{\hat{G}(t_i-)} + \frac{\big(1 - \hat{S}_i(\tau)\big)^2\;\mathbf{1}\{t_i > \tau\}}{\hat{G}(\tau)}\right].$$

where:

• $\widehat{\text{BS}}(\tau)$: the estimated Brier score at horizon $\tau$
• $n$: the number of units in the evaluation sample
• $\hat{S}_i(\tau)$: the model's predicted survival probability for unit $i$ at horizon $\tau$
• $t_i$: the recorded follow-up time of unit $i$
• $\delta_i$: the event indicator for unit $i$, equal to 1 when the failure was observed
• $\hat{G}(u)$: the estimated survival function of the censoring distribution, evaluated at time $u$

The first term collects observed failures by $\tau$: their squared error is $\hat S_i(\tau)^2$, with weight $1/\hat G(t_i-)$. The second term selects units observed strictly past $\tau$: their squared error is $(1-\hat S_i(\tau))^2$, with weight $1/\hat G(\tau)$. Units censored at or before $\tau$ contribute zero to this particular estimator, although censoring exactly at $\tau$ reveals a survivor outcome. With true censoring probabilities, positive support, and the stated independence assumptions, these selected weighted contributions recover the population Brier error in expectation. For a fixed probability forecast and independent, identically distributed evaluation units, an oracle law of large numbers makes the true-weight average converge to that expectation. One sufficient condition for estimated weights to reach the same target is uniform consistency of the required censoring probabilities, including their left limits, with the true probabilities bounded away from zero over the evaluation window. Uniform control keeps the reciprocals controlled; pointwise consistency alone does not guarantee this. For a forecast fitted on independent training data, condition on the fit and target that fitted forecast's population Brier error. The identity is not a promise that a finite sample uses every known outcome or has no information loss.

For this strict Brier estimator, require $G(\tau)>0$ and $G(u-)>0$ at contributing failure times $u\le\tau$; the implementation also checks their estimated counterparts. Very small values can make inference unstable. For general covariate-dependent losses, the pooled Kaplan–Meier censoring weights used here are justified under the sufficient condition that $C$ is independent of the joint pair $(T,x)$. If failure and censoring are only conditionally independent given $x$, use a correctly specified conditional censoring model $\hat G(u\mid x_i)$ in each unit's weights instead. A training-based censoring model also needs to describe the evaluation population. When the horizon is not fixed in advance, the integrated Brier score summarises the path:

$$\widehat{\text{IBS}} = \frac{1}{\tau_{\max}}\int_0^{\tau_{\max}} \widehat{\text{BS}}(s)\,ds \;\approx\; \frac{1}{\tau_{\max}}\sum_{k=1}^{K-1}\frac{\widehat{\text{BS}}(s_k) + \widehat{\text{BS}}(s_{k+1})}{2}\,(s_{k+1} - s_k),$$

where:

• $\widehat{\text{IBS}}$: the estimated integrated Brier score over the full follow-up window
• $\tau_{\max}$: the largest horizon included in the integration, the upper end of the evaluation window
• $\widehat{\text{BS}}(s)$: the estimated IPCW Brier score at horizon $s$
• $s_k$: the $k$-th point of the integration grid, with $s_1=0<\dots<s_K=\tau_{\max}$; for a window $[a,b]$ such as the implementation's $[300,3000]$, integrate over that window and divide by $b-a$
• $K$: the number of grid points

This is the trapezoid rule applied to the grid. Reading the right-hand side: each trapezoid's area is its average height times its width, and dividing the total by $\tau_{\max}$ normalises the integral to the scale of a single Brier score. The factor $1/\tau_{\max}$ in front plays the role of the $1/\tau_{\max}$ in the integral definition $\widehat{\text{IBS}} = \frac{1}{\tau_{\max}}\int_0^{\tau_{\max}} \widehat{\text{BS}}(s)\,ds$, so the trapezoid approximation lands on the same scale.

• With finite nonnegative pair weights and a positive comparable-pair denominator, concordance lies in $[0,1]$. Conditional on those pairs and weights, an independent random ranking with exchangeable score pairs gives $0.5$ in expectation, including when tied scores receive half credit. In exact arithmetic, a strictly increasing transformation on the score's domain preserves its pairwise rankings and ties. Examples include positive rescaling, exponentiation over the real numbers, and a logarithm when every score is positive; nonpositive rescaling or a transformation that merges distinct scores need not preserve concordance.
• When at least one comparable pair contributes positive weight, a constant risk score gives concordance $0.5$ because every contributing pair receives half credit. This includes a marginal Kaplan–Meier baseline. With no contributing pair, Harrell or truncated IPCW concordance is undefined rather than $0.5$.
• Harrell's concordance depends on the censoring distribution through the set of comparable pairs. Uno's concordance targets a censoring-independent quantity but requires independent censoring and a positive censoring survival function.
• With the true censoring probabilities and the stated independence assumptions, the IPCW Brier score has the same expectation as the complete-outcome Brier score, minimised by the true conditional survival probability. Under the consistency and reciprocal-control conditions stated above, estimated weights recover this target asymptotically; they do not give an exact finite-sample unbiasedness guarantee.
• For a binary horizon outcome $Y\in\{0,1\}$ and probability forecast $F\in[0,1]$, let $m(F)=P(Y=1\mid F)$ and $p=P(Y=1)$. Here $Y=1$ means survival past the horizon, so $p$ is the marginal survival probability. The expected Brier score is $\mathbb E[(F-m(F))^2]-\operatorname{Var}(m(F))+p(1-p)$. The reliability term measures the gap between each forecast and the conditional probability at that forecast level; the resolution term measures how those conditional probabilities vary. The uncertainty term $p(1-p)$ is the marginal outcome variance. It is not a lower bound on prediction error, because informative forecasts can offset it through resolution. Exact forecast levels, or groups with a constant forecast, give this identity; coarse bins with varying forecasts require additional terms. This probability-score decomposition does not turn concordance into a calibration measure: concordance uses pairwise ranking credit rather than squared probability errors.
• In finite samples the IPCW Brier estimator can fall outside $[0,1]$, because a small $\hat{G}$ can make a single term large; small censoring survival can increase its variance, with the effect depending on the forecasts and event distribution.