Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Evaluating Survival Predictions › Visualizing Survival Prediction Evaluation

Michael BrenndoerferOctober 27, 202612 min read

Machine Learning from ScratchAll chapters

Part 6 of 15Survival Analysis

Chapter 52Evaluating Survival Predictions › Visualizing Survival Prediction Evaluation

Reading section 5 of 10 · Visualizing Survival Prediction Evaluation · 12 minSection 5 of 10 · 12 min

Section 5 of 10 · 12 min read

Evaluating Survival Predictions › Visualizing Survival Prediction EvaluationSection view · Sections in this chapter5 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

We start with a cohort small enough to inspect by hand. Six units, followed for up to twelve months, with a risk score and a predicted survival probability at an eight-month horizon attached to each. The pairwise and weighted-error examples use this table and can be checked line by line. A separate illustration uses two explicitly specified Weibull curves to show how a survival probability becomes a ranking score and a squared error.

ConsoleOut [4]
CollapseExpandunit  time  event  risk  pred_surv_at_8
   A   3.0      1   0.9            0.55
   B   5.0      1   0.6            0.62
   C   6.0      0   0.7            0.70
   D   8.0      1   0.3            0.75
   E   9.0      0   0.4            0.80
   F  12.0      0   0.1            0.85

Comparing pairs on a timelineLink Copied

Each row of the cohort is a follow-up window that opens at time zero and closes either with a failure or with censoring. For the strictly unequal recorded times used by our comparable-pair rule, the pair is usable only if the earlier row closes with a failure: then its failure time is known to precede the other unit's true failure time. This six-unit table has no recorded-time ties.

VisualizationOut [6]
CollapseExpand

Six timelines show recorded durations and risk scores. A fails at month 3, D at month 8, and C is censored at month 6. Connectors distinguish the known A-D order from the unknown C-D order. — Follow-up windows for the six-unit cohort, with durations plotted on a shared time axis. Circles mark failures and crosses mark censoring. The teal connector links the usable pair A-D; the rose connector links C-D, whose failure order is unknown because C was censored first.

The figure makes the asymmetry of censoring concrete. Unit A fails at month 3 with the highest risk score in the cohort, so every pair in which A is the earlier member is usable, and every one of those pairs is concordant. Unit C is censored at month 6 with a score of 0.70, which is higher than D's score of 0.30. If we mistakenly treated C as a failure at month 6, we would count the pair (C, D) as concordant and give the model unsupported credit. The data do not support that inference: C might have failed at month 7 or at month 20. The observed-pair concordance formula excludes this pair through the indicator $\mathbf{1}\{t_i < t_j, \delta_i = 1\}$ .

The timeline follows individual units; @fig-pair-status shows all fifteen pair decisions together. Read either triangle of the symmetric matrix once: nine pairs earn concordance credit, two are discordant, and four have an unknown failure order. The two discordant pairs are B-C and D-E.

VisualizationOut [7]
CollapseExpand

A symmetric six-by-six matrix for units A through F. The 15 unique off-diagonal pairs contain nine C cells, two D cells, and four X cells, mirrored across a diagonal of dashes. — Pair status for all fifteen unordered pairs in the six-unit cohort. Nine pairs are concordant, two are discordant, and four are excluded because the earlier observed time is censored; each unordered pair is mirrored across the diagonal, and a dash marks the same unit.

From a survival curve to a risk score and an errorLink Copied

A predicted survival curve plays two roles. Read vertically at a fixed horizon, its height orders units by risk. Read as a function, it supplies the probability that enters the Brier score. The next figure shows both readings on two illustrative Weibull curves with the same shape but different scales.

PythonIn [8]
CollapseExpand# Two Weibull survival curves with equal shape and different scales.
rho_ill = 1.6
lam_low, lam_high = 2200.0, 900.0
tau_ill = 1500.0
t_grid = np.linspace(0.0, 3600.0, 400)

surv_low = np.exp(-((t_grid / lam_low) ** rho_ill))
surv_high = np.exp(-((t_grid / lam_high) ** rho_ill))
s_high_tau = float(np.exp(-((tau_ill / lam_high) ** rho_ill)))
s_low_tau = float(np.exp(-((tau_ill / lam_low) ** rho_ill)))

score_failure_target = (tau_ill, s_high_tau / 2.0)
score_survivor_target = (tau_ill, (1.0 + s_high_tau) / 2.0)
score_failure_offset = (44, 28)
score_survivor_offset = (44, 28)
score_failure_segment = ([tau_ill, tau_ill], [0.0, s_high_tau])
score_survivor_segment = ([tau_ill, tau_ill], [s_high_tau, 1.0])

VisualizationOut [9]
CollapseExpand

Two decreasing survival curves, one steep and one shallow, with a vertical dotted line at 1500 hours and thick vertical segments marking the distance from the curve to zero and to one. — Survival curves for a high-risk unit with a short scale and a low-risk unit with a long scale. The high-risk curve falls below the low-risk curve at every positive horizon, so the model ranks the two units consistently. At a fixed horizon, the vertical distances from each curve to 0 and to 1 are the residual magnitudes whose squares enter the Brier score.

The high-risk unit has $\hat{S}(\tau) = e^{-(1500/900)^{1.6}}$ and the low-risk unit has $\hat{S}(\tau) = e^{-(1500/2200)^{1.6}}$ at $\tau = 1500$ hours, computed above into s_high_tau and s_low_tau, so the model already ranks them: the steep curve sits below the shallow one at every positive horizon; both equal one at time zero. That ranking is all concordance needs, and it is why concordance can be computed from a single risk score per unit.

ConsoleOut [10]
CollapseExpandhigh-risk S_hat(tau) = 0.104
low-risk  S_hat(tau) = 0.582

The Brier score needs more. If the high-risk unit failed before 1,500 hours, the model's squared error is $\hat{S}(\tau)^2$ ; if it survived, the error is $(1 - \hat{S}(\tau))^2$ . The same prediction is excellent or terrible depending on an outcome we may never observe, which is exactly the situation IPCW weights are designed to handle.

How censoring weights reweight the Brier scoreLink Copied

Now we compute the per-unit contributions at $\tau=8$ months using a naive survivor rule and IPCW. For these untied teaching data, swapping event and censoring indicators in Kaplan–Meier estimates $G(u)=P(C>u)$ . With failure/censoring ties and the convention $T\le C$ for an observed failure, reverse Kaplan–Meier must remove original failures from the censoring risk set before the censoring jump at that time, and failure weights use the left limit $\hat G(t_i-)$ .

PythonIn [11]
CollapseExpand# Reuse censoring_curve and step_value from the helper cell above.
tiny_times = tiny["time"].to_numpy(dtype=float)
tiny_event = tiny["event"].to_numpy(dtype=float)
tiny_risk = tiny["risk"].to_numpy(dtype=float)
tiny_pred = tiny["pred_surv_at_8"].to_numpy(dtype=float)

tiny_g_times, tiny_g = censoring_curve(tiny_times, tiny_event)
g_at_t = step_value(tiny_times, tiny_g_times, tiny_g)
tau_tiny = 8.0
g_tau = float(step_value(tau_tiny, tiny_g_times, tiny_g))

failed_by_tau = (tiny_times <= tau_tiny) & (tiny_event == 1)
alive_at_tau = tiny_times > tau_tiny

g_before_t = step_value(tiny_times, tiny_g_times, tiny_g, side="left")
assert g_tau > 0 and np.all(g_before_t[failed_by_tau] > 0)
contrib_ipcw = np.zeros_like(tiny_times)
contrib_ipcw[failed_by_tau] = (
    tiny_pred[failed_by_tau] ** 2 / g_before_t[failed_by_tau]
)
contrib_ipcw[alive_at_tau] = (1.0 - tiny_pred[alive_at_tau]) ** 2 / g_tau
contrib_naive = np.where(failed_by_tau, tiny_pred**2, (1.0 - tiny_pred) ** 2)

ConsoleOut [12]
CollapseExpandunit  time  event  pred_surv_at_8  g_hat  naive   ipcw
   A   3.0      1            0.55  1.000 0.3025 0.3025
   B   5.0      1            0.62  1.000 0.3844 0.3844
   C   6.0      0            0.70  0.750 0.0900 0.0000
   D   8.0      1            0.75  0.750 0.5625 0.7500
   E   9.0      0            0.80  0.375 0.0400 0.0533
   F  12.0      0            0.85  0.000 0.0225 0.0300

G_hat(tau = 8) = 0.750
naive Brier at tau = 0.2337
IPCW  Brier at tau = 0.2534

VisualizationOut [14]
CollapseExpand

Grouped bar chart with six pairs of bars, comparing naive and IPCW contributions per unit; the third unit has a positive naive bar and a zero IPCW bar, while the fourth, fifth and sixth units have larger IPCW bars. — Per-unit contributions to the Brier score at tau = 8 months, comparing the naive survivor rule with IPCW. The naive rule scores the censored unit C as though it had survived, which hands the model a small error it never had to earn. The IPCW rule gives C zero contribution and reweights the selected observed contributions by $1/\hat{G}$, leaving terms whose relevant $\hat{G}=1$ unchanged.

Unit C, censored at month 6, receives a naive contribution of $(1 - 0.70)^2 = 0.09$ because the naive rule assumes it survived past month 8. The model had predicted a comfortable 0.70 for a unit whose fate we never learn, and the naive score rewards it. Under IPCW, C contributes exactly nothing, and the units that carry information are upweighted: D's contribution rises from $0.5625$ to $0.75$ because $1/\hat{G}(8) = 1/0.75 \approx 1.3333$ , and E and F rise by the same factor. Here the IPCW score is about 8.4 percent larger than the survivor-imputed score. The size and direction of the difference depend on the predictions and the censoring pattern.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Comparing pairs on a timelineLink Copied

From a survival curve to a risk score and an errorLink Copied

How censoring weights reweight the Brier scoreLink Copied

Comments

Add to the conversation

Stay up to date