Michael Brenndoerfer | Data & AI, Private Equity, Technology

Michael Brenndoerfer

Time-to-Event Data, Censoring, and Hazard Functions › Example

Michael BrenndoerferOctober 25, 20266 min read

Machine Learning from ScratchAll chapters

Part 6 of 15Survival Analysis

Chapter 48Time-to-Event Data, Censoring, and Hazard Functions › Example

Reading section 6 of 10 · Example · 6 minSection 6 of 10 · 6 min

Section 6 of 10 · 6 min read

Time-to-Event Data, Censoring, and Hazard Functions › ExampleSection view · Sections in this chapter6 / 10 ↓

Sections in this chapter10

Reading tools Definitions, links & notes

Choose your expertise level to adjust how many terms are explained. Beginners see more tooltips, experts see fewer to maintain reading flow. Hover over underlined terms for instant definitions.

Article links

Make inline references clickable

Notes & comments

Notes & commentsLeave notes and join the discussionSelection toolbarShow comment actions when selecting text. Turn off for the browser menu only.Comment indicatorsShow small counts beside annotated passagesPassage highlightsGently shade annotated text selections

To see the likelihood in action, let us work through a small hand-computable example. Suppose we observe six subjects, each of whose observed times and event indicators are as follows.

Observed times and event indicators for six subjects.
Subject $i$	Observed time $Y_i$	Event $\Delta_i$
1	2	1
2	3	1
3	5	0
4	1	1
5	4	1
6	6	0

Four of the six records are observed events, and two are right-censored. We do not know the underlying $T_3$ or $T_6$ . All we know is that $T_3 > 5$ and $T_6 > 6$ . Those two censored records are the entire reason this example is worth doing. If we knew all six event times, the exercise would be an ordinary exponential fit. Because we do not, the example shows how the likelihood turns partial information into a usable estimate. Suppose we adopt an exponential model, $h(t) = \lambda$ for all $t$ , so that $S(t) = e^{-\lambda t}$ , $f(t) = \lambda e^{-\lambda t}$ , and $H(t) = \lambda t$ . The log-likelihood becomes

\ell(\lambda) = \sum_{i=1}^{6} \Big[ \Delta_i \log h(Y_i) - H(Y_i) \Big] = D \log \lambda - \lambda \sum_{i=1}^{6} Y_i,

where $D = \sum_i \Delta_i$ is the number of events and the sum inside the second term is over all records, not just the events.

The two sufficient statistics are

D = 1 + 1 + 0 + 1 + 1 + 0 = 4, \qquad \sum_{i=1}^{6} Y_i = 2 + 3 + 5 + 1 + 4 + 6 = 21.

To maximize $\ell(\lambda)$ , take the derivative with respect to $\lambda$ :

\frac{d\ell}{d\lambda} = \frac{D}{\lambda} - \sum_{i=1}^{6} Y_i,

which is the standard one-variable calculus problem from Maximum Likelihood Estimation. Setting it to zero gives $\hat{\lambda} = D / \sum_i Y_i = 4 / 21 \approx 0.1905$ . The second derivative is $-D/\lambda^2 < 0$ , so this is a maximum. Notice what the estimator is doing. It divides the number of observed events by the total time that subjects were under observation. Censored records still contribute exposure time to the denominator, which is exactly how they inform the hazard without pretending that their events were observed.

Two alternative estimates reveal what the correct likelihood is protecting us from.

If we treat every observed record as an event, that is, we ignore censoring completely, we would maximize the product of densities over all six records, $\prod_i \lambda e^{-\lambda Y_i}$ , whose logarithm is $\ell_{\text{naive}}(\lambda) = 6 \log \lambda - \lambda \sum_{i=1}^{6} Y_i = 6 \log \lambda - 21 \lambda$ , since $\sum_{i=1}^{6} Y_i = 21$ . The maximizer is $\hat{\lambda}_{\text{naive}} = 6/21 \approx 0.2857$ , which is fifty percent higher than the honest estimate. That corresponds to an expected lifetime of $1 / 0.2857 \approx 3.50$ time units, versus $1 / 0.1905 \approx 5.25$ for the correct fit.

If instead we drop the two censored records, the "complete cases" approach, we maximize $\sum_{\Delta_i = 1} \log(\lambda e^{-\lambda Y_i}) = 4 \log \lambda - \lambda \sum_{\Delta_i = 1} Y_i = 4 \log \lambda - 10 \lambda$ , where the total observed time $\sum_{\Delta_i = 1} Y_i = 2 + 3 + 1 + 4 = 10$ only includes the four observed events. The maximizer is $\hat{\lambda}_{\text{drop}} = 4/10 = 0.4$ , which is $2.1$ times the censoring-aware estimate, or more than double it. That corresponds to an expected lifetime of $2.5$ time units.

PythonIn [10]
CollapseExpandY_ex = np.array([2.0, 3.0, 5.0, 1.0, 4.0, 6.0])
d_ex = np.array([1, 1, 0, 1, 1, 0])

D_ex = int(d_ex.sum())
sumY_ex = float(Y_ex.sum())

lam_hat = D_ex / sumY_ex
lam_naive = len(Y_ex) / sumY_ex
sumY_event = float(Y_ex[d_ex == 1].sum())
lam_drop = D_ex / sumY_event

# Independent library comparison: an exponential distribution with fixed location.
reference_data = CensoredData.right_censored(Y_ex, d_ex == 0)
reference_loc, reference_scale = expon.fit(reference_data, floc=0.0)
lam_reference = 1.0 / reference_scale
np.testing.assert_allclose(lam_reference, lam_hat, rtol=1e-10, atol=1e-12)

ll_hat = D_ex * np.log(lam_hat) - lam_hat * sumY_ex
ll_naive = len(Y_ex) * np.log(lam_naive) - lam_naive * sumY_ex
ll_drop = D_ex * np.log(lam_drop) - lam_drop * sumY_event

Out[11]:

	estimate	lambda_hat	expected_lifetime
0	correct MLE	0.1905	5.25
1	treat censored as events	0.2857	3.50
2	drop censored	0.4000	2.50

SciPy's CensoredData.right_censored marks the two censored observations, and expon.fit(..., floc=0) estimates the exponential scale while holding the location at zero. Its reciprocal agrees with $4/21$ to the tolerance asserted in the executed cell. This checks the six-record exponential case against an established censored-distribution implementation; it is not a check of covariate regression or delayed entry.

On these six records, treating censorings as events increases the rate from $4/21$ to $6/21$ , and dropping them gives $4/10$ . The censored durations contribute eleven units of known event-free exposure. Dropping them removes that exposure without changing the event count. This calculation demonstrates the effect on this dataset; it is not proof of the direction of every regression bias. The three fits use different observation rules, so their maximized objective values would not be valid likelihood comparisons on a common dataset.

To see the maximum visually, plot the log-likelihood as a function of $\lambda$ .

PythonIn [12]
CollapseExpandlam_grid = np.linspace(0.05, 0.45, 200)
ll_grid = D_ex * np.log(lam_grid) - lam_grid * sumY_ex

PythonIn [13]
CollapseExpandimport matplotlib.pyplot as plt
from book_plot_style import PALETTE, polish_axes, use_book_style

use_book_style()
fig, ax = plt.subplots(figsize=(6.0, 4.0))
ax.plot(lam_grid, ll_grid, color=pal(0), linewidth=1.1, label="log-likelihood")
ax.axvline(
    lam_hat,
    color=pal(1),
    linestyle="--",
    linewidth=1.0,
    label=f"MLE  $\\hat\\lambda = {lam_hat:.4f}$",
)
ax.axvline(
    lam_naive,
    color=pal(2),
    linestyle=":",
    linewidth=1.0,
    label=f"naive $\\hat\\lambda = {lam_naive:.4f}$",
)
ax.scatter([lam_hat], [ll_hat], color=pal(1), zorder=5)
ax.set_xlabel("$\\lambda$")
ax.set_ylabel("log-likelihood $\\ell(\\lambda)$")
ax.set_title("Exponential rate: six-record likelihood")
ax.legend(loc="upper center", bbox_to_anchor=(0.5, -0.2))
polish_axes(ax)
plt.tight_layout()
plt.show()

VisualizationOut [13]
CollapseExpand

Log-likelihood profile for the exponential model on six records, four events and two censored. The concave curve peaks at the censoring-aware MLE of about 0.1905, marked by the dashed line, while the naive estimate that counts censorings as events sits further to the right, showing how the censoring indicator affects the fitted rate.

For this exponential rate objective, $D>0$ and positive total exposure imply a strictly negative second derivative $-D/\lambda^2$ and a unique positive maximizer. The claim is about this parameterization and dataset, not a general concavity guarantee for every survival family or optimizer. The dashed line marks the maximum; the dotted naive estimate sits to its right. Next we compare the analytic estimate, a primitive numerical objective, and a library fit that directly accepts censored observations.

Comments

Add to the conversation

A question, a correction, or an example from your own work.

No comments yet. Be the first to share your thoughts!

Comments

Add to the conversation

Stay up to date