2.3 Nonparametric estimators

As an introductory example, look at an extremely simple data set: \(4\), \(2^*\), \(6\), \(1\), \(3^*\) (starred observations are right censored; in the figure, deaths are marked with \(+\), censored observations with a small circle), see Figure 2.4.

FIGURE 2.4: A simple survival data set.

How should the survival function be estimated? The answer is that we take it in steps. First, the hazard “atoms” are estimated. It is done nonparametrically, and the result as such is not very useful. Its potential lies in that it is used as the building block in constructing estimates of the cumulative hazards and survival functions.

2.3.1 The hazard atoms

In Figure 2.5, the observed event times in Figure 2.4 are marked, the vertical dashed lines at durations 1, 4, and 6, respectively. In the estimation of the hazard atoms, the concept of is of vital importance.

FIGURE 2.5: Preliminaries for estimating the hazard function.

The risk set \(R(t)\) at duration \(t, \; t > 0\) is defined mathematically as

\[\begin{equation} R(t) = \{\text{all individuals under observation at $t-$}\}, \tag{2.7} \end{equation}\] or in words, the risk set at time \(t\) consists of all individuals present and under observation just prior to \(t\). The reason we do not say ``present at time \(t\)’’ is that it is vital to include those individuals who have an event or are right censored at the exact time \(t\). In our example, the risk sets \(R(1)\), \(R(4)\), and \(R(6)\) are the interesting ones. They are:

\[\begin{equation*} \begin{split} R(1) &= \{1, 2, 3, 4, 5\} \\ R(4) &= \{1, 3\} \\ R(6) &= \{3\} \\ \end{split} \end{equation*}\]

\[\begin{equation} \begin{split} \hat{h}(1) &= \frac{1}{5} = 0.2 \\ \hat{h}(4) &= \frac{1}{2} = 0.5 \\ \hat{h}(6) &= \frac{1}{1} = 1 \\ \end{split} \tag{2.8} \end{equation}\]

The estimation of the hazard atoms is simple. First, we assume that the probability of an event at times where no event is observed, is zero. Then, at times where events do occur, we count the number of events and divides that number by the size of the corresponding risk set. The result is shown in (2.8).

FIGURE 2.6: Nonparametric estimation of the hazard function.

See also Figure 2.6.

As is evident from Figure 2.6, the estimated hazard atoms will be too irregular to be of practical use; they need smoothing. The simplest way of smoothing them is to calculate the cumulative sums, which leads to the Nelson-Aalen estimator of the cumulative hazards function, see the next section. There are more direct smoothing techniques to get reasonable estimators of the hazard function itself, e.g., kernel estimators , but they will not be discussed here. See e.g. Silverman (1986) for a general introduction to kernel smoothing.

2.3.2 The Nelson-Aalen estimator

FIGURE 2.7: The Nelson-Aalen estimator.

From the theoretical relation we immediately get

\[\begin{equation*} \hat{H}(t) = \sum_{s \le t} \hat{h}(s), \quad t \ge 0 \end{equation*}\]

which is the Nelson-Aalen estimator (Nelson 1972; Aalen 1978), see Figure 2.7. The sizes of the jumps are equal to the heights of the “spikes” in Figure 2.6.

2.3.3 The Kaplan-Meier estimator

From the theoretical relation (2.6) we get

\[\begin{equation} \hat{S}(t) = \prod_{s < t} \bigl(1 - \hat{h}(s)\bigr), \quad t \ge 0, \tag{2.9} \end{equation}\]

see also Figure 2.8. Equation (2.9) may be given a heuristic interpretation: In order to survive time \(t\), one must survive all “spikes” (or shocks) that come before time \(t\). The multiplication principle for conditional probabilities then gives equation (2.9).

FIGURE 2.8: The Kaplan-Meier estimator.

References

Aalen, O. O. 1978. “Nonparametric Inference for a Family of Counting Processes.” Annals of Statistics 6: 701–26.

Nelson, W. 1972. “Theory and Applications of Hazard Plotting for Censored Failure Data.” Technometrics 14: 945–65.

Silverman, B. W. 1986. Density Estimation. Chapman & Hall.