Description

Estimate the mean of a Poisson distribution given a sample of data that has been subjected to Type I censoring, and optionally construct a confidence interval for the mean.

Usage

Arguments

Details

If x or censored contain any missing (NA), undefined (NaN) or infinite (Inf, -Inf) values, they will be removed prior to performing the estimation.

Let \underline{x} denote a vector of N observations from a Poisson distribution with mean lambda=λ.

Assume n (0 < n < N) of these observations are known and c (c=N-n) of these observations are all censored below (left-censored) or all censored above (right-censored) at k fixed censoring levels

T_1, T_2, …, T_k; \; k ≥ 1 \;\;\;\;\;\; (1)

For the case when k ≥ 2, the data are said to be Type I multiply censored. For the case when k=1, set T = T_1. If the data are left-censored and all n known observations are greater than or equal to T, or if the data are right-censored and all n known observations are less than or equal to T, then the data are said to be Type I singly censored (Nelson, 1982, p.7), otherwise they are considered to be Type I multiply censored.

Let c_j denote the number of observations censored below or above censoring level T_j for j = 1, 2, …, k, so that

∑_{i=1}^k c_j = c \;\;\;\;\;\; (2)

Let x_{(1)}, x_{(2)}, …, x_{(N)} denote the “ordered” observations, where now “observation” means either the actual observation (for uncensored observations) or the censoring level (for censored observations). For right-censored data, if a censored observation has the same value as an uncensored one, the uncensored observation should be placed first. For left-censored data, if a censored observation has the same value as an uncensored one, the censored observation should be placed first.

Note that in this case the quantity x_{(i)} does not necessarily represent the i'th “largest” observation from the (unknown) complete sample.

Finally, let Ω (omega) denote the set of n subscripts in the “ordered” sample that correspond to uncensored observations.

Estimation

Maximum Likelihood Estimation (method="mle")
For Type I left censored data, the likelihood function is given by:

L(λ | \underline{x}) = {N \choose c_1 c_2 … c_k n} ∏_{j=1}^k [F(T_j)]^{c_j} ∏_{i \in Ω} f[x_{(i)}] \;\;\;\;\;\; (3)

where f and F denote the probability density function (pdf) and cumulative distribution function (cdf) of the population (Cohen, 1963; Cohen, 1991, pp.6, 50). That is,

f(t) = \frac{e^{-λ} λ^t}{t!}, \; x = 0, 1, 2, … \;\;\;\;\;\; (4)

F(t) = ∑_{i=0}^t f(i) = ∑_{i=0}^t \frac{e^{-λ} λ^i}{i!} \;\;\;\;\;\; (5)

(Johnson et al., 1992, p.151). For left singly censored data, equation (3) simplifies to:

L(λ | \underline{x}) = {N \choose c} [F(T)]^{c} ∏_{i = c+1}^n f[x_{(i)}] \;\;\;\;\;\; (6)

Similarly, for Type I right censored data, the likelihood function is given by:

L(λ | \underline{x}) = {N \choose c_1 c_2 … c_k n} ∏_{j=1}^k [1 - F(T_j)]^{c_j} ∏_{i \in Ω} f[x_{(i)}] \;\;\;\;\;\; (7)

and for right singly censored data this simplifies to:

L(λ | \underline{x}) = {N \choose c} [1 - F(T)]^{c} ∏_{i = 1}^n f[x_{(i)}] \;\;\;\;\;\; (8)

The maximum likelihood estimators are computed by maximizing the likelihood function. For right-censored data, taking the derivative of the log-likelihood function with respect to λ and setting this to 0 produces the following equation:

\bar{x} = λ \{1 - ∑_{j=1}^K \frac{c_j}{n} [\frac{f(T_j)}{1-F(T_j)}] \} \;\;\;\;\;\; (9)

where

\bar{x} = \frac{1}{n} ∑{i \in Ω} x_i \;\;\;\;\;\; (10)

Note that the quantity defined in equation (10) is simply the mean of the uncensored observations.

For left-censored data, taking the derivative of the log-likelihood function with respect to λ and setting this to 0 produces the following equation:

\bar{x} = λ \{1 + ∑_{j=1}^K \frac{c_j}{n} [\frac{f(T_j - 1)}{F(T_j - 1)}] \} \;\;\;\;\;\; (11)

The function epoisCensored computes the maximum likelihood estimator of λ by solving Equation (9) (right-censored data) or Equation (11) (left-censored data); it uses the sample mean of the uncensored observations as the initial value.

Setting Censored Observations to Half the Censoring Level (method="half.cen.level")
This method is applicable only to left censored data. This method involves simply replacing all the censored observations with half their detection limit, and then computing the mean and standard deviation with the usual formulas (see epois).

This method is included only to allow comparison of this method to other methods. Setting left-censored observations to half the censoring level is not recommended.

Confidence Intervals
This section explains how confidence intervals for the mean λ are computed.

Likelihood Profile (ci.method="profile.likelihood")
This method was proposed by Cox (1970, p.88), and Venzon and Moolgavkar (1988) introduced an efficient method of computation. This method is also discussed by Stryhn and Christensen (2003) and Royston (2007). The idea behind this method is to invert the likelihood-ratio test to obtain a confidence interval for the mean λ. Equation (3) above shows the form of the likelihood function L(λ | \underline{x}) for multiply left-censored data, and Equation (7) shows the function for multiply right-censored data.

Following Stryhn and Christensen (2003), denote the maximum likelihood estimate of the mean by λ^*. The likelihood ratio test statistic (G^2) of the hypothesis H_0: λ = λ_0 (where λ_0 is a fixed value) equals the drop in 2 log(L) between the “full” model and the reduced model with λ fixed at μ_0, i.e.,

G^2 = 2 \{log[L(λ^*)] - log[L(λ_0)]\} \;\;\;\;\;\; (11)

. Under the null hypothesis, the test statistic G^2 follows a chi-squared distribution with 1 degree of freedom.

A two-sided (1-α)100\% confidence interval for the mean λ consists of all values of λ_0 for which the test is not significant at level alpha:

λ_0: G^2 ≤ χ^2_{1, {1-α}} \;\;\;\;\;\; (12)

where χ^2_{ν, p} denotes the p'th quantile of the chi-squared distribution with ν degrees of freedom. One-sided lower and one-sided upper confidence intervals are computed in a similar fashion, except that the quantity 1-α in Equation (12) is replaced with 1-2α.

Normal Approximation (ci.method="normal.approx")
This method constructs approximate (1-α)100\% confidence intervals for λ based on the assumption that the estimator of λ is approximately normally distributed. That is, a two-sided (1-α)100\% confidence interval for λ is constructed as:

[\hat{λ} - t_{1-α/2, m-1}\hat{σ}_{\hat{λ}}, \; \hat{λ} + t_{1-α/2, m-1}\hat{σ}_{\hat{λ}}] \;\;\;\; (13)

where \hat{λ} denotes the estimate of λ, \hat{σ}_{\hat{λ}} denotes the estimated asymptotic standard deviation of the estimator of λ, m denotes the assumed sample size for the confidence interval, and t_{p,ν} denotes the p'th quantile of Student's t-distribuiton with ν degrees of freedom. One-sided confidence intervals are computed in a similar fashion.

The argument ci.sample.size determines the value of m and by default is equal to the number of uncensored observations. This is simply an ad-hoc method of constructing confidence intervals and is not based on any published theoretical results.

When pivot.statistic="z", the p'th quantile from the standard normal distribution is used in place of the p'th quantile from Student's t-distribution.

When λ is estimated with the maximum likelihood estimator (method="mle"), the variance of \hat{λ} is estimated based on the inverse of the Fisher Information matrix. When λ is estimated using the half-censoring-level method (method="half.cen.level"), the variance of \hat{λ} is estimated as:

\hat{σ}^2_{\hat{λ}} = \frac{\hat{λ}}{m} \;\;\;\;\;\; (14)

where m denotes the assumed sample size (see above).

Bootstrap and Bias-Corrected Bootstrap Approximation (ci.method="bootstrap")
The bootstrap is a nonparametric method of estimating the distribution (and associated distribution parameters and quantiles) of a sample statistic, regardless of the distribution of the population from which the sample was drawn. The bootstrap was introduced by Efron (1979) and a general reference is Efron and Tibshirani (1993).

In the context of deriving an approximate (1-α)100\% confidence interval for the population mean λ, the bootstrap can be broken down into the following steps:

1. Create a bootstrap sample by taking a random sample of size N from the observations in \underline{x}, where sampling is done with replacement. Note that because sampling is done with replacement, the same element of \underline{x} can appear more than once in the bootstrap sample. Thus, the bootstrap sample will usually not look exactly like the original sample (e.g., the number of censored observations in the bootstrap sample will often differ from the number of censored observations in the original sample).

2. Estimate λ based on the bootstrap sample created in Step 1, using the same method that was used to estimate λ using the original observations in \underline{x}. Because the bootstrap sample usually does not match the original sample, the estimate of λ based on the bootstrap sample will usually differ from the original estimate based on \underline{x}.

3. Repeat Steps 1 and 2 B times, where B is some large number. For the function epoisCensored, the number of bootstraps B is determined by the argument n.bootstraps (see the section ARGUMENTS above). The default value of n.bootstraps is 1000.

4. Use the B estimated values of λ to compute the empirical cumulative distribution function of this estimator of λ (see ecdfPlot), and then create a confidence interval for λ based on this estimated cdf.

The two-sided percentile interval (Efron and Tibshirani, 1993, p.170) is computed as:

[\hat{G}^{-1}(\frac{α}{2}), \; \hat{G}^{-1}(1-\frac{α}{2})] \;\;\;\;\;\; (15)

where \hat{G}(t) denotes the empirical cdf evaluated at t and thus \hat{G}^{-1}(p) denotes the p'th empirical quantile, that is, the p'th quantile associated with the empirical cdf. Similarly, a one-sided lower confidence interval is computed as:

[\hat{G}^{-1}(α), \; ∞] \;\;\;\;\;\; (16)

and a one-sided upper confidence interval is computed as:

[0, \; \hat{G}^{-1}(1-α)] \;\;\;\;\;\; (17)

The function epoisCensored calls the R function quantile to compute the empirical quantiles used in Equations (15)-(17).

The percentile method bootstrap confidence interval is only first-order accurate (Efron and Tibshirani, 1993, pp.187-188), meaning that the probability that the confidence interval will contain the true value of λ can be off by k/√{N}, where kis some constant. Efron and Tibshirani (1993, pp.184-188) proposed a bias-corrected and accelerated interval that is second-order accurate, meaning that the probability that the confidence interval will contain the true value of λ may be off by k/N instead of k/√{N}. The two-sided bias-corrected and accelerated confidence interval is computed as:

[\hat{G}^{-1}(α_1), \; \hat{G}^{-1}(α_2)] \;\;\;\;\;\; (18)

where

α_1 = Φ[\hat{z}_0 + \frac{\hat{z}_0 + z_{α/2}}{1 - \hat{a}(z_0 + z_{α/2})}] \;\;\;\;\;\; (19)

α_2 = Φ[\hat{z}_0 + \frac{\hat{z}_0 + z_{1-α/2}}{1 - \hat{a}(z_0 + z_{1-α/2})}] \;\;\;\;\;\; (20)

\hat{z}_0 = Φ^{-1}[\hat{G}(\hat{λ})] \;\;\;\;\;\; (21)

\hat{a} = \frac{∑_{i=1}^N (\hat{λ}_{(\cdot)} - \hat{λ}_{(i)})^3}{6[∑_{i=1}^N (\hat{λ}_{(\cdot)} - \hat{λ}_{(i)})^2]^{3/2}} \;\;\;\;\;\; (22)

where the quantity \hat{λ}_{(i)} denotes the estimate of λ using all the values in \underline{x} except the i'th one, and

\hat{λ}{(\cdot)} = \frac{1}{N} ∑_{i=1}^N \hat{λ_{(i)}} \;\;\;\;\;\; (23)

A one-sided lower confidence interval is given by:

[\hat{G}^{-1}(α_1), \; ∞] \;\;\;\;\;\; (24)

and a one-sided upper confidence interval is given by:

[0, \; \hat{G}^{-1}(α_2)] \;\;\;\;\;\; (25)

where α_1 and α_2 are computed as for a two-sided confidence interval, except α/2 is replaced with α in Equations (19) and (20).

The constant \hat{z}_0 incorporates the bias correction, and the constant \hat{a} is the acceleration constant. The term “acceleration” refers to the rate of change of the standard error of the estimate of λ with respect to the true value of λ (Efron and Tibshirani, 1993, p.186). For a normal (Gaussian) distribution, the standard error of the estimate of λ does not depend on the value of λ, hence the acceleration constant is not really necessary.

When ci.method="bootstrap", the function epoisCensored computes both the percentile method and bias-corrected and accelerated method bootstrap confidence intervals.

Value

a list of class "estimateCensored" containing the estimated parameters and other information. See estimateCensored.object for details.

Note

A sample of data contains censored observations if some of the observations are reported only as being below or above some censoring level. In environmental data analysis, Type I left-censored data sets are common, with values being reported as “less than the detection limit” (e.g., Helsel, 2012). Data sets with only one censoring level are called singly censored; data sets with multiple censoring levels are called multiply or progressively censored.

Statistical methods for dealing with censored data sets have a long history in the field of survival analysis and life testing. More recently, researchers in the environmental field have proposed alternative methods of computing estimates and confidence intervals in addition to the classical ones such as maximum likelihood estimation. Helsel (2012, Chapter 6) gives an excellent review of past studies of the properties of various estimators for parameters of a normal or lognormal distribution based on censored environmental data.

In practice, it is better to use a confidence interval for the mean or a joint confidence region for the mean and standard deviation (or coefficient of variation), rather than rely on a single point-estimate of the mean. Few studies have been done to evaluate the performance of methods for constructing confidence intervals for the mean or joint confidence regions for the mean and coefficient of variation of a Poisson distribution when data are subjected to single or multiple censoring.

Author(s)

Steven P. Millard (EnvStats@ProbStatInfo.com)

References

Cohen, A.C. (1991). Truncated and Censored Samples. Marcel Dekker, New York, New York, 312pp.

Cox, D.R. (1970). Analysis of Binary Data. Chapman & Hall, London. 142pp.

Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7, 1–26.

Efron, B., and R.J. Tibshirani. (1993). An Introduction to the Bootstrap. Chapman and Hall, New York, 436pp.

Forbes, C., M. Evans, N. Hastings, and B. Peacock. (2011). Statistical Distributions, Fourth Edition. John Wiley and Sons, Hoboken, NJ.

Helsel, D.R. (2012). Statistics for Censored Environmental Data Using Minitab and R, Second Edition. John Wiley \& Sons, Hoboken, New Jersey.

Johnson, N. L., S. Kotz, and A. Kemp. (1992). Univariate Discrete Distributions, Second Edition. John Wiley and Sons, New York, Chapter 4.

Millard, S.P., P. Dixon, and N.K. Neerchal. (2014; in preparation). Environmental Statistics with R. CRC Press, Boca Raton, Florida.

Nelson, W. (1982). Applied Life Data Analysis. John Wiley and Sons, New York, 634pp.

Royston, P. (2007). Profile Likelihood for Estimation and Confdence Intervals. The Stata Journal 7(3), pp. 376–387.

Stryhn, H., and J. Christensen. (2003). Confidence Intervals by the Profile Likelihood Method, with Applications in Veterinary Epidemiology. Contributed paper at ISVEE X (November 2003, Chile). http://people.upei.ca/hstryhn/stryhn208.pdf.

Venzon, D.J., and S.H. Moolgavkar. (1988). A Method for Computing Profile-Likelihood-Based Confidence Intervals. Journal of the Royal Statistical Society, Series C (Applied Statistics) 37(1), pp. 87–94.

See Also

Poisson, epois, estimateCensored.object.

Examples