Estimating prevalence using composites - Springer Link

Environmental and Ecological Statistics 8, 213±236, 2001

Estimating prevalence using composites Â N , G A N A P AT I P. PAT I L , S I LV E S T R E C O LO and C H A R L E S TA I L L I E Center for Statistical Ecology and Environmental Statistics, Department of Statistics, The Pennsylvania State University, 421 Thomas Building, University Park, PA 16802, USA Received June 1998; Revised December 2000 We are interested in estimating the fraction of a population that possesses a certain trait, such as the presence of a chemical contaminant in a lake. A composite sample drawn from a population has the trait in question whenever one or more of the individual samples making up the composite has the trait. Let the true fraction of the population that is contaminated be p. Classical estimators of p, such as the MLE and the jackknife, have been shown to be biased. In this study, we introduce a new shrinking estimator which can be used when doing composite sampling. The properties of this estimator are investigated and compared with those of the MLE and the jackknife. Keywords: Burrows estimator, composite sampling, jackknife estimator, shrinking estimator 1352-8505 # 2001

Kluwer Academic Publishers

1. Introduction The problem studied in this paper is that of the estimation of prevalence. Consider a population whose members may or may not have a certain trait. A good example is the presence or absence of a chemical contaminant. Let p be the proportion with the trait and k the composite size. If the presence and absence of the trait are coded as 1 and 0, respectively, then p is the population mean and at ®rst glance it might appear that we should estimate p by using X, the average of the individual responses. Here, however, the composite response is not the mean of the individual responses. Instead, the composite is free of the trait exactly when each individual sample is free of the trait. Thus, the composite response is the maximum of the individual responses. Because of this nonlinearity, the maximum likelihood estimator of p, based upon n composites, is biased and the bias cannot be completely removed. Let k be the composite sample size. Then, the information per composite is not proportional to k. Asymptotically, for large n, the information increases with k, then reaches a maximum at k kopt , and then falls as k increases beyond kopt . (For ®nite sample sizes, Burrows (1987) has found that, in some instances, the reciprocal of the mean square error can be a multimodal function of k.) For k4kopt , the drop off in information can be precipitous, even to the point where it is better to make n measurements on individual sampling units rather than n measurements on composites of size k. Thus, in designing a composite sampling study, it is wise to ensure that k kopt and our analytical investigations will be limited to this range. 1352-8505 # 2001

Kluwer Academic Publishers

ColoÂn, Patil, Taillie

214

This paper examines three estimators of prevalence based on composite samples: 1. The maximum likelihood estimator (MLE). 2. The jackknife estimator. 3. A class of shrinking estimators which shrink the MLE toward zero. It turns out that the MLE is positively biased and shrinking can reduce the bias as well as simultaneously reduce the mean square error (MSE) of estimation. The trick is not to shrink too much, so we consider a family of estimators with two adjustable parameters, denoted by b and c, that regulate the degree of shrinking. A particular member of the family, with bc

k

1 2k

;

has been previously proposed by Burrows (1987). The parameter c has only a small effect on the performance of the shrinking estimator and ®rst appears in the third order terms of the asymptotic expansions of the bias and MSE in powers of 1n. Actually, c was included only so that the Burrows estimator would be a special case. It will be shown that the parameter b has a greater impact upon performance. It appears in the lowest order term 1n of the bias and the second order term n12 of the MSE. This suggests two criteria for choosing b. First, one may try to reduce the magnitude of the bias which leads to Burrows choice, b k2k1, and completely eliminates the lowest order term in the bias, uniformly in p. Second, one may try to minimize the second order term in the MSE (the lowest order term equals the asymptotic variance and is the same for all the estimators). No single choice of b will minimize uniformly in p. A compromise choice of b is proposed (the 75 rule) that works well over a broad range of p, provided k kopt . The above considerations are based on the asymptotic expansions, so it becomes necessary to assess the performance of the estimators for ®nite sample sizes, n. Analytical results are not feasible, but the random variables involved are all transformations of binomially distributed variates, so the moments can be computed `èxactly'' without recourse to simulations. The computational algorithms for the small sample moments of the jackknife estimator require some rather intricate derivations which will be described below.

2. Large sample approximations Let p be the prevalence of the trait for the individual samples, k be the composite sample size, p pk be the prevalence of the trait across all possible composites of size k. Let n be the number of composites. Since the trait is absent from the composite if and only if it is absent from the k individual samples, the parameters p and p are related by the formula 1

p 1

or p1

1

pk

1

1

pk : Hp:

We call H ? the prevalence transformation. Fig. 1 shows the graph of H for different

Estimating prevalence using composites

215

Figure 1. The prevalence transformation Hp for k 1; 2; 4; 8.

values of k. From the ®gure, we see when k41 the prevalence transformation is monotone increasing, convex, and highly nonlinear when p is large. For future reference, we note that 1 H 1 p 1 k H 2 p

k

1

pk 1

k2

1

1

40; 1

2

pk

40:

2.1 Bias and MSE of the maximum likelihood estimator ^ so the MLE for p is The MLE for p is the sample proportion p p^ H^ p: Since H is nonlinear, the MLE p^ is biased, and the bias can be substantial for ®nite sample sizes. Using the Taylor series expansion, we have p^ Hp ^ p

pH 1 p ^ p

2

p H 2 p=2 :

But, p Hp; so this gives p^

p ^ p

p pH1 p ^

p2 H 2 p=2 :

Taking expectations, the bias of the MLE is given by Bias^ p

? X i1

gi

Hi p ; i! i

where gi E^ p pi and H i d dHp i p . We may rearrange terms to obtain the asymptotic expansion in powers of 1n, 4 H 2 p H 3 p 1 2 H p g3 3g2 O 3 ; Bias^ p g 2 2 3! 4! n where


216 g1 0; p1

g2

p n

p1

g3

;

p1 n2

2p

:

After simpli®cation we have,

d1 d2 1 Bias^ p 2 O 3 ; n n n

where d1

k 1 1 1 pk ; 2k2 1 pk 1

and d2

k

12k

11

1

pk 24k4

2

p4k

3p kp

:

The mean square error MSE is given by 2

MSE^ p g2 H 1 p g3 H1 pH 2 p 2 1 2 3 2 3 1 H p H pH p O 3 : g2 4 n Observe that MSE p^ can be expressed in the form A B 1 MSE^ p 2 O 3 ; n n n where A

1

pk

1

pk

k2 1

2

;

and B

k

11

2

pk

2

pk4 3p 4k4

7p

:

It is interesting to observe that the asymptotic variance of p^ due to compositing is varc ^p H 1 p2 var^ p H 1 p2

p1 p ; nc

where n nc is the number of composites analyzed. In the individual sampling case, the variance is vari ^p

p1 p ni


217

where ni is the number of individuals analyzed. If measurement instead of sample acquisition is the primary cost factor, the relative cost of the two sampling procedures can be measured by the ratio of the samples needed to achieve the same variance for the two designs. So, by the above equations, we have that the asymptotic relative cost of compositing compared with individual sampling is RC

k

nc 1 1 p : ni k2 p1 pk 1

From this equation, we note that RC51 indicates nc 5ni in favor of the compositing over individual sampling. In addition, observe that the relative cost satis®es the following: * *

RC? 1k as p?0, and RC ? ? as p?1 (unless k 1).

Therefore, neither sampling design is uniformly better than the other and the compositing tends to be better for small p while individual sampling is better for large p. In Fig. 2, we have plotted the asymptotic relative cost RC versus p for various values of k. From the lower envelope of these curves, we can determine the optimal value composite sample of size kopt . In practice one has to determine an appropriate value of k by using some preliminary assessment of p. If this preliminary assessment of p is too small, then k will be overestimated and the achieved performance of compositing may be seriously degraded, perhaps to the point of being worse than individual sampling. Therefore, in the preliminary design stage, it is better to overestimate p and underestimate k. Table 1 shows the values of p for which the optimal value of k makes a transition from kopt k to ^k & 0:80. This is kopt k 1. When the optimal value of k (kopt ) is used, we observe that p close to the region of high bias due to the nonlinearity or curvature of H (see Fig. 1). From the table, we also observe that kopt gets large very fast as p ? 0. In fact, kopt *1:594=p when p is small. Often kopt may be too large for practical use in contrast

Figure 2. The asymptotic cost of compositing relative to individual sampling as a function of the true prevalence p for k 1; 2; 3; 4; 20; 100.


218

Table 1. Values of p pk where the optimal composite sample size makes a transition from kopt k to kopt k 1. When p is slightly larger than pk , the optimal composite sample size is k; when p is slightly smaller than pk , the optimal sample size is k 1. The composite prevalence pk H 1 pk corresponding to pk is also tabulated. K

pk

pk

1 2 3 4 5 6 7 8 9 50 100 500

0.667 0.475 0.367 0.299 0.252 0.218 0.192 0.171 0.155 0.031 0.016 0.003

0.667 0.725 0.747 0.759 0.776 0.771 0.774 0.777 0.779 0.794 0.795 0.796

p with the Dorfman (1943) procedure where kopt O1= p as p?0. In these cases, a suboptimal but more practical choice of k5kopt can still give large performance gains over individual sampling (see Fig. 2)

2.2 Shrinking estimator ^ toward For k 4 1, H is monotone increasing, so we can reduce the Bias in p^ by shrinking p zero before applying the prevalence transformation. Since Hp is ¯atter on the left this should also reduce the variance and the MSE. This suggests that we estimate p as H^ p , ^ a^ where p p with 0 a 1. See Fig. 3.

Figure 3. Shrinking prevalence transformation Hap for k 2 and for different a.


219

Since the bias goes to zero as 1n, the constant a has to depend upon the sample size n. In particular, we must have a ? 1 as n??. A natural choice is b b bc 1 1 2 O 3 : a1 n c n n n p Ha^ p using the Taylor series expansion as We can express p^ H^ p p^ Hp a^

pH0 p a^ p

p2 H00 p=2 :

k

If we let tk Ea^ p

p , we can express the bias as ? X

H i p : i! i1 h i 2 Similarly, expanding MSE E ^ p Hp gives i 2 X X ? X H p H i p H j p : t2i 2 ti j MSE p^ i! i! j! i4j i1 b 1 Putting a 1 nc 1 bn bc n2 O n3 , we have b bc 1 t1 2 pO 3 ; n n n 2b b2 2 1 t2 1 g2 2 p O 3 ; n n n 3b 1 g2 p O 3 ; t3 g3 n n 1 t4 3g22 O 3 : n p An Bn2 O n13 , where So the bias for p^ can be written as Bias^ 1 p1 pk 1 1 1 A 1 ; b 2 k k 1 1 pk 1 1 1 ; b 2 k k1 pk 1 Bias p^

ti

and B

1 1 24k4

1

pk

6 3p k4

2

p 26b

3k2 4 3p 4b2 p 2k3 4 p 6b2 p

5p 4bp 2 12b1

c cp:


220 Similarly, the mean square can be written as A B 1 MSE m m2 O 3 ; n n n where Am

2

p1

pk

1

k2

1 k2 1

p

k

k

2

1 p

and 2 1 2 1 pk p7p 2k6b 5p 4k4 k2 4 3p 4b2 p 4b2 p:

Bm

2

We see that the lowest order term in the bias can be eliminated by the choice b k 2k 1. In applications, however, the MSE may be more important than the bias, and we may want to know how we can reduce the MSE. The low order term in MSE is the asymptotic variance, and there is nothing we can do to reduce this. The second order term depends on b but not on c and is quadratic in b, so we can easily minimize it to obtain 1 1 bopt 2 p

3 : 2k

Unfortunately, this expression depends upon p so there is no uniformly best choice of b. From earlier results, if the optimal k is used, then pk &0:80 (0:75pk 50:8). This is a region of potentially high bias due to the curvature of the prevalence transformation H. We can study the effect of the value of b on the MSE by examining three rules for choosing b: * * *

3 75 rule: b75 11 6 2k. 3 80 rule: b80 74 2k . Eliminate bias rule or Eb: bEb k 2k 1.

Figure 4. 75 rule, 80 rule, and eliminate bias rule (Eb).


221

The 75 rule and the 80 rule are obtained by putting p 0:75 and p 0:80 in the above expression for bopt . From Fig. 4, we observe that the 75 rule and 80 rule shrink the estimator more than the Eb rule. This may introduce a negative bias, but may also reduce the variance and thereby reduce the MSE. To see how the different rules affect the MSE, we compare the ratio of the second order term to the ®rst order term in the Taylor expansion of the MSE. This ratio, divided by the sample size, is the percentage increase in the MSE due to the second term relative to the ®rst order term (asymptotic variance). Thus, ratio

Bm Am

Note that

ratio 1 O 2 : MSE=asyvar 1 n n

where asyvar is the asymptotic variance. Also note that a negative ratio is possible and desirable. From Fig. 5, we see that there is no practical difference between the 75 rule and 80 rule. Both of these yield a smaller MSE than does the eliminate bias rule. So we compare the 75 rule with the Eliminate bias rule by de®ning MSEjump ratioEb rule

ratio75 rule:

When MSEjump is divided by the sample size and then multiplied by 100, the result is the percentage increase in the MSE that results from using the eliminate bias rule instead of the 75 rule. In Fig. 6, MSEjump is plotted versus p for different values of k. Note that the vertical axis values should be multiplied by 100 n in order to get the percentage increase in MSE. Therefore, if we are willing to accept some bias (in the form of underestimate on average), then the MSE can be reduced by choosing the 75 rule, that is, selecting 3 b b75 11 6 2k. For n 100 and k5kopt , the reduction in the MSE is between 2 and 5%.

Figure 5. Ratio comparison for the 75 rule, 80 rule, and eliminate bias rule.


222

Figure 6. MSEjump versus p.

We can investigate which values of c are better when the 75 rule is used by de®ning nratio

MSE3 ; MSE1

where MSE1 and MSE3 are the ®rst and third order terms in the expansion of MSE[SHRINKING] respectively. Thus, nratio75 is the nratio when b b75 is used. It can be shown that nratio75 R S c; where R is given by R

1 54k4 1 30k2 3

2

p

54kp

2p 9p2 k4 66 318p

19p 13p2 2k3 81

245p2

396p 289p2

and S is given by S

9

11k4p 3 : 9k1 p

Note that nratio75 is linear in c. We pick out the slope to determine the sign. Letting nratio75Slope be the slope of the nratio75 and using the fact that p 1 1 pk in Fig. 7 we plot the slope versus p for various values of k to determine the sign. From Fig. 7, we see that the slope vanishes in the general vicinity of the optimal p corresponding to k. For p close to this optimal prevalence, it will not make much difference how we choose c because its coef®cient is close to zero. The range of practical interest is 05p5popt . In this


223

Figure 7. nratio75Slope versus p for various k.

range the coef®cient of c is positive, so the MSE can be reduced by making c small. Although, c could be negative as long as n c is positive, it seems unnatural to use negative c. Thus, examination of the third order term suggests that c 0 is reasonable choice when the 75 rule is adopted.

2.3 Burrows estimator ^b n n b p ^ 1 n b b^ The Burrows estimator is p p, where b k 2k 1 ; so that it is a special case of the shrinking estimator family with c b. Observe that the ®rst order term from the asymptotic expansion of the bias is eliminated, so the bias becomes Bb 1 Bias^ pb 2 O 3 ; n n where Bb

1

k2 1

The mean square error is

1

p 2 k p 24k3

2p

Amb Bmb 1 MSE^ pb 2 O 3 ; n n n

where

:


224 Amb Bmb

2 k

1

1

p k2

p

k

12 1 pk 2k4

; 2

2 2

p

:

Burrows (1987) has shown that the bias and the mean squared error are smaller than for the maximum likelihood estimator.

2.4 Jackknife estimator The jackknife MLE of p is p^jack

n 1X n^ p n e1

n

1^ pe

where pê is the MLE of p when observation number e is deleted from the data set. i Let hi H i!p ; i 1; 2; 3; . . . ; . The Taylor expansion of p^jack about p is ! n 1 X n p^ p n 1 pê p p^jack p n e1 n p^

n

p

n

n 1X pê

p:

e1

But, p^

p

? X i1

hi ^ p

pi

and pê

p

n X i1

hi ^ pe

i

p ;

ê is the MLE when observation number e is deleted from the data set. Thus, where p ! n n X n 1X i i hi n^ p p ^ p p : p^jack p n e1 e i1 Taking expectations gives the bias of the jackknife estimator as ? X n 1 Mn 1;i hi nMn;i Bias jack n i1 where the Mn;i is the ith central moment of the binomial distribution with parameters n and p. To obtain the mean square error of the jackknife estimator, we need to square the above expansion of p^jack p. So, let us write

Estimating prevalence using composites n 1X

n

pi

Ti n^ p

n

e1

225 pi

^ pe

which gives p^jack

? X

p

i1

hi Ti :

Squaring and taking the expectations gives p2 ;

MSE jack E^ pjack MSE jack

? X ? X ii j1

hi hj ETi Tj :

2

But, pi j

Ti Tj n2 ^ p

n X

1

e0

1

n X e1

n

n

n

1

^ p

1

n

i

p

^ pe

pi ^ p e0

p ^ pe 0

2 X n X n e 1 e0 1

pi ^ p

^ pe

pj

j

pj :

3

We break the last term into two sums, where e e0 and where e 6 e0 . The last term then becomes

n

1 n

2 X n ^ pe

p

ij

n

1

2

n

e1

X e;e0 1;e6e0

^ pe

pi ^ pe 0

pj :

Now, let p Mn;i E^

pi

1

p ^ p

2

pi ^ pe 0

i

Mn;i;j E^ pe pe Mn;i;j E^

j

p pj

where e 6 e0 . Taking the expectation of both sides of Equation (3) gives ETi Tj n2 Mn;i j

n

1 n

nn 2

Mn

1

1

1Mn;i;j Mn;j;i 1;i j

n

1 n

3

2

Mn;i;j :

Finally , we need to compute M1 and M2 . But,

4


226 ^ p

x1 xn n

p

n

ê Note that p ^ p

1 x1

n n

p xn p n z}|{ p xe p xn p xe p n 1 n

1 n

^ pe

p

p

xe

x1

p n

:

p is independent of xe p. Applying the binomial theorem, ! j m j j n 1m X j m 1 j m p ^ pe p xe p ; n n m0 m ! j 1 X j m m j m n 1 ^ j pe p xe p : n m0 m i

p and taking the expectations gives ! j 1 X j j n 1m Mn 1;m i M1; j m : n m0 m

Multiplying by ^ pe 1

Mn;i; j

5

2

Next, we look at Mn;i; j . Now ê p

p

n n

2 ^ p 0 1 e;e

p

^ e0 p

p

n n

2 ^ p 0 1 e;e

p

1 n

1

xe0

p;

xe

p;

1 n

1

ê;e0 is the MLE of p with observation numbers e and e0 (with e 6 e0 ) deleted. where p Therefore, pi ^ pe0

^ pe

i i X

1 n

j X

j

m0

m

p j

n

pe;e0 2l ^

pe;e0 2m ^

pm xe

1i l 0 ! n

!

l

j i X X

1

i

!

1i j l 0 m 0 l

n

j

2 Mn;i; j

1 n

j i X X

i

1i j l 0 m 0 l

pj

pi

l

1 n

1 j

m

!

m

Taking expectations gives

pl xe0

!

pe;e0 2l m ^

n

j m

pl m xe0

pi

l

pj

xe

m

! n

2l m Mn

2;l m M1;i

l M1; j

m:

6

The Equations (2), (4), (5), and (6) give the formal expression for MSE jack in terms of the central moments Mn;i ; Mn 1;i ; Mn 2;i ; M1;i of the binomial distribution. Next we need to study the order of these expressions in 1n to determine how many terms have to be carried along.


227

Since there are no adjustable parameters in the jackkni®ng estimator, it should be adequate to obtain MSE jack to within On13 . Note that three of the four terms in Equation 2 (4) have multiplicative factors that are On . Therefore the others must be computed to 1 O n5 . ETi Tj is O

Theorem 1.

1 n3

if i j48.

The proof of this theorem is a consequence of Equations (4)±(6) and the following lemma. Lemma (a) Mn;i O n1t where t ceil2i . Here, ceilx rounds x upward to an integer. 1 (b) Mn;i;j O n15 if i j48. More speci®cally, the nth term in Equation (5) is O n15 if m52i j 8, thus the sum in (5) can be started at m max0; 2j i 8. 2 (c) Mn;i;j O n15 if i j48. More speci®cally, the l; m term in Equation (6) is O n15 if l m52i j 8. Thus, the double sum in (6) can be computed as i X

X

l0

m max0;2i j

: 8

l

Proof: (a) Straightforward. (b) The relevant factor in the mth term of Equation (5) is n

1

m

nj By (a), Mn

Mn

1;m i :

1;m i

d j ceil

is O

mi 2

7

! 1 ceil m i 2

n

so (7) has order

m:

mi 1 Write ceilmi 2 2 emi where e emi is 0 if m i is even and 2 if m i is odd. Then

i d j e 2 Thus (7) is O i j e 2

1 n5

m : 2 if d44, i.e., if

m 44; 2

which will be true if j

i 2

m 44: 2

8

But the last will be true for all terms in (5) if it is true for m 0; 1; . . . ; j: But this will be so if it is true for the largest m, i.e., for m j. Thus (5) is O n15 if.


228 j

i 2

j 44 2

or i j48 Also, we see that we can ignore those m satisfying (8) when we compute the sum in (5). Thus, we can start the sum at m 2j i 8 j 8 i j: (c) Similar reasoning. & Efron (1982) has shown that the bias of the jackknife estimator can be written as Bj 1 Bias^ pjack 2 O 3 ; n n where Bj

1

k1

2k1

1

pk 24k4

2

p4k

3p kp

:

Observe that the jackknife estimator eliminates the ®rst order bias from an estimator, as is well known. Also, the mean square error of the jackknife estimator has the form Amj Bmj 1 2 O 3 ; MSE n n n where Amj Bmj

k

p2=k k2

1

2

1

p 2

1 1 pk 2k4

; 2 2

p

:

Note that Bmj Bmb , so the MSE for the Burrows and jackknife estimators are equal to second order term.

2.5 Comparisons For all the three estimators, we ®nd that the bias and the mean square error have the forms A B 1 bias 2 O 3 ; n n n Am B m 1 2 O 3 ; MSE n n n where Am is the same for all three estimators (Am =n is the asymptotic variance). For the bias expansion, the shrinking estimator and the jackknife estimator have different values of B, but the form of B does not involve the constant c of the shrinking estimator. Also B


229

Figure 8. rterm2 versus p.

depends upon p and neither the shrinking estimator nor jackkni®ng is uniformly better than the other in minimizing B. Observe that the asymptotic bias expansion, for the Burrows and the jackknife estimator both eliminate the ®rst order term. Therefore it is useful to compare the second terms by de®ning rterm2

B : p

When rterm2 is divided by n2 and then multiplied by 100, the result is the relative ( percent) bias due to second term. From Fig. 8, we have that the rterm2 for the jackknife is always negative and rterm2 for Burrows estimator is always positive. But, with a sample size as small as n 10 and a sensible choice of k, the contribution to the bias is only 0.10% to 0.50% for Burrows estimator and 0.20% to 4.5% for the jackknife estimator. Thus, the jackknife estimator tends to underestimate and the Burrows estimator tends to overestimate p, at least asymptotically. It is interesting to observe that the MSE expression for both the jackknife estimator and Burrows estimator are equal up to the second order term in the asymptotic expansion. So one would expect that when n is large enough, the MSE for both estimators would be similar. In the next section, we study the behavior of these estimators for small sample sizes.

3. Exact small sample computation of bias and MSE 3.1 Maximum likelihood estimator We already have p^ H^ p Hny where y*Binomialn; p. Therefore bias and the mean square error can be computed as


230 bias(MLE) E^ p where by; n; p

n y

p;

n X y H n y0

pn

py 1

y

. Similarly,

p2 ;

MSE(MLE) E p^

p by; n; p;

n X y H n y0

2 p by; n; p:

3.2 Shrinking estimator b We have a 1 nc where b,c are given. Remember that the shrinking estimator is de®ned by p^s Ha ny. Then, the bias and the mean square error in the shrinking case are

bias(SHRINKING) E p^s

n X y H a n y0

p

p by; n; p

and 2

MLE(SHRINKING) E p^s

p

n X y H a n y0

2 p by; n; p:

3.3 Jackknife estimator We have seen earlier that p^jack

p n p^

n

p

n

n 1X pê

p;

e1

where pê is the MLE when observation number e is deleted from the data set. Thus, the bias can be computed as bias jack n biasMLE; n

n

1 bias MLE; n

1:

For the mean square error computation, we have pjack

p2 n2 p^

p2

2n

1

n X e1

n

1 n

n

e 1 e0 1e0 6e

1 n

2 X n n X 2 X n e1

pê

pê

p2 :

pê

p p^

p pê0

p p


231

The mean square error for the jackknife then becomes MSE jack n2 MSE1

n

1

MSE2

n

2nn

1 MSE3

13

n

MSE4;

n

where MSE1 MSEMLE; n; MSE2 MSEMLE; n

1;

MSE3 E pê

p p^

p

MSE4 E pê

p pê0

p:

We need to calculate MSE3 and MSE4. We have y y y^ y n p^ H 1 H e e ; n n n

where yê y1 yn ye , and pê H nyê1 . We use iterated conditional expectation and the fact that yê and ye are independent to obtain E^ pe

p^ p yê

p EEUj^ ye ; e pH yê y p. Since ye *Binomial1; p, where U H n 1 n yê y^ y^ 1 EUj^ ye H p 1 pH e pH e n n 1 n

p

and, E^ pe

p^ p

p

Similarly, pê H Thus,

nX1 y0

yê;e0 ye0 n 1

EUj^ ye;e0

H

E^ pe p^ p e0 h y^ y where U H e;en 1 e 0

0

n

y 1

p 1

and pê0 H

y1 pH pH n n y

yê;e0 ye n 1

p by; n

where yê;e0 y1 yn

ye

0

yê;e0 ye E H n 1

p^ p

p

nX2

1

y0

ye0 .

p EEUj^ ye;e0 ih i y^ y p H e;en 1 e p . Since yê;e0 ; ye ; ye0 are independent, we have 2 pj^ ye;e0

1

pH

yê;e0 yê;e0 1 pH n 1 n 1

Thus, E^ pe

1; p:

y y1 pH pH n 1 n 1

p

2

by; n

2; p:

2 p

:

232


Figure 9. Relative bias expressed as a percent (BIASper) for small samples n i 23 n 20 with p 0:75.

Figure 10. Relative bias expressed as a percent (BIASper) for n i 1415 n 80 with p 0:75.

Figure 11. MSEs for MLE, Burrows, 75 rule and jackknife n i 23 n 80 with p 0.75.


233

Figure 12. MSEper of the estimators with p 0:75 and n i 2 3 n 20.

3.4 Comparisons for small sample size The relative bias (expressed as a percent) of an estimator of p is given by BIASper 100

Eestimator p

p

:

In Figs. 9±14, it is important to read the captions because the horizontal axis of the ®gure often does not represent the sample size n. The horizontal axis in Figs. 9±14 is represented by i and the relationship between n and i always described in the caption. Looking at Fig. 9, it is clear that for n less than 20, the Burrows estimator is more effective at the bias reduction than the jackknife, 75 rule and MLE.

Figure 13. MSEper of the estimators with p 0:75 and n i 29 30 n 80.

234


Figure 14. DMSEper for p 0:70 (left) and p 0:80 (right) with n i 2 3 n 100. Negative values of DMSEper indicate that 75 rule is more effective than the Burrows estimator in reducing MSE.

In Fig. 10, there are 4 plots which, for a given k, depict the bias of the four estimators for larger sample sizes. In each plot, we can see that for n greater than 20, the performance of the jackknife and the Burrows estimator in bias reduction is almost identical and always better than the 75 rule and the MLE. Fig. 11 shows how MSE is affected by sample size. As n becomes large, the MSEs of all four estimators converge to the asymptotic variance. In order to distinguish among the estimators, we de®ne

Estimating prevalence using composites MSEper MSE(estimator) asyvar 100

235

asyvar

where asyvar is the asymptotic variance. The quantity MSEper is the relative difference per observation between the MSE and the asymptotic variance. In Fig. 12, we observe for small samples n520, that the MLE and jackknife estimators have large values of MSEper indicating that they are ineffective in reducing the MSE. Fig. 13 shows that MSEper for the jackknife and Burrows estimator are approximately equal for n greater than 50. This is consistent with our earlier results showing these two estimators had the same MSE to second order in n. We can compare MSEper for the Burrows and 75 rule de®ning DMSEper MSEper75 rule

MSEperBurrows:

Note that DMSEper nMSE75 rule MSEBurrows asyvar 100: In Fig. 14, there are 8 plots. The four plots on the right represent DMSEper for p 0:7 and the four plots on the left represent the same thing but for p 0:8. These values of p were chosen because for k kopt we expect pk to be between 0.7 and 0.8. In all 8 plots, we see that for n greater than 10, the 75 rule achieves a 10 to 30% reduction in MSE per observation compared with the Burrows estimator. The anomalous behavior of DMSE per for very small sample sizes indicates that the asymptotic theory leading to the 75 rule is inapplicable in this range.

Notes Prepared with partial support from the Statistical Analysis and Computing Branch, Environmental Statistics and Information Division, Of®ce of Policy, Planning, and Evaluation, United States Environmental Protection Agency, Washington, DC under a Cooperative Agreement Number CR-825506-01-0. The contents have not been subjected to Agency review and therefore do not necessarily re¯ect the views of the Agency and no of®cial endorsement should be inferred.

References Burrows, P.M. (1987) Improved estimation of pathogen transmission rates by group testing. Phytopathology, 77(2), 363±65. Dorfman, R. (1943) The detection of defective members of large populations. Annals of Mathematical Statistics, 14, 436±40. Efron, B.E. (1982) The Jackknife, the Bootstrap and Other Resampling Plans, SIAM, Philadelphia, PA. Johnson N.L. and Kotz, S. (1969) Distributions in Statistics: Discrete Distributions, Wiley, New York. Rhode, C.A. (1976) Composite Sampling. Biometrics, 32, 273±82. Wolfram, S. (1990) Mathematica, Addison-Wesley, Redwood City, CA.

236


Biographical Sketches Silvestre ColoÂn is an Instructor in the Department of Mathematics at the University of Puerto Rico. G.P. Patil is Distinguished Professor of Mathematical Statistics and Director of the Center for Statistical Ecology and Environmental Statistics in the Department of Statistics at the Pennsylvania State University. C. Taillie is Senior Research Associate in the Center for Statistical Ecology and Environmental Statistics in the Department of Statistics at the Pennsylvania State University.