Statistical Methods in Medical Research 2004; 13: 1^21
Sample size requirements for the design of reliability study: review and new results MM Shoukri Department of Epidemiology and Biostatistics, University of Western Ontario, London, Ontario, Canada and Department of Biostatistics, Epidemiology and Scienti¢c Computing, King Faisal Specialist Hospital and Research Centre, Riyadh, Kingdom of Saudi Arabia, MH Asyali Department of Biostatistics, Epidemiology and Scienti¢c Computing, King Faisal Specialist Hospital and Research Centre, Riyadh, Kingdom of Saudi Arabia and A Donner Department of Epidemiology and Biostatistics, University of Western Ontario, London, Ontario, Canada
The reliability of continuous or binary outcome measures is usually assessed by estimation of the intraclass correlation coefficient (ICC). A crucial step for this purpose is the determination of the required sample size. In this review, we discuss the contributions made in this regard and derive the optimal allocation for the number of subjects k and the number of repeated measurements n that minimize the variance of the estimated ICC. Cost constraints are discussed for both normally and non-normally distributed responses, with emphasis on the case of dichotomous assessments. Tables showing optimal choices of k and n are given along with the guidelines for the efficient design of reliability studies.
1
Introduction
Measurement errors can seriously affect statistical analysis and interpretation; it therefore becomes important to assess the magnitude of such errors by calculating a reliability coefficient and assessing its precision. Although the topic of reliability has gained much attention in the literature,1–3 investigations into sample size requirements remain scarce. In this review, we revisit the issue of sample size requirements for reliability studies having either continuous or binary outcomes. In either case, the measurement of reliability must distinguish within-subject variation from between-subjects variation. A widely recognized index that possesses this property is the intraclass correlation coefficient (ICC) defined as: r ¼ s2s =(s2s þ s2e ), where s2s and s2e are the among-subjects and within-subjects components of variance, respectively. It is seen that r is the proportion of between-subject variation relative to the total variation. In the most frequently adopted design, k subjects are each rated by the same n raters (for inter-rater reliability). A similar approach, however, can also be adopted when a single subject is assessed repeatedly on each of several occasions (test–retest reliability), or when replicates consisting of different occasions are taken on different Address for correspondence: MM Shoukri, Department of Biostatistics, Epidemiology and Scientific Computing, King Faisal Specialist Hospital and Research Centre, MBC-03, PO Box 3354, Riyadh 11211, Kingdom of Saudi Arabia. E-mail:
[email protected]
# Arnold 2004 Techset Composition Ltd, Salisbury Printed: 8/6/04 Page#:
10.1191/0962280204sm365ra Doc: {HODDER}smmr/SM365RA.3d 21 page(s) Opp: Task:
ManuNo. 0000 3B2 Version: 7.51f/W (Mar 4 2002)||TechRef:9.01/H (March 20 1997)
2 MM Shoukri, MH Asyali and A Donner subjects by a single rater.4 In each of these cases, and for both continuous and binary assessments, r can be estimated from an appropriate one-way analysis of variance (ANOVA).5,6 The question then arises as to the optimal combination (n, k) that permits an accurate estimation of r. One of the limiting factors in designing and implementing reliability studies in many settings is the difficulty of arranging for the replicated observations. In clinical situations, for instance, there are typically only a few specialists available, in a hospital or clinic, who are willing to participate in a study and who are qualified to make the observations. For example, Walter et al.7 reported a study in which physiotherapists were required to evaluate the gross motor function of children with Down’s syndrome, using videotapes. Owing to respondent burden and other practicalities of scheduling, it was not possible to arrange for more than three or four therapies to assess a given child. Had live observations been made, it would have been unreasonable to increase the number of observers beyond this point, because of likely intimidation of the child, particularly since evaluations are often done in their home. Also repetitive observations may lead to fatigue and aversion effects. Similar considerations apply in many reliability studies, and n is typically limited by a maximum tolerable respondent burden such as in an interview of physical assessment of clinical condition. The scientific question is related to the optimal allocation of (n, k) to conduct a reliability study. Another study is on the reliability of diagnosing dysplasia in the urothelial lining of the bladder, which may be an indicator of poor prognosis in patients with superficial bladder cancer. However, in a British Medical Research Council (BMRC) randomized multi-centre clinical trial of treatment for superficial bladder cancer, considerable discrepancies appeared to exist between the assessment of dysplasia by local pathologists and by the reference pathologist for the trial.8 Therefore, it was decided to establish a board of five pathologists who specialized in urology to evaluate the extent of their agreement on the assessment of dysplasia. At the time of design of the study, no formal methods for calculating appropriate sample sizes were available. However, the investigators independently assessed 100 slides, a number determined to estimate a reliability coefficient with prespecified level of precision. We note that the number of replicates (five per slide) is fixed a priori, and the issue of sample size in this situation is entirely different from that of the previous example. The third example is based on a study by Awni et al.9 in bioavailability=bioequivalence (BA=BE) study. In a typical BA=BE trial, the area under the blood concentration time curve (AUC) is considered an important parameter. To assess the intra-subject variability with respect to a specific drug formulation, the ICC is used and therefore, several measurements of the AUC are needed from each subject. Given that in a BA=BE study subjects are paid volunteers, the study design must address the issues of cost and time constraints while producing precise estimate for ICC. For a fixed number of replicates, Donner and Eliasziw10 have provided contours of exact power for selected values of k and n, while Eliasziw and Donner11 used these results to identify optimal designs that minimize the study costs. Walter et al.7 developed a simple approximation to these exact results that allow the calculation of the required value of k when n is fixed. We note that reliability studies, however, are primarily designed to estimate the level of observer agreement, with their results invariably reported as measures of such agreement. Yet, power considerations in the design of reliability studies necessarily require specification of the hypotheses to be tested. Rejection of H0: r ¼ 0 based on results from a reliability
Design of reliability study 3 study is particularly unhelpful, since the investigator needs to know more than the fact that the observed level of reliability is unlikely to be due to chance.12 Given that reliability studies are essentially estimation procedures, it is natural to base the sample size calculations on the attainment of a specified level of precision for the estimated r. For example, Bonett,13 focusing on the case of a continuous outcome measure, provided sample size requirements based on the need to achieve a specified expected width for a confidence interval (CI) on r. These results are parallel to those of Donner,14 who focused on estimating the number of subjects required to construct a CI with fixed width about the intraclass kappa coefficient for the case of a binary outcome measure. In this review, we revisit the literature on sample size requirements when interest is focused on estimating the ICC reliability from a single sample of subjects. In Section 2, we discuss issues of power, efficiency and fixed length CIs when the response variable is normally distributed. Methods that use the calculus of optimization to find the combination (n, k) that minimizes the variance of the estimated ICC when the response variable is continuous are investigated in detail. Issues of cost are investigated in the situation when both k and n are determined so that the variance of the estimated ICC is minimized subject to cost constraints. We devote Section 3 to the issue of optimal design when the assessments are binary, and present an overall discussion of our results in Section 4. 2
Continuous outcome measures
2.1 Normal case 2.1.1 Power considerations The most commonly used model for estimating reliability is the one-way random effects model, yij ¼ m þ si þ eij
(1)
where m is the grand mean of all measurements in the population, si reflects the effect of subject i, and eij is the error of measurement, i ¼ 1, 2, . . . , k; j ¼ 1, 2, . . . , n. It is assumed that the subject effects {si} are normally and identically distributed with mean 0 and variance s2s , the errors {eij} are normally and identically distributed with mean 0 and variance s2e and the {si} and {eij} are independent. Letting MSA and MSW denote the among-subjects and within-subject mean squares, respectively, the ANOVA estimator of r is given by r^ ¼
MSA MSW F1 ¼ MSA þ (n 1)MSW F þ n 1
(2)
where F ¼ MSA=MSW. Donner and Eliasziw10 investigated the values of k and n required to test H0: r ¼ r0 versus H1: r > r0, where r0 is a specified criterion value of r. For the case n ¼ 2, that is, test–retest data, we may use Fisher’s normalizing transformation,6 analogous to the well known Fisher transformation of the Pearson product–moment or interclass
4
MM Shoukri, MH Asyali and A Donner
correlation. Fisher showed that u ¼ (1=2) ln {(1 þ r^ )=(1 r^ )} is nearly normally distributed with mean m(r) ¼ (1=2) ln {(1 þ r)=(1 r)} and variance 1=(k 1:5). Let za and zb denote the values of the standard normal distribution corresponding to the chosen level of significance a and power 1 7 b. The required number of subjects for testing H0: r ¼ r0 versus H1: r ¼ r1 > r0 is obtained directly from the previous theory as (za ¼ 1:64) þ (zb ¼ 0:84) 2 k ¼ 1:5 þ (3) m(r0 ) m(r1 ) Table 1 gives the required values of k according to the values of r0 and r1, a ¼ 0.05 and b ¼ 0.20. The results in Table 1 indicate that the required sample size k depends critically on the values of r0 and r1, and in particular, on their difference. For example, much more effort is required to distinguish values that differ by 0.1 compared with those that differ by 0.2. Note also that larger samples are required to detect relatively small values of r1 for a given difference r1 7 r0. Walter et al.7 developed a simple approximation that allows the calculation of the required number of subjects k for an arbitrary number of replicates n. The interest here is in testing H0: r ¼ r0 versus H1: r ¼ r1 > r0 using F ¼ MSA=MSW ¼ {1 þ (n 1)r^ }=(1 r^ ) where r^ is given by Equation (2). The critical value for the test statistic is CFa,n1 ,n2 , where C ¼ 1 þ {nr0 =(1 r0 )} and Fa,n1 ,n2 is the 100(1 7 a) per cent point in the cumulative F-distribution with degrees of freedom given by n1 ¼ k 7 1, and n2 ¼ k(n 7 1). Their approximation uses a simple formula, avoiding the intensive numerical work required to implement exact methods and providing the investigator increased flexibility for exploring various design options. As described by Donner and Eliasziw,10 the test of H0: r ¼ r1 has power given by 1 b ¼ Pr[F C0 Fa,n1 ,n2 ]
(4)
where b is the type II error and C0 ¼ (1 þ nf0 )=(1 nf), with f0 ¼ r0 =(1 r0 ) and f ¼ r1 =(1 r1 ). To solve Equation (4), Walter et al.7 used a result by Fisher,6 regarding the asymptotic distribution of Z ¼ (1=2) ln F. Omitting details, the estimated number of subjects is given by k¼1þ
nA(a þ b)
(5)
(n 1)( ln C0 )2
where A(a, b) ¼ 2(za þ zb )2 . Table 1 Number of subjects k at n ¼ 2 (a ¼ 0.05 and b ¼ 0.20) r0
r1
k
0.2 0.2 0.4 0.8 0.6
0.6 0.8 0.6 0.9 0.8
27 9 86 46 39
Design of reliability study 5 Table 2 Approximate sample size (a ¼ 0.05 and b ¼ 0.10) r0
r1
n
ln C0
k
0.2 0.4 0.6 0.8 0.2
0.4 0.8 0.8 0.9 0.6
10 20 10 10 2
0.784 1.732 0.941 0.797 0.981
32 6 22 31 36
Table 2 shows the required values of k for typical values of n, and according to the given values of r0 and r1, at a ¼ 0.05 and b ¼ 0.10. They indicate that the required sample size k depends on the values of r0 and r1, and particularly, on their difference. The remarks made concerning the values of k in Table 1 also apply to the values of k in Table 2. 2.1.2 Specified width of a CI Giraudeau and Mary12 and Bonett13 argued that the approach of hypothesis testing may not be appropriate while planning a reliability study. This is because one has to specify both the values of r0 and r1, which may in practice be a difficult task. An alternative approach is to instead focus on the width of the CI for r. Indeed, in single sample problems, the results of a reliability study are usually expressed as a point estimate of r and its associated CI. The sample size calculations are then aimed at achieving an interval estimate that has sufficient precision. The approximate width of a 95% CI on r is equal to 2za=2 var(r^ )1=2 , where var(r^ ) ¼
2(1 r)2 {1 þ (n 1)r}2 kn(n 1)
(6)
is the approximate variance of the ICC r^ as derived by Fisher6 and za=2 is the critical value of the standard normal distribution exceeded with probability a=2. However, Equation (6) requires samples of at least moderate size (e.g., k 30) to insure its validity. An approximation to the sample size that yields an exact CI for r having desired width w is obtained by setting w ¼ 2za=2 {var(r)}1=2 , with r replaced by ‘planning value’ r , and then solving for k to give k¼
8z2a=2 (1 r )2 {1 þ (n 1)r }2 w2 n(n 1)
(7)
which is then rounded up to the nearest integer. The approximation suggested by Bonett13 is k* ¼ k þ 1, where k is given by Equation (7). Table 3 gives the required sample size for typical planned values of r for w ¼ 0.2, a ¼ 0.05 and various values of n. As can be seen from Table 3, the value of k* is a decreasing function of n for any given value of r . Thus, if the cost of sampling an additional subject is relatively high, it may be less costly to increase the number of replicates per subject than to increase
6
MM Shoukri, MH Asyali and A Donner Table 3 Values of k* for planned values of r, with w ¼ 0.2 r
0.6 0.7 0.8 0.9
n 2
3
5
10
158 101 52 15
100 67 36 11
71 51 28 9
57 42 24 8
the number of subjects. Besides the usual advantages of interval estimation over hypothesis testing, Bonett13 argued that the effect of an inaccurate planning value (required for both procedures) is more serious in the context of hypothesis testing. For example, to test H0: r ¼ 0.7 at a ¼ b ¼ 0.05, with n ¼ 3, the required sample size obtained using the approach of Walter et al.7 is about 3376, 786 and 167 for r1 ¼ 0.725, 0.75 and 0.80, respectively. In comparison, the sample size required to estimate r with a 95% CI having width of 0.2 is 60, 52 and 37 for r ¼ 0:725, 0:75 and 0.80, respectively. 2.1.3 Efficiency requirements In this section, it is assumed that the investigator is interested in determining the number of replicates n per subject needed to minimize the variance of the estimated r, where we assume the total number of measurements N ¼ nk is fixed due to practical constraints. Following Shoukri et al.,15 substitution of N ¼ nk into Equation (6) gives var(r^ ) ¼ f (n, r) ¼
2(1 r)2 {1 þ (n 1)r}2 N(n 1)
(8)
A necessary condition for f(n, r) to have a minimum is that qf =qn ¼ 0, with a sufficient condition given by q2 f =qn2 > 0.16 Differentiating f(n, r) with respect to n, equating to zero and solving for n we obtain n0 ¼
1þr r
(9)
Moreover, (q2 f =qn2 )jn¼n0 ¼ 4{r3 (1 r2 )}=N > 0 and the sufficient condition for a unique minimum is therefore satisfied. Note that the range of r is strictly positive, since within the framework of reliability studies negative values are usually not of interest. Equation (9) indicates that, when r ¼ 1, n0 ¼ 2 is the minimum number of replicates needed per subject. The smaller the value of r, the larger the required n, and hence a smaller number of subjects k ¼ N=n would be needed. Table 4 shows the optimal combination (n, k) that minimizes the variance of r^ for different values of r.
120
90
60
N
k var(r^ ) k var(r^ ) k var(r^ )
4.45 (0.011) 8.18 (0.007) 10.9 (0.005)
0.1 (n ¼ 11) 10 (0.017) 15 (0.011) 20 (0.008)
0.2 (n ¼ 6) 13.8 (0.020) 20.8 (0.013) 27.7 (0.010)
0.3 (n ¼ 4.3) 17.1 (0.019) 25.7 (0.013) 34.3 (0.010)
0.4 (n ¼ 3.5)
Table 4. Optimal combinations of (n, k) which minimize the variance of r^
20 (0.017) 30 (0.011) 40 (0.008)
0.5 (n ¼ 3)
r
22.5 (0.013) 33.75 (0.008) 45 (0.006)
0.6 (n ¼ 2.7)
24.7 (0.008) 37 (0.006) 49.4 (0.004)
0.7 (n ¼ 2.4)
26.7 (0.004) 40 (0.003) 53.3 (0.002)
0.8 (n ¼ 2.25)
28.4 (0.001) 42.6 (0.001) 56.8 (0.001)
0.9 (n ¼ 2.1)
Design of reliability study 7
8
MM Shoukri, MH Asyali and A Donner
Remarks 1. Because N ¼ nk is fixed, a larger number of replicates n lead to a much smaller number of recruited subjects, which may limit the generalizability potential of the study. However, if r is expected to be moderately high (>0.6), not more than two or three replicates per subject are required. 2. The previous remark is similar to the conclusion made by Giraudeau and Mary,12 who based their sample size requirements on the achievement of a specific width for the 95% CI. These guidelines are also consistent with the results reported in Table 3 of Walter et al.7 3. In practice, only integer values of (n, k) are used and because N ¼ nk is fixed a priori, optimum values of n should first be rounded to the nearest integer and then k ¼ N=n is rounded to the nearest integer as well. Shoukri et al.15 showed that the net loss=gain in efficiency of the estimated reliability coefficient is negligible. For example, when r ¼ 0.7, N ¼ 60, the optimal allocations are n ¼ 2.43 and k ¼ 24.69, giving var(r) ¼ 0.0084, whereas the rounded integer allocations are n ¼ 2 and k ¼ 30, giving var(r) ¼ 0.0087 (i.e., a net loss in efficiency of 3.7%). 2.1.4 Incorporation of cost constraints Funding constraints will often determine the cost of recruiting subjects for a reliability study. Although too small a sample may lead to a study that produces an imprecise estimate of the reliability coefficient, too large a sample may result in a waste of resources. Thus a critical a decision in many such studies is to balance the cost of recruiting subjects with the need to obtain a reasonably precise estimate of r. There have been attempts to address the issue of statistical power in the presence of funding constraints. Eliasziw and Donner11 presented a method for determining the number of subjects k and the number of replicates n that minimize the overall cost of conducting a reliability study, while providing acceptable power for tests of hypotheses concerning r. They also provided tables showing optimal choices for k and n under various cost constraints. Shoukri et al.15 addressed the issue of obtaining the combinations (n, k) that minimize the variance of r^ subject to cost constraints. In their attempt to construct a flexible cost function, they adhered to the general guidelines identified by Flynn et al.17 and Eliasziw and Donner.11 First, one has to identify approximately the sampling costs and overhead costs. The sampling cost depends primarily on the size of the sample, and includes data collection costs, travel costs, management and other staff costs. On the other hand, overhead costs remain fixed regardless of sample size, including, for example, the cost of setting the data collection form. Following Sukhatme et al.,18 it is assumed that the overall cost function is given as: C ¼ c0 þ kc1 þ nkc2
(10)
where c0 is the fixed cost, c1 the cost of recruiting a subject, and c2 is the cost of making a single observation. Using the method of Lagrange multipliers,16 the objective function G is given as G ¼ var(r^ ) þ l(C c0 kc1 nkc2 )
(11)
Design of reliability study 9 where var(r^ ) is given by Equation (8) and l is the Lagrange multiplier. The necessary conditions for the minimization of G are qG=qn ¼ 0, qG=qk ¼ 0 and qG=ql ¼ 0, with the sufficient condition for G to have constrained relative minimum given by a theorem in Rao.16 Differentiating G with respect to n, k and l, and equating to zero, we obtain n3 rc2 n2 c2 (1 þ r) nc1 (2 r) þ (1 r)c1 ¼ 0
(12)
and k¼
C c0 c1 þ nc2
(13)
The explicit expression for the optimal solution to these equations was given by Shoukri et al.15 as nopt ¼
(1 þ r)=r þ A1=3 =r B 3
(14)
where A ¼ 3r 3R (R þ 1)2 r4 (6R2 þ 4R 2)r3 þ 12R(R þ 1)r2 (8R2 þ 10Rþ 1=2 2)r R 1 þ 9R(r3 r2 þ r) þ (r þ 1)3 , B ¼ {3Rr(r 2) (r þ 1)2 }=rA1=3 and R ¼ c1 =c2 . Once the value of nopt is determined and r, the corresponding optimal k for given C 7 c0, c1, c2, is obtained by substitution into Equation (13). The results of the optimization procedure appear in Table 5. Without loss of generality, we assume that C 7 c0 ¼ 100. It is apparent from Table 5 that when c1 (the cost per subject) increases, the required number of subjects (k) decreases, while the required number of replicates per subject (n) increases. However, when c2 increases, both k and n decrease. On the other hand, when c1 and c2 are fixed, an increase in r results in a decline in the required value of n and an increase in k. This trend reflects two intuitive facts: the first is that it is sensible to decrease the number of items associated with a higher cost, while increasing those with lower cost; the second is that when r is large (high reproducibility) then a fewer number of replicates per subject are needed, while a higher number of subjects should be recruited. Note that this is similar to the conclusion reached in the previous section, when the cost was not explicitly considered. We also note that at the higher levels of c1 and c2, the optimal allocation is quite stable with respect to changes in sampling cost. This is advantageous in practice, since it is often difficult to forecast the exact cost prior to the initiation of the study. Finally, it can be noted that by setting c1 ¼ 0 and c2 ¼ 1 in Equation (12), we obtain nopt ¼ (1 þ r)=r, as in Equation (9). This means that a special cost structure is implied by the optimal allocation procedure discussed in the previous section. Moreover, setting r ¼ 1 in Equation (12) gives nopt ¼ 1 þ (1 þ R)1=2 2, emphasizing that the ratio R ¼ c1=c2 is an important factor in determining the optimal allocation of (n, k). Example 1 This example is given in Shoukri et al.15 where, to assess the accuracy of Doppler echocardiography (DE) in determining aortic valve area (AVA) prospective evaluation on patients with aortic stenosis, an investigator wishes to demonstrate a high
10 MM Shoukri, MH Asyali and A Donner Table 5 Optimal values of n and k that minimize var(r^ ) c2
c1
r 0.25
0.25 0.5 1 3 5 15 25
0.7 0.8 0.9 0.7 0.8 0.9 0.7 0.8 0.9 0.7 0.8 0.9 0.7 0.8 0.9 0.7 0.8 0.9 0.7 0.8 0.9
0.5
1
3
5
15
25
n
k
n
k
n
k
n
k
n
k
n
k
n
k
3 2.76 2.57 2.74 2.52 2.36 2.60 2.40 2.24 2.5 2.3 2.1 2.5 2.3 2.1 2.4 2.3 2.1 2.4 2.3 2.1
100 106 112 62 66 70 35 38 40 13 14 15 8 8 9 3 3 3 2 2 2
3.4 3.15 2.9 3 2.76 2.57 2.74 2.53 2.36 2.54 2.35 2.20 2.5 2.3 2.2 2.4 2.3 2.1 2.4 2.3 2.1
73 78 81 50 53 56 31 33 35 12 13 14 8 8 9 3 3 3 2 2 2
4.1 3.77 3.48 3.44 3.15 2.92 3 2.76 2.57 2.64 2.44 2.28 2.56 2.37 2.22 2.47 2.29 2.15 2.45 2.27 2.1
49 51 53 37 39 41 25 26 28 11 12 13 7 8 8 3 3 3 2 2 2
6 5.45 5 4.69 4.27 3.93 3.8 3.48 3.21 3 2.76 2.57 2.79 2.58 2.4 2.56 2.37 2.22 2.51 2.32 2.17
22 23 24 19 19 20 15 15 16 8 9 9 6 6 7 2 3 3 2 2 2
7.33 6.65 6.07 5.6 5.10 4.67 4.42 4.03 3.71 3.30 3.0 2.81 3.00 2.76 2.57 2.64 2.44 2.28 2.56 2.37 2.22
15 15 15 13 13 14 11 11 11 7 7 7 5 5 6 2 2 3 2 2 2
11.73 10.60 9.64 8.68 7.86 7.16 6.54 5.93 5.43 4.42 4.03 3.72 3.80 3.48 3.21 3 2.76 2.57 2.8 2.58 2.4
6 6 6 5 5 5 5 5 5 4 4 4 3 3 3 2 2 2 2 2 2
14.8 13.35 12.12 10.82 9.78 8.90 8.03 7.28 6.65 5.25 4.77 4.38 4.42 4.03 3.72 3.30 3.03 2.81 3.00 2.76 2.57
4 4 4 3 4 4 3 3 3 2 3 3 2 2 2 2 2 2 1 1 1
degree of reliability (r ¼ 90%) in estimating AVA using the ‘velocity integral method’. Suppose that the total cost of making the study is fixed at $1600. We assume that the travel costs for a patient in going from the health center to the tertiary hospital (where the procedure is done) is $15. The administrative cost of the procedure and the cost of using the DE is $15 per visit. It is assumed that c0, the overhead cost, is absorbed by the hospital. From Table 5, nopt for R ¼ 1 and r ¼ 0:9 is 2.57, which should be rounded up to 3. From Equation (13) kopt ¼
1600 ffi 27 15 þ 3 15
That is, we need 27 patients, with three measurements each. The minimized value of var(r^ ) is 0.00097. 2.2 Non-normal case As indicated earlier, the sampling distribution and formula for the variance of the estimated ICC rely on an assumption of normality, which in practice can only be approximately satisfied. In this regard, it should be noted that for statistical inferences in the one-way random effects ANOVA model, it has been found that the distribution of the ratio of mean squares is often quite robust with respect to non-normality. In particular, Scheffe´19 concluded that the impact of non-normality on inferences for
Q1
Design of reliability study 11 means is slight, but can be serious for inferences on variances of random effects whose kurtosis differ from zero. Although, Scheffe´’s conclusions were based on inferences of the variance ratio f ¼ s2s =s2e , they likely have similar implications for r ¼ f=(1 þ f). 2.2.1 Efficiency requirements Tukey20 obtained the variance of the variance component estimates under various ANOVA models by employing ‘polykeys’. For the one-way random effects model, application of the delta method shows that to a first order approximation,21,22 var(r^ ) ¼
g 2(1 r2 ){1 þ (n 1)r}2 g þ r2 (1 r2 ) s þ e kn(n 1) k kn
(15)
where gs ¼ E(s4i )=s4s and ge ¼ E(e4ij )=s4e . Note that when gs ¼ ge ¼ 0, var(r^ ) reduces to the corresponding expression for the normal case, given by Equation (6). Differentiating Equation (15) with respect to n and equating to zero, the optimal value for n is obtained as: n ¼ 1 þ
1
(16)
r(1 þ gs )1=2
Remarks 1. At gs ¼ 0, n* is equal to n0, as given by Equation (9). Moreover, for large values of gs, reflecting increasing departures from normality, a smaller number of replicates is needed with a correspondingly larger number of subjects. Thus the recommended strategy for choosing n and k is the same as that for the normal case. 2. The actual form of the error distribution does not influence the optimal number of replicates. However, both the error distribution and the between-subjects random effect distributions do affect the level of precision associated with r^ . Nonetheless, as can be seen from Equation (15), the influence of ge on the estimated precision is much smaller than the influence of gs provided N ¼ nk is large. Table 6 Optimal values of k and n that minimize var(r^ ) R ¼ c1=c2
r
gs ¼ ge 0.1
0.1 1 10
0.7 0.8 0.9 0.7 0.8 0.9 0.7 0.8 0.9
0.5
2
n
k
n
k
n
k
2.48 2.31 2.16 2.98 2.75 2.57 5.55 5.08 4.67
38.72 41.58 44.17 25.12 26.64 28.02 6.43 6.63 6.82
2.43 2.28 2.16 2.91 2.73 2.56 5.36 5.00 4.65
39.46 41.93 44.26 25.55 26.84 28.07 6.51 6.67 6.82
2.29 2.22 2.14 2.73 2.64 2.54 4.91 4.77 4.60
41.77 43.12 44.59 26.83 27.50 28.26 6.71 6.77 6.85
12 MM Shoukri, MH Asyali and A Donner
Figure 1 Optimal n versus R ¼ c1=c2 for gs ¼ ge ¼ 0:1 and r ¼ 0.7, 0.8, 0.9.
2.2.2 Incorporation of cost constraints Again using the method of Lagrange multipliers, we construct the objective function as G ¼ var(r^ ) þ l(C c0 kc1 nkc2 ), where var(r^ ) is given by Equation (15) and l is the Lagrange multiplier. Differentiating G with respect to n, k and l, and equating to zero, we obtain n3 rc2 n2 c2 (1 þ r) nc1 (2 r) þ (1 r)c1 þ n2 (n 1)2 c2 r2 (1 r)2 gs (n 1)2 c1 r2 (1 r)2 ge ¼ 0
(17)
as equations for n, with k again given by Equation (13). We note that when gs ¼ ge ¼ 0 Equation (17) reduces to Equation (12), which was obtained for the normal case. Collecting powers of n, we obtain the following 4th degree polynomial for the optimal n: c2 r2 (1 r)2 gs n4 þ {rc2 2c2 r2 (1 r)2 gs }n3 þ { c2 (1 þ r) þ c2 r2 (1 r)2 gs c1 r2 (1 r)2 ge }n2 þ {2c1 r2 (1 r)2 ge c1 (2 r)}n þ c1 (1 r) c1 r2 (1 r)2 ge ¼ 0
Design of reliability study 13
Figure 2 Optimal n versus R ¼ c1=c2 for gs ¼ ge ¼ 0:5 and r ¼ 0.7, 0.8, 0.9.
Although an explicit solution to this equation is available, the resulting expression is complicated and does not provide any useful insight. However, we may summarize the results of the optimization procedure as in Table 6, where we provide the optimal n and k for various values of r, R and gs ¼ ge. Once nopt is determined, it is substituted in Equation (13) to determine the corresponding optimal kopt as
kopt ¼
C c0 (C c0 )=c2 ¼ c1 þ nopt c2 R þ nopt
In this brief table, where without loss of generality we set (C c0 )=c2 ¼ 100, our principal aim is to establish the behaviour of the optimal (n, k) as r, and gs ¼ ge vary. In Figures 1–3, we provide more detailed plots of optimal n versus R ¼ c1=c2 for different values r and gs ¼ ge. Most of the remarks made with respect to Table 5 hold for Table 6 as well. Moreover, from Table 6 and from the figures it is seen that for fixed R and r, the optimal value of n decreases, while the optimal value of k increases with increasing departure from normality.
14 MM Shoukri, MH Asyali and A Donner
Figure 3 Optimal n versus R ¼ c1=c2 for gs ¼ ge ¼ 2 and r ¼ 0.7, 0.8, 0.9.
3
Binary outcome measures
In assessing inter-rater reliability, a choice must be made on how to measure the condition under investigation. One of the practical aspects of this decision concerns the relative advantages of measuring the trait either on a continuous scale, as in the proceeding sections or on a categorical scale. In this review, attention is restricted to the case of dichotomous judgments. Kraemer23 pointed out that it is useful to distinguish between two conceptually distinct types of dichotomous scores. The first of these is a truly dichotomous variable, which categorizes members of a population into nonoverlapping subpopulations on a nominal scale. Examples would include a variable denoting the presence or absence of a morbid condition, such as sleep apnea. As the time and expense of monitoring the occurrence of this event on a long term basis might be substantial, physicians might argue on practical grounds that a continuous ‘surrogate variable’ be used in its place. For example, the use of heart rate variability measures to examine the progress of the condition could result in a trial of shorter duration and lower cost.24 The second type of dichotomous variable arises when an inherently continuous variable is dichotomized to correspond closely to how the variable is used in clinical practice. A common example is the dichotomization of patient blood pressure scores into hypertensive and normotensive categories. Donner
Design of reliability study 15 and Eliasziw25 investigated the statistical implications associated with the process of dichotomization as compared with the case of a truly dichotomous variable, and concluded that the effect of dichotomization on the efficiency of reliability estimates can be severe. In the following section, we investigate the issues of efficiency and cost in designing reliability studies for dichotomous responses, where no distinction between the two types of variables is made.
3.1 Power considerations We now consider issues related to the choice of outcome measure. One of the practical aspects of this decision concerns the relative advantages of measuring the trait of interest on a continuous versus a dichotomous scale. First, we outline sample size requirements for the case of binary outcome measure when k subjects are judged by two raters. An underlying model for this case was given by Mak.26 Let yij denote the binary observation for the jth rater of the ith subject, i ¼ 1, 2, . . . , k; j ¼ 1, 2 and let p ¼ Pr[yij ¼ 1] be the probability that the trait is present. Then, for the special case considered here p1 (k) ¼ Pr[yi1 ¼ 1, yi2 ¼ 1] ¼ p2 þ kp(1 p) p2 (k) ¼ Pr[yi1 ¼ 0, yi2 ¼ 1] ¼ Pr[yi1 ¼ 1, yi2 ¼ 0] ¼ 2p(1 p)(1 k)
(18)
2
p3 (k) ¼ Pr[yi1 ¼ 0, yi2 ¼ 0] ¼ (1 p) þ kp(1 p) where k may be interpreted as a coefficient of interobserver agreement. This model is often referred as the ‘common correlation model’ since it assumes that ki ¼ k, i ¼ 1, 2, . . . , k. It will also be shown in Section 3.3 that the model in Equation (18) is a special case of the beta-binomial distribution discussed by Haseman and Kupper.27 If the observed frequencies are given as in Table 7, the k may be estimated by
k^ ¼
where p^ ¼
Pk P2 i¼1
j¼1 yij =(2k)
1 n2 2kp^ (1 p^ )
(19)
¼ (2n1 þ n2 )=(2k) is the sample estimate of p.
Table 7 Frequencies for the two-rater agreement case Category
Frequency
Probability
(1,1) (1,0) or (0,1) (0,0) Total
n1 n2 n3 n
P1(k) P2(k) P3(k) 1
Q2
16 MM Shoukri, MH Asyali and A Donner The estimator k^ has been shown under this model to be the maximum likelihood estimator of k, with large sample variance given by28
(1 k) k(2 k) var(k^ ) ¼ (1 k)(1 2k) þ (20) k 2p(1 p) Donner and Eliasziw29 used the goodness-of-fit test procedure to facilitate sample size calculations that may be used to enroll a sufficient number of subjects in a study of interobserver agreement involving two raters. They showed that the number of subjects needed to test H0: k ¼ k0 versus H1: k ¼ k1 is given by ( )1 [p(1 p)(k1 k0 )]2 2[p(1 p)(k1 k0 )]2 [p(1 p)(k1 k0 )]2 k¼A þ (21) þ p2 þ p(1 p)k0 p(1 p)(1 k0 ) (1 p)2 þ p(1 p)k0 2
where A2 ¼ (z1a=2 þ z1b )2 . Example 2 Suppose that it is of interest to test H0: k0 ¼ 0.60 versus H1: k0 6¼ 0.60 where k0 ¼ 0.60 corresponds to the value of kappa characterized as representing ‘substantial’ agreement.30 To ensure with 80% probability a significant result at a ¼ 0.05 and p ¼ 0:30 when k1 ¼ 0.90, the required number of subjects from Equation (21) is k ¼ 66. In Table 8, we present some values of the required number of subjects for different values of k0, k1, p, a ¼ 0.05 and 1 7 b ¼ 0.80. 3.1.1 Specified width of a CI As in case of a continuous measurement, we may base our sample size calculation on the required width of a CI. Suppose that an interobserver agreement study is to be Table 8 Number of required subjects k at a ¼ 0.05 and b ¼ 0.20 k1
k0
p 0.4
0.4 0.6 0.7 0.8 0.9
0.1 0.3 0.5 0.1 0.3 0.5 0.1 0.3 0.5 0.1 0.3 0.5 0.1 0.3 0.5
404 190 165 179 84 73 101 47 41 64 30 26
0.6
0.7
0.8
0.9
334 148 126
121 52 45 1090 474 400
49 21 18 195 83 71 770 336 282
17 7 6 46 20 17 103 44 37 413 177 149
1339 595 502 335 148 125 149 66 55
1090 474 400 272 118 100
779 336 282
Design of reliability study 17 conducted such that a CI for the intraclass kappa statistic given by Equation (19) has a desired width w. Setting w ¼ 2za=2 {var(k^ )}1=2 where var(k^ ) is given by Equation (20), replacing k by a planned value k , and solving for k, the required number of subjects is given by k¼
4z2a=2
w2
k (2 k ) (1 k ) (1 k )(1 2k ) þ 2p(1 p)
(22)
Example 3 Suppose that an interobserver agreement study involving two raters is designed to achieve a value of k ¼ 0.80. We also assume that the probability of positive rating is 0.30, the desired width of the CI is w ¼ 0.20, with level of confidence 0.95. Then we have k¼
4(1:64)2 (0:2)2
(0:8)(1:2) (0:2) (0:2)(0:6) þ ¼ 117 2(0:3)(0:7)
3.2 E⁄ciency requirements We now focus attention on seeking the allocation that minimizes the variance of the estimator of r for fixed N ¼ nk when the outcome measure is binary. Let yij denote the jth rating made on the ith subject, where yij ¼ 1 if the condition is present and 0 otherwise. Landis and Koch,30 analogous to the continuous case, employed the one-way random effects model yij ¼ mi þ eij
(23)
where mi ¼ mi þ si for i ¼ 1, 2, . . . , k; j ¼ 1, 2, . . . , n. Analogous to the case of a continuous outcome measure, we assume the {si} are idd with mean 0 and variance s2s , the {eij} are idd with mean 0 and variance s2e , and that the {si} and {eij} independent. We may therefore write E(yij ) ¼ p ¼ Pr[yij ¼ 1] and s2 ¼ var(yij ) ¼ p(1 p)
(24)
Letting d ¼ Pr[yij ¼ 1, yil ¼ 1] ¼ E(yij yil ), it follows for j 6¼ l and i ¼ 1, 2, . . . , k that d ¼ cov(yij , yil ) þ E(yij yil ) ¼ rp(1 p) þ p2
(25)
where r ¼ (d p2 )=[p(1 p)]. It is clear from the previous set up that the probability that two measurements taken from the same subject are in agreement is Po ¼ p2 þ (1 p)2 þ 2rp(1 p). Substituting r ¼ 0 in Po, we obtain agreement by chance as, Pe ¼ p2 þ (1 p)2 . Therefore, the beyond chance agreement is given by
18 MM Shoukri, MH Asyali and A Donner k ¼ (Po Pe )=(1 Pe ) ¼ r. It is therefore clear that r is directly analogous to the components of kappa described by Cohen31 and Fleiss and Cohen.32 Following Landis and Koch,30 let s2s ¼ rp(1 p) s2e ¼ (1 r)p(1 p)
(26)
denote the variance components of yij. Then s2 ¼ s2s þ s2e and the corresponding estimator of r is given, analogous to Equation (2), by r ¼
MSA MSW MSA þ (n 1)MSW
where 8 P 2 9 k > > Pk 2 < = i¼1 yi 1 i¼1 yi MSA ¼ , > k 1> nk : n ; ( Pk 2 ) k n X X 1 i¼1 yi y yij and yi ¼ MSW ¼ k(n 1) i¼1 i n j¼1 We note first that the statistic r* depends only on the total yi, and not on the individual binary responses. Crowder33 and Haseman and Kupper27 demonstrated the equivalence of the ANOVA model given earlier to the well known beta-binomial model which arises when conditional on the subject effect mi, the subject’s total yi has a binomial distribution with conditional mean and variance given, respectively, by E(yi j mi ) ¼ nmi and var(yi j mi ) ¼ nmi (1 mi ). The parameter mi is assumed to follow the beta distribution f (mi ) ¼
G(a þ b) a1 mi (1 mi )b1 G(a)G(b)
(27)
where a ¼ p(1 r)=r, and b ¼ (1 p)(1 r)=r. Therefore, the ANOVA model and the beta-binomial model are virtually indistinguishable.34 Because the optimal number of replicates for the non-normal case under the former model was shown to be n ¼ 1 þ 1={r(1 þ gs )1=2 } and since gs is the kurtosis of the subject effect distribution, one may use the kurtosis of the beta distribution to determine the optimal number of replications.
Design of reliability study 19 One can derive gs for the beta distribution from the recurrence given by m01 ¼ p1 m0l
m0l1
¼
ðl 1Þr þ pð1 rÞ , l ¼ 2, 3, . . . 1 þ ðl 1Þr
where m0l ¼ E[mli ]. Then gs ¼ m4 =(m2 )2 , where m4 ¼ m04 4m03 m01 þ 6m02 (m01 )2 3(m01 )4 and m2 ¼ m02 (m01 )2 .21 Substituting gs into Equation (16) we obtain 1=2 (1 r)(1 2r) n ¼ 1 þ p(1 p) c(p, r)
(28)
where c(p, r) ¼ p[r þ p(1 r)][2r þ p(1 r)][3r þ p(1 r) 4p(1 þ 2r)] þ (1 þ r)(1 þ 2r)[6p3 (1 p)r þ 3p4 þ p2 (1 p)2 r2 ] Table 9 shows the optimal number of replications n* and the corresponding optimal number of subjects k ¼ N=n*. In contrast to the continuous measurement model, the optimal allocation in the case of a binary outcome measure depends on the mean of the response variable p. We also note that for fixed N the optimal allocations are equivalent for p and 1 p. Remarks 1. When p is small, as few as two replicates are required, but with a corresponding larger value for the required number of subjects. When p ¼ 0:5, a fewer number of subjects should be recruited with not more than three replicates per subject. 2. There is a noted similarity between the results given in Tables 4 and 9. In both cases, higher values of r imply that as few as n ¼ 2 replicates are needed and hence, a larger number of subjects should be recruited. In particular, when p ¼ 0:5 and 0.6 r 0.8, the optimal allocation for the case of binary outcome measure is close to that required in the case of a continuous outcome measure. Table 9 Optimal allocation at N ¼ 60 for a binary response variable r
p 0.4
0.1 0.3 0.5
0.5
0.6
0.7
0.8
0.9
n
k
n
k
n
k
n
k
n
k
n
k
1.81 2.36 2.53
33 25 24
1.64 2.10 2.25
37 29 27
1.53 1.94 2.08
39 31 29
1.46 1.82 1.95
41 33 31
1.40 1.73 1.85
43 35 32
1.36 1.65 1.77
44 36 34
20 MM Shoukri, MH Asyali and A Donner 4
Discussion
A crucial decision facing a researcher in the design stage of a reliability study is the determination of the required values of n and k. When the investigator has prior knowledge of what is regarded as an acceptable level of reliability, the hypothesis testing approach may be used and sample size calculations can be performed using the results of Donner and Eliasziw10 and Walter et al.7 However, in many cases, values of the reliability coefficient under the null and alternative hypotheses may be difficult to specify. Moreover, the estimated value of ICC depends on the level of heterogeneity of the sampled subjects: the more the heterogeneity, the higher the value of ICC. As most reliability studies focus on estimation of the ICC, a principal aim of this review is to provide the values (n, k) that maximizes the precision of the estimated ICC. It is fortuitous that this approach produces estimates of sample size that are in close agreement with results from procedures based on power considerations. An overall conclusion from the earlier results is that for both continuous and binary outcome measures, the variance of the estimated ICC is minimized with only a small number of replicates provided the true value of the ICC is reasonably high. In many clinical investigations an ICC at least 0.60 is required as the minimal acceptable value. Under such circumstances, one can safely recommend only two or three replications per subject. Finally, it is noted that in practice the optimal allocations must be integer values, and that the net loss=gain in precision as a result of rounding the values of (n, k) is negligible. Ideally one should adopt one of the available combinatorial optimization algorithms, often referred to as integer programming models. These models are suited for the optimal allocations problems that we reviewed in this study, as the main concern was to find the best solution(s) in a well defined discrete space. This topic needs further investigation.
References 1 2
3
4 5
Dunn G. Design and analysis of reliability studies. Statistical Methods in Medical Research 1992; 1: 123–57. Shoukri MM. Agreement. In Armitage P, Colton T, eds Encyclopedia of biostatistics. New York: John Wiley & Sons, 1999: 117–30. Shoukri MM. Agreement. In Gail MH, Benichou J, eds Encyclopedia of epidemiologic methods. New York: John Wiley & Sons, 2000: 43–49. Haggard ER. Intraclass correlation and the analysis of variance. New York: Dryden Press, 1958. Elston R. Response to query: estimating ‘heritability’ of a continuous trait. Biometrics 1977; 33: 232–33.
6 Fisher RA. Statistical methods for research workers. London: Oliver & Boyd, 1925. 7 Walter DS, Eliasziw M, Donner A. Sample size and optimal design for reliability studies. Statistics in Medicine 1998; 17: 101–10. 8 Freedman LS, Parmar MKB, Baker SG. The design of observer agreement studies with binary assessments. Statistics in Medicine 1993; 12: 165–79. 9 Awni WM, Skaar DJ, Schwenk MH. Interindividual and intraindividual variability in labetalol pharmacokinetics. Journal of Clinical Pharmacology 1988; 28: 344–49. 10 Donner A, Eliasziw M. Sample size requirements for reliability studies. Statistics in Medicine 1987; 6: 441–48.
Design of reliability study 21 11 Eliasziw M, Donner A. A cost-function approach to the design of reliability studies. Statistics in Medicine 1987; 6: 647–55. 12 Giraudeau B, Mary JY. Planning a reproducibility study: how many subjects and how many replicates per subject for an expected width of the 95 per cent confidence interval of the intraclass correlation coefficient. Statistics in Medicine 2001; 20: 3205–14. 13 Bonett DG. Sample size requirements for estimating intraclass correlations with desired precision. Statistics in Medicine 2002; 21: 1331–35. 14 Donner A. Sample size requirements for interval estimation of the intraclass kappa statistic. Communications in Statistics – Simulation 1999; 28: 415–29. 15 Shoukri MM, Asyali MH, Walter SD. Issues of cost and efficiency in the design of reliability studies. Biometrics 2003; 59: 1107– 12. 16 Rao SS. Optimization: theory and applications. New Delhi: Wiley Eastern Limited, 1984. 17 Flynn NT, Whitley E, Peters T. Recruitment strategy in a cluster randomized trial: cost implications. Statistics in Medicine 2002; 21: 397–405. 18 Sukhatme PV, Sukhatme BV, Sukhatme S, Asok C. Sampling theory of surveys with applications. Ames, IA: Iowa State University Press, 1984. 19 Scheffe´ H. The analysis of variance. New York: John Wiley & Sons, 1959. 20 Tukey JW. Variance of variance components I: balanced designs. Annals of Mathematical Statistics 1956; 27: 722–36. 21 Kendall M, Stuart A. The advanced theory of statistics, Vol 1. London: Griffin, 1986. 22 Hemmersley IM. The unbiased estimate and standard error of the intraclass variance. Metron 1949; 15: 189–205.
23 24
25
26 27
28 29
30 31 32
33 34
Kraemer CH. Ramification of a population model for k as a coefficient of reliability. Psychometrika 1979; 44: 461–72. Frederic R, Gaspoz JM, Fortune IC, Minini P, Pichot V, Duverney D et al. Screening of obstructive sleep apnea syndrome by heart rate variability analysis. Circulation 1999; 100: 1411–15. Donner A, Eliasziw M. Statistical implications of the choice between a dichotomous or continuous trait in studies of interobserver agreement. Biometrics 1994; 50: 550–55. Mak TK. Analyzing intraclass correlation for dichotomous variables. Applied Statistics 1988; 20: 37–46. Haseman JK, Kupper LL. Analysis of dichotomous response data from certain toxicological experiments. Biometrics 1979; 35: 281–93. Bloch DA, Kraemer HC. 2 2 Kappa coefficients: measures of agreement or association. Biometrics 1989; 45: 269–87. Donner A, Eliasziw M. The goodness-of-fit approach to inference procedures for the kappa statistic: confidence interval construction, significance testing, and sample size estimation. Statistics in Medicine 1992; 11: 1511–19. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977; 33: 159–74. Cohen JA. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 1960; 20: 37–46. Fleiss J, Cohen JA. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Measurement 1973; 33: 613–19. Crowder M. Beta-binomial ANOVA for proportions. Applied Statistics 1978; 27: 34–37. Cox DR, Snell EJ. Analysis of binary data. London: Chapman and Hall, 1989.
Q3
ARNOLD JOURNALS JOURNAL…SMMR… …
ARTID…SM365RA……….
TO: CORRESPONDING AUTHOR AUTHOR QUERIES - TO BE ANSWERED BY THE AUTHOR The following queries have arisen during the typesetting of your manuscript. Please answer these queries. Q1
Please specify the country for the currency `$1600’. For example `US$’ etc.
Q2
The article does not contain Section 3.3 but it is cross-referred in text. Please check.
Q3
Please supply all authors in reference [24].
Journals Offprint Order Form A member of the Hodder Headline Group Arnold Journals 338 Euston Road London NW1 3BH Tel: +44 (0)20 7873 6000 Fax: +44 (0)20 7873 6376
This form should be returned at once to the above address • Title of Journal: • 1st Author: A. FREE OFFPRINTS - 25 offprints of your article will be supplied free of charge Please indicate opposite, the name and full postal address to whom they should be sent. In the case of multi-author articles, free offprints are only sent to the corresponding author.
Statistical Methods in Medical Research 13(4) [provisional] ____________________________________________ ____________________________________________ ____________________________________________ ____________________________________________
B. PURCHASE OF ADDITIONAL OFFPRINTS Please note that if an article is by more than one author, only one offprint form is sent and all offprints should be ordered on that form in consultation with the co-authors. Offprint Price List (£ sterling UK and Europe; US$ Rest of World) 25 1-4 pages 5-8 pages 9-16 pages 17-24 pages Extra 8 pages
£ 76 100 131 149 15
50 $ 122 160 210 238 24
£ 99 131 149 173 22
100 $ 158 210 238 277 35
£ 149 200 227 250 26
150 $ 238 320 363 400 42
£ 227 276 301 357 32
$ 363 442 482 571 51
200 £ 273 359 399 454 45
$ 437 574 638 726 72
For larger quantities contact the publisher for a quotation. Add 100% for any offprints including colour reproduction.
IMPORTANT 1. Cheques drawn on a UK or US bank should be made payable to Hodder Headline Group. We are unable to accept credit or debit card payments. 2. Orders will not normally be mailed until the publisher is in receipt of either the appropriate payment or an official purchase order. 3. The above are prepublication prices and apply only to orders received before the publication goes to press. 4. All despatches are by surface mail, normally within four weeks of publication. 5. Claims cannot be considered more than three months after despatch. VAT will be added to UK invoices. Members of the EU will be required to pay VAT unless a VAT number is provided with order. VAT Number: ……………………………………………………..
I wish to purchase .........................additional offprints ADDRESS FOR DELIVERY (please print in capitals) __________________________________________________
_____________________________________________ _____________________________________________ ____________________________________________
ADDRESS FOR INVOICE (please print in capitals)
________________________________ ________________________________ ________________________________ ________________________________ Payment enclosed Please invoice Official order follows Official order attached Order no.
...................................................
Signed.......................................................... Date........................../................../..................
TRANSFER OF COPYRIGHT Please return completed form to: Arnold Journals Statistical Methods in Medical Research 338 Euston Road London NW1 3BH UK.
A ARNOLD
A member of the Hodder Headline PLC Group
STATISTICAL METHODS IN MEDICAL RESEARCH In consideration of the publication in the above journal Statistical Methods in Medical Research the contribution entitled:................................................................................................................................ .................................................................................................................................. ("the contribution") by (all authors' names): ............................................................................................................................... 1.
To be filled in if copyright belongs to you I/we warrant that I am/we are the sole owner/s of the complete copyright in the Contribution and I/we hereby assign to Arnold (Publishers) Limited the complete copyright in the Contribution in all formats and media.
2.
To be filled in if copyright does not belong to you
a)
Name and address of copyright holder............................................................................................ ............................................................................................................................................................. .................................................................................................................................................... I/we warrant that I am/we are the sole owner/s of the complete copyright in the Contribution and I/we hereby grant Arnold (Publishers) Limited the non-exclusive right to publish the Contribution throughout the world in all formats and media and to deal with requests from third parties in the manner specified in paragraphs 2 and 4 overleaf.
b)
3.
To be filled in if US Government exemption applies I/we certify that the Contribution was written in the course of employment by the United States Government, and therefore copyright protection is not available.
4.
I/we warrant that I/we have full power to enter into this Agreement, and that the Contribution does not infringe any existing copyright, or contain any scandalous, defamatory, libellous or unlawful matter. Signed as (tick one)
the sole author(s) of the Contribution one author authorised to execute this transfer on behalf of all the authors of the Contribution the copyright holder or authorised agent of the copyright holder of the Contribution
Name (block letters) ........................................................................................................................... Address .............................................................................................................................................. Signature ................................................................................. Date .................................................. (Additional authors should provide this information on a separate sheet please)
Notes for Contributors 1.
The Journal's policy is to acquire copyright in all contributions. There are two reasons for this: (a) ownership of copyright by one central organisation tends to make it easier to maintain effective international protection against unauthorised use; (b) it also allows for requests from third parties to reprint or reproduce a contribution, or part of it, to be handled in accordance with a general policy which is sensitive both to any relevant changes in international copyright legislation and to the general desirability of encouraging the dissemination of knowledge.
2.
Arnold co-operates in various licensing schemes which allow organisations to copy material within agreed restrains (e.g. the CLA in the UK and the CCC in the USA).
3.
All contributors retain the rights to reproduce their paper for their own purposes provided no sale is involved, and to reprint their paper in any volume of which they are editor or author. Permission will automatically be given to the publisher of such a volume, subject to the normal acknowledgement.
4.
It is understood that in some cases copyrights will be held by the contributor's employer. If so, Arnold requires non-exclusive permission to deal with requests from third parties, on the understanding that any requests it receives will be handled in accordance with paragraph 3.
5.
Arnold will provide each contributor with a complimentary copy of the issue of the Journal in which the Contribution appears.
