COMMENTS ON SPECIES SENSITIVITY DISTRIBUTIONS SAMPLE SIZE DETERMINATION BASED ON NEWMAN ET AL. (2000) Frederik A.M. Verdonck1*, Joanna Jaworska2 and Peter A. Vanrolleghem1 1
2
Ghent University, Department of Applied Mathematics, Biometrics & Process Control (BIOMATH), Coupure Links 653, B-9000 Gent, Belgium (*Phone: +32(0)9/264.59.37, Fax: +32(0)9/264.62.20,
[email protected]) Procter & Gamble, ETC, Temselaan 100, B-1853 Strombeek-Bever, Belgium
Introduction After introduction of species sensitivity distributions (SSD) of laboratory derived ecotoxicity data to assess effects on the ecosystems and to derive environmental quality criteria by Van Straalen and Denneman (1989), the method has been extensively debated. The most common current approach is to derive Predicted No Effect Concentration from the median 5th percentile of SSD (EU-TGD 1995). Historically that value is known as Hazardous Concentration at p-protection level or HCp. The debates evolved around ecological significance of p–level and statistical and ecological requirements for sample size. In the determination of the optimal sample size, a balance between representativeness of the ecosystem taxonomic groups at small samples against unnecessary costs of collecting excessive data should be found. In this commentary we analyze the recent approach of Newman et al. (2000) to find optimal sample size and focus only on statistical considerations pertaining to accuracy of the prediction of the pth-percentile. Newman et al. (2000) describe a method for determining a sufficient number of species in SSD's using a modified version of bootstrapping. The novel modification concerns use of resample size bigger than the actual sample. Their results are: "Approximate optimal sample sizes for HC5 estimation ranged from 15 to 55 with a median of 30 species-sensitivity values. Similar sample sizes were needed for HC10 and HC20 estimation: estimates ranged from 10 to 75. …These sample sizes are much higher than those recommended as acceptable for regulatory purposes". Since indeed these numbers are rather high and different from existing recommendations of 5-8 depending on the source, we investigated Newman’s approach both theoretically and numerically.
Theoretical considerations of bootstrapping with replacement The general approach in bootstrap simulation is to assume a distribution which describes the quantity of interest, to perform x replications of the data set of n by randomly drawing, with replacement, m = n-1 or smaller values, and then calculate x values of the statistic of interest (Efron and Tibshirani 1993). However Manly (1992) writes: "… One of the key aspects of bootstrapping is that samples are taken with replacement rather than without replacement. Hence, a pilot sample of size P (here named n) can be bootstrap-sampled to produce samples of any size, including sizes that are > P (here named m) (Bickel & Freedman, 1981).…". This is somewhat misleading as Bickel and Freedman (1981) only make a statement on bootstrapping statistics when m and n are varied separately under the condition that both m and n have to tend to infinity i.e. they study the asymptotic properties only.
Newman et al. (2000) approach Newman et al. (2000) modified the bootstrap by taking more resamples m than the actual, original data set size n. In Newman et al. (2000)'s study, the resample size varied between 5 and 100 in increments of 5. For each resample size, the HC5, 10 and 20 and their 95% confidence interval were calculated and plotted (Figure 1 for HC20). Logically, the confidence interval around the HCp estimate decreased as resample size increased. The resample size where no further visual improvement was found in narrowing of a confidence interval (note the subjectivity) was taken as the optimal m. Newman et al. (2000) set the optimal sample size n equal to the optimal resample size m found from such bootstrap study.
1
Numerical examples The Newman et al. (2000) procedure was investigated in two ways: first for or the convergence of the confidence intervals, as the resample size increases; second, for the consistency of the method by varying the initial sample size n.
Concentration (mg Zn/kg)
First, the procedure was repeated for 60 data points for a Zn toxicity data set. The data only serve as an example, data quality and other SSD issues are not discussed in this short paper. The resample size varied from 5 to 1000 in order to investigate the convergence of the confidence intervals (Fig.1). The optimal resample size of 65 was selected as Newman’s (2000) did by a visual assessment. 160
95% (lower confidence limit)
140
50% (lower confidence limit)
120
5% (lower confidence limit)
HC20
100 80 60 40 20 0 0
20
40
60
80
100
Resample size m Figure 1: Curves for estimating sample size (HC20 as illustrated with the Zn data (60 points). The symbols indicate the HCp values ranked at the 50% (□), 5% (∆), and 95% (X) of the 10,000 values generated by bootstrapping
Numerical simulations show that as the resample size increases (becomes 1000), the confidence intervals converge to zero (Fig.2). In other words, the ideal resample size m would be infinity because the uncertainty would become zero and since these are numerical simulations, it is no issue to resample as much as possible. This is in conflict with confidence interval theory that shows that there is always uncertainty, even at infinite sample size. Second, suppose the original data set contained 30 samples instead of 60 (30 samples were ad random removed), and the entire procedure was repeated, what would then be the optimal (re)sample size? Using the same criteria as above 40 is a good sample size (Fig 2). So, starting with 60 data points leads to an optimal sample size of 65, whereas starting from half of these data leads to an optimal sample size of 40 . This indicates that the method for optimal sample size determination is not consistent and starting from different initial conditions leads to different conclusions. This is not due to lack of a statistical criterion for a cutoff. Using a possible criterion of HC2095/HC2050 < 2 the recommended numbers would be 39 for 30 points and 360 for 60 points. 160 95% (lower confidence limit)
Concentration (mg Zn/kg)
140
50% (lower confidence limit)
120
HC20
5% (lower confidence limit)
100 80 60 40 20 0 0
20
40
60
Resample size m
80
100
200
300
400
500
600
700
800
900
1000
Resample size
Figure 2: Curves for estimating sample size for (HC20) illustrated with the limited Zn data (30 points).
2
Conclusions No theoretical background was found to support the approach of Newman et al. (2000). Further, numerical examples show that when the resample size exceeds the sample size logical and statistical inconsistencies arise. Indeed, one cannot get more information from a data set than the data itself contains. After this analysis the authors feel that Newman’s et al. (2000) recommendations should not be considered.
Acknowledgements This research has been funded by a scholarship from the Flemish Institute for the Improvement of Scientific-Technological Research in the Industry (IWT).
References Bickel, P. J., and Freedman, D. A. (1981). Some asymptotic theory for the bootstrap. The Annals of Statistics 9, 1196-1217. Efron, B., and Tibshirani, R. J. (1993). An introduction to the bootstrap. Chapman & Hall, New York. EU-TGD (1995). Environmental risk assessment of new and existing substances. Technical Guidance Document. Manly, B. F. J. (1992). Bootstrapping for determining sample sizes in biological studies. Journal of Exp. Mar. Biol. Ecol. 158, 189-196. Newman, M. C., Ownby, D. R., Mézin, L. C. A., Powell, D. C., Christensen, T. R. L., Lerberg, S. B., and Anderson, B.-A. (2000). Applying species-sensitivity distributions in ecological risk assessment: assumptions of distribution type and sufficient number of species. Environmental Toxicology and Chemistry 19, 508-515. Van Straalen, N. M., and Denneman, C. A. J. (1989). Ecotoxicological evaluation of soil quality criteria. Ecotoxicological Environmental Safety 18, 241-251.
3
