Two forms of grey literature important for reviews in ...

5 downloads 2880 Views 295KB Size Report
[email protected].uk (H.R. Bayliss). BIOC-06497; No of Pages 3 ..... Contact with research producers via personal contacts, email lists or social networks ...
BIOC-06497; No of Pages 3 Biological Conservation xxx (2015) xxx–xxx

Contents lists available at ScienceDirect

Biological Conservation journal homepage: www.elsevier.com/locate/bioc

Discussion

Shades of grey: Two forms of grey literature important for reviews in conservation Neal R. Haddaway a,⁎, Helen R. Bayliss b a b

Mistra Council for Evidence-based Environmental Management (EviEM), Royal Swedish Academy of Sciences, Box 50005, SE-104 05 Stockholm, Sweden Centre for Evidence-based Conservation, School of the Environment, Natural Resources and Geography, Bangor University, Bangor LL57 2UW, UK

a r t i c l e

i n f o

Article history: Received 13 April 2015 Received in revised form 2 August 2015 Accepted 8 August 2015 Available online xxxx Keywords: Unpublished Synthesis Meta-analysis Systematic reviews Systematic maps Evidence-based conservation

a b s t r a c t Methods for reviewing research, such as systematic reviews and syntheses, are becoming increasingly common in conservation. It is widely recognised that grey literature, research not published in traditional academic journals, forms a vital part of the evidence base of these reviews. To date guidance and practice in searching for and including grey literature in conservation reviews has taken a broad approach, involving searching of a wide variety of resources. We argue that there are two distinct forms of grey literature and that each must be considered separately in order to assess potential importance and an appropriate searching strategy for every review undertaken. ‘File drawer’ research is as yet unpublished academic research that is important for countering possible publication bias and can be targeted via specific repositories for preprints, theses and funding registries, for example. ‘Practitioner-generated research’ includes organisational reports, government papers and monitoring and evaluation reports, and is important for ensuring comprehensiveness in conservation reviews. By considering the relative importance and appropriate strategies for inclusion of both types of grey literature, reviewers can optimise resource efficiency and comprehensiveness, and minimise publication bias. © 2015 Elsevier B.V. All rights reserved.

1. Introduction Methods for reviewing research (for example meta-analyses, systematic reviews or synopses) are increasingly being used to identify trends across studies and generate recommendations for evidencebased environmental management (Sutherland et al., 2004; Pullin, 2012; Lortie, 2014; Dicks et al., 2014). These methods may be particularly valuable for conservation, where standard features of experimental design, for example sufficient replication or the use of control sites, may be infeasible or unethical in practice. By drawing on a large number of similar studies, generalised effects can be identified across studies, and any factors affecting the implementation of interventions (the reasons for heterogeneity between studies) can be investigated (Glasziou and Sanders, 2002). Identifying the circumstances under which an intervention is most likely to be successful can help direct effective resource use and ensure the best outcomes possible for conservation (Sutherland, 2005). The wider the pool of relevant studies included in a review, the greater the ability to identify trends and patterns in the data (i.e., statistical power of the analyses), meaning that they can more reliably inform conservation policy and practice. Early meta-analyses in ecology, conservation and environmental sciences were occasionally criticised

⁎ Corresponding author. E-mail addresses: [email protected] (N.R. Haddaway), [email protected] (H.R. Bayliss).

for their lack of robust, repeatable methods for identifying suitable materials for inclusion, meaning that they were susceptible to missing influential information (Gates, 2002). If syntheses are based on only a subset of the available literature in this way, their reliability is compromised. One major criticism of early meta-analyses related to publication bias (the tendency for papers presenting statistically significant, positive results to be more likely to be published than those that report non-significant, replicated or conflicting results (e.g., Gurevitch and Hedges, 1999; Lortie et al., 2007) (Fig. 1)) where positive and significant research is over-represented in the published corpus of research (Smith, 1980). Guidance for undertaking systematic reviews has been published aiming to improve the reliability of literature reviews in conservation (Pullin and Stewart, 2006). This guidance recommends that, along with critically appraising all included literature for its reliability, reviewers search both the scientific literature and the ‘grey’ or noncommercially published literature to ensure that a comprehensive and representative sample of available research is retrieved. For many aspects of applied conservation, grey literature may constitute a significant proportion of the available documented support for management decisions (Corlett, 2011), and whilst much of this may not have been subject to academic-style peer-review, it may hold vital evidence for conservation practitioners. A range of techniques (for example, using internet search engines and organisational repositories and websites) has been proposed to facilitate the inclusion of grey literature in reviews (e.g., CEE, 2013; Bayliss and Beyer, 2014). However, we argue that blanket recommendations to include grey literature mask two different issues with potentially different solutions, and that a one-size-fits-all

http://dx.doi.org/10.1016/j.biocon.2015.08.018 0006-3207/© 2015 Elsevier B.V. All rights reserved.

Please cite this article as: Haddaway, N.R., Bayliss, H.R., Shades of grey: Two forms of grey literature important for reviews in conservation, Biological Conservation (2015), http://dx.doi.org/10.1016/j.biocon.2015.08.018

2

N.R. Haddaway, H.R. Bayliss / Biological Conservation xxx (2015) xxx–xxx

help identify research that, due to publication lag times, is not yet available in the scientific literature. 3. Searching for research outside academia

Fig. 1. An Egger Plot of study sample sizes against effect sizes, with each square on the plot representing one study. The figure shows an example of evidence of publication bias, where larger (more accurate) studies lie at the top of the triangle, whilst smaller, more positive studies lie lower down (visualised as vertical and horizontal asymmetry). The shaded polygon (bottom left) highlights a dearth of smaller, negative studies where larger, negative studies exist (top left). If publication bias is responsible for this dearth, we might expect to populate this part of the graph using the effect sizes of studies retrieved though searches of un-commercially published academic literature such as theses, conference proceedings or pre-prints.

approach to information retrieval in conservation syntheses fails to adequately address either problem. 2. Countering publication bias The inclusion of grey literature during the searching phase of a review is frequently suggested as a way to help counter the issue of publication bias (e.g., CEE, 2013; Bayliss and Beyer, 2014). Publication bias results in an unrepresentative set of studies, and has been estimated to affect the conclusions of between 15–21% of ecological metaanalyses (Jennions and Møller, 2002). There is some uncertainty as to the true extent of the problem, as a study of Finnish and Swedish theses and their subsequent publication success in journals found no significant evidence of bias (Koricheva, 2003). In reality, there are a range of publication-related biases that may operate at the study level (e.g., relating to the outcome of statistical tests or agreement with the hypothesis (e.g., Lortie et al., 2007)), at the manuscript level (e.g., relating to the length, the number, and gender of authors and their motivation to publish (e.g., Koricheva, 2003; Lortie et al., 2007)), and at the journal level (e.g., reputation, cost of submission or access, and visibility (e.g., Lortie et al., 2007)), that are likely to affect conservation studies. These may be applied, unintentionally or otherwise, by the authors or by external actors such as editors or peer-reviewers (Lortie et al., 2007). Since many meta-analyses currently fail to test for publication bias (Koricheva and Gurevitch, 2014), the true extent of the problem for ecological syntheses may remain unknown. To mitigate these potential sources of bias, authors of syntheses are advised to attempt to identify unpublished scientific studies in addition to conducting searches of academic databases (e.g., Pullin and Stewart, 2006; CEE, 2013). A systematic approach to these searches can greatly increase the reliability of reviews and meta-analyses. This can be achieved by performing searches for unpublished research in several key types of resource, namely: catalogues of academic theses (i.e., master's and doctoral dissertations); databases that include conference proceedings; pre-print servers (e.g., biorxiv.org); funders' databases of on-going research; and calls for submission of evidence to relevant scientific communities. The use of these sources could potentially also

A second issue, and one that is particularly relevant to conservation syntheses, is the inclusion of practitioner-generated research (i.e., research undertaken by conservation managers). Syntheses exploring topics in conservation and environmental management, such as the effectiveness of commonly used interventions or the impacts of certain factors, may require data that are more practical, applied and repetitive than those produced through academic research, and are more likely to be generated by those undertaking the activity in the field; for example land managers, environmental consultants and non-governmental conservation organisations. However, many practitioners are not motivated by academic publication, and resource constraints may pose considerable barriers to those that are (Sutherland et al., 2013). Furthermore, there may be reasons not to make research publically available. For example, organisations and individuals with commercial interests, such as consultancies, may actively avoid making information publicly available for reasons of competition, and relevant datasets are likely to be held privately by the organisations involved. Alternatively, organisations may wish to withhold detailed information, such as records for populations of critically endangered species that may be negatively affected by publication of research findings. Thus, many practitioner-generated studies are not easily accessible using standard search techniques, and so reviews must be based on the best evidence available to authors rather than the best evidence (Pullin and Salafsky, 2010). Reviewers attempting to retrieve practitioner-generated studies should search specialised sources in a systematic manner. Such resources include: the journal Conservation Evidence (accessible via www.conservationevidence.com) that is tailored specifically to practitioner-generated studies; academic search engines such as Google Scholar that catalogue both academic and grey literature; websites of relevant organisations that may produce non-academic reports; specialist online catalogues of grey literature (e.g., http://ea-lit. freshwaterlife.org/; www.opengrey.eu): and calls for evidence from relevant organisations, groups or networks. Direct contact with practitioners working in organisations that are likely to collect relevant data may be more effective than extensive online searches (Bayliss and Beyer, 2014), but an approach that utilises multiple resources will be more reliable. The exact choice of sources to be searched is likely to be very context-specific and will depend on the subject area, geographic scope and the sector (i.e., commercial, public, charitable) involved. Finally, reviewers should carefully consider the implications of language in searching for practitioner-generated research, since there is a greater likelihood that this may be published in non-English languages. 4. Implications for future syntheses We suggest three key steps for authors to consider when assessing the role of grey literature in their reviews. Firstly, reviewers should assess whether there is a need to include both of the above types of grey literature to ensure that a comprehensive dataset is available to address their review question (Table 1). In some cases, searches for research outside academia may be highly unlikely to yield results and efforts can be minimised. For example, this may be the case where there are significant commercial interests, such as with genetically modified organisms, or where sampling equipment is particularly expensive, such as that for measuring greenhouse gas fluxes. Secondly, researchers should plan their searching efforts based on the types of grey literature that they wish to retrieve. Table 1 provides some examples of resources that are specifically tailored to the two forms of evidence described herein. Thirdly, authors should describe and justify efforts made to retrieve grey literature when writing up the findings of their review in order to maximise transparency. Such transparency is a cornerstone of

Please cite this article as: Haddaway, N.R., Bayliss, H.R., Shades of grey: Two forms of grey literature important for reviews in conservation, Biological Conservation (2015), http://dx.doi.org/10.1016/j.biocon.2015.08.018

N.R. Haddaway, H.R. Bayliss / Biological Conservation xxx (2015) xxx–xxx

3

Table 1 When and how to search for the two major forms of grey literature to counter publication bias or retrieve research produced outside of academia. File drawer research (countering publication bias)

Practitioner-generated research (searching for research outside academia)

• When a subject area is relatively novel and may not be widely used in practice • When little evidence exists but a theory is still widely supported • When challenges to accepted dogma are unlikely to be readily accepted • When competition within the field is such that risks of questionable research practices (QRP) are high • When research is likely to be undertaken by students and unlikely to be written up. How to search (examples) • Pre-print servers (e.g., arxiv.org/biorxiv.org) • Author contact (corresponding author contacts/social networking sites/mail servers) • Institutional repositories (e.g., opendoar.org) • Databases of theses (e.g., ethos.bl.uk, www.theses.com) • References to unpublished research within citations • Funders registers of research (e.g., nerc.ac.uk), • Specialist grey literature repositories (e.g., opengrey.eu). When to include searches

systematic review methods (e.g., CEE, 2013) and significantly improves rigour of other reviews with only minimal resource requirements. In all cases, the inclusion of an informatician with experience in searching for grey literature will increase review reliability (Doerr et al., 2015; Koffel, 2015). 5. Conclusions The two distinct forms of grey literature described herein require different techniques to obtain, and yet to date they have been included in reviews using a generalist approach, which threatens efficacy and rigour of grey literature searches. Poorly considered attempts to include grey literature may increase the resources needed to complete a review without providing additional value. Furthermore, if the reasons for including grey literature are not adequately considered and addressed accordingly, non-expert readers could place unwarranted faith on a review that fails to appropriately mitigate publication bias or extend its inclusion of the available evidence. It may not be necessary for all reviews to address both publication bias and practitioner-generated research. However, whichever approach is chosen, comprehensive reporting and justification of the sources selected remain necessary to ensure that the review methods used are transparent and repeatable. Acknowledgements NRH was supported by employment at MISTRA EviEM. HRB was supported by employment at Bangor University. References Bayliss, H.R., Beyer, F.R., 2014. Information retrieval for ecological syntheses. Res. Synth. Methods. http://dx.doi.org/10.1002/jrsm.1120. Collaboration for Environmental Evidence, 2013. Guidelines for systematic review and evidence syntheses in environmental management. Version 4.2. Collaboration for

• When data collection by practitioners or policy-makers has low resource requirements • When access to sampling systems is not a challenge • When data is readily recorded for other purposes (e.g., monitoring requirements) or as routine measurements from permanent sampling regimes (e.g., meteorology) • When practitioners are unlikely to restrict access to research (i.e., no on-going commercial interests). • Government databases (e.g., www.nass.usda.gov/Publications) • NGO websites (e.g., wwf.org) • Policy repositories (e.g., uropa.eu/publications/index_en.htm, www.unrol.org/dr.aspx) • Web searching (e.g., Google Scholar) • Industry contacts (e.g., businesses, consultancies) • Contact with research producers via personal contacts, email lists or social networks (subject-dependent) • Specialist grey literature repositories (e.g., opengrey.eu).

Environmental Evidence (http://environmentalevidence.org/wp-content/uploads/ 2014/06/Review-guidelines-version-4.2-final.pdf). Corlett, R.T., 2011. Trouble with the gray literature. Biotropica 43, 3–5. Dicks, L.V., Walsh, J.C., Sutherland, W.J., 2014. Organising evidence for environmental management decisions: a ‘4S’ hierarchy. Trends Ecol. Evol. 29, 607–613. Doerr, E.D., Dorrough, J., Davies, M.J., Doerr, V.A., McIntyre, S., 2015. Maximizing the value of systematic reviews in ecology when data or resources are limited. Austral Ecol. 40, 1–11. Gates, S., 2002. Review of methodology of quantitative reviews using meta-analysis in ecology. J. Anim. Ecol. 71, 547–557. Glasziou, P.P., Sanders, S.L., 2002. Investigating causes of heterogeneity in systematic reviews. Stat. Med. 21, 1503–1511. Gurevitch, J., Hedges, L.V., 1999. Statistical issues in ecological meta-analyses. Ecology 80, 1142–1149. Jennions, M.D., Møller, A.P., 2002. Publication bias in ecology and evolution: an empirical assessment using the ‘trim and fill’ method. Biol. Rev. Camb. Philos. Soc. 77, 211–222. Koffel, J.B., 2015. Use of recommended search strategies in systematic reviews and the impact of librarian involvement: a cross-sectional survey of recent authors. PLoS ONE 10 (5), e0125931. Koricheva, J., 2003. Non-significant results in ecology: a burden or a blessing in disguise? Oikos 102, 397–401. Koricheva, J., Gurevitch, J., 2014. Uses and misuses of meta-analysis in plant ecology. J. Appl. Ecol. 102, 828–844. Lortie, C.J., 2014. Formalized synthesis opportunities for ecology: systematic reviews and meta-analyses. Oikos 123, 897–902. Lortie, C.J., Aarssen, L.W., Budden, A.E., Koricheva, J.K., Leimu, R., Tregenza, T., 2007. Publication bias and merit in ecology. Oikos 116, 1247–1253. Pullin, A.S., 2012. Realising the potential of environmental data: a call for systematic review and evidence synthesis in environmental management. Environ. Evid. 1, 2. Pullin, A.S., Salafsky, N., 2010. Save the whales? Save the rainforest? Save the data! Conserv. Biol. 24, 915–917. Pullin, A.S., Stewart, G.B., 2006. Guidelines for systematic review in conservation and environmental management. Conserv. Biol. 20, 1647–1656. Smith, M.L., 1980. Publication bias and meta-analysis. Eval. Educ. 4, 22–24. Sutherland, W.J., 2005. How can we make conservation more effective? Oryx 39, 1–2. Sutherland, W.J., Pullin, A.S., Dolman, P.M., Knight, T.M., 2004. The need for evidencebased conservation. Trends Ecol. Evol. 19, 305–308. Sutherland, W.J., Mitchell, R., Walsh, J., Amano, T., Ausden, M., Beebee, T.J.C., Bullock, D., Daniels, M., Deutsch, J., Griffiths, R.A., Prior, S.V., Whitten, T., Dicks, L.V., 2013. Conservation practice could benefit from routine testing and publication of management outcomes. Conserv. Evid. 10, 1–3.

Please cite this article as: Haddaway, N.R., Bayliss, H.R., Shades of grey: Two forms of grey literature important for reviews in conservation, Biological Conservation (2015), http://dx.doi.org/10.1016/j.biocon.2015.08.018