Developing systematic reviews
Karin Hannes Featuring Mieke Heyvaert Centre for Methodology of Educational Research
Program • • • • • • •
14.00: Introduction 14.30: Searching the literature 15.00: Critical appraisal 15.20: Break 15.30: Critical appraisal exercise 16.00: Data-analysis: a worked example (M. Heyvaert) 16.45: Anatomy of a systematic review
Introducing Cochrane, Campbell and systematic reviews
Outline • Systematic reviews – – – – –
What is it and why do we need them? How do they look like? Who is producing them and where can we locate them? What is their (potential) impact? How can we start producing one ourselves?
What is it? A systematic review is a „systematic identification, evaluation and synthesis of all relevant studies regarding a specific topic, based on an explicit and pre-defined methodology‟ If high quality research studies exist... Summarize them!
Why do we need them? Half of what you learn in school will be shown to be either dead wrong or out-ofdate within 5 years of your graduation; the trouble is that nobody can tell you which half! (Dr. Sydney Burnwell) • Evidence of „effectiveness‟: the extent to which an intervention, when used appropriately, achieves the intended effect. • Evidence of „feasibility‟: the extent to which an intervention is practical and practicable, whether or not an intervention is physically, culturally or financially practical or possible within a given context. • Evidence of „appropriateness‟ the extent to which an intervention fits with a situation, how an intervention relates to the context in which it is given. • Evidence of „meaningfulness‟: the extent to which an intervention is positively experienced by the population and relates to the personal experience, opinions, values, beliefs and interpretations of the population.
Why do we need them? “Over two million articles are published annually in the biomedical literature in over 20,000 journals – literally a small mountain of information … a stack … would rise 500 metres” Cynthia Mulrow, in Systematic Reviews (BMJ Publishing Group, 1995)
And within that stack, we would be looking for a needle…
What is it? Traditional review
Systematic Review
• Lack rigor • Methodology not transparent • Different reviewers reach different
conclusions • Become out of date
• Scientific rigor to minimise bias • Explicit and reproducible methodology • Regularly updated
(Cochrane/Campbell)
What is it?
Predefined format: To help review authors be systematic To help people reading the reviews to find information quickly Transparancy Reliability
Key characteristics of a SR 1. A clearly stated set of objectives with pre-defined eligibility criteria for studies 2. An explicit, reproducible methodology
3. A well defined, systematic search that attempts to identify all studies that would meet the eligibility criteria 4. An assessment of the methodological quality of the findings of the included studies (assessment of risk of bias ) 5. A systematic extraction, synthesis, and presentation of the characteristics and findings of the included studies
PART 2: SYSTEMATIC REVIEWS OF EFFECTIVENESS HOW DO THEY LOOK LIKE?
What is a meta-analysis? Optional part of a systematic review • To identify a common effect among a set of studies Systematic reviews • To improve precision of an estimate • To investigate whether Meta-analyses the effect is constant • To answer controversies arising from conflicting studies or to generate new hypotheses
Meta-analysis Skills training versus usual curricula in the prevention of drug use in school kids.
Gives a more precise estimate of effect than when derived from the individual studies included within a review 1. Summary statistic + CI for each study individually (RR, OR, RD…) 2. Pooled intervention effect + CI is calculated as a “weighted average” Intervention
no effect
Comparison
Meta-analysis Skills training versus usual curricula in the prevention of drug use in school kids. Facilitates investigations of the consistency of Heterogeneity! evidence across studies, and the exploration of differences across studies.
Intervention
no effect
Standard
CC & C2: meta-analyse
„Skill-based interventions‟ prove to be an effective strategy to prevent from marihuana use in schoolkids.
Narrative summary
PART 3: SYSTEMATIC REVIEWS WHO IS PRODUCING THEM?
Astronomy, 17th century. “Combinations of data might be better than attempts to choose amongst them”
Karl Pearson, 1904. “Many of the groups are far too small to allow of any definite opinion being formed at all, having regard to the probable error involved”
Glass, 1976.
Archie Cochrane‟s challenge
“It is surely a great criticism of our profession that we have not organised a critical summary, by specialty or subspecialty, adapted periodically, of all
relevant randomised controlled trials.” (1979)
Cochrane Collaboration: 1993
an international not-for-profit organisation which aims to help people make well-informed decisions about healthcare by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of health care interventions
Cochrane Collaboration
Steering Group: Policy making body of the Cochrane Collaboration. Elected representatives of each of the Collaboration entities
Review Groups: Focus on Treatment of disease or health problems To prepare and maintain systematic reviews To develop a specialised trials register
Collaborative Review Groups
Fields Fields: Represent a population, group, or type of care that overlaps multiple Review Group area. Examples: Primary Health Care, Health Care of Older Adults, Complementary Medicine
Centres: Help organise and register review groups Facilitate collaboration among reviewers Provide training and consultation Establish liaisons Promote the Cochrane Collaboration Provide unique contribution
Centres
Steering Group
The Consumer Network Consumer Network: Provides consumer input Helps set priorities Helps with dissemination
Methods Groups
Methods Groups: Develop methods and products integral to internal functioning of the Collaboration Develop state of the art methods for systematic review Examples: Statistical, Economics, Placebo Effects, Informatics, Qualitative
Campbell Collaboration (C2): 2000
an international organisation which aims to help people make wellinformed decisions about public policy (crime & justice, education, social welfare) by preparing, maintaining and promoting the accessibility of systematic reviews of the effects of social and behavioural interventions
Steering group: strategic and policy making body Collaboration Campbell Coordinating Groups: Provide editorial services and support to authors of Campbell reviews and build links with users of systematic reviews:
Coordinating Groups
Crime and Justice Social Welfare Education International Development
The Belgian Campbell Group http://www.campbellcollaboration.be/Belgian_Campb ell_collaboration/Welcome.html
Steering Group
Users Group
Partners
Partners: Institutes who have a formal agreement with Campbell and support the same goals.
Methods Groups
Methods Groups: Improving the methodology of research synthesis, and disseminating guidelines for stateof-the-art review methods
Users group: Knowledge translation to increase the impact of Campbell reviews in policy and practice arenas, and to make the information more accessible
Examples of C2 Review Protocols and Reviews are available on: http://www.campbellcollaboration.org/library.php
CC & C2 Collaboration goals • To ensure high quality, up-to-date systematic reviews are available across a broad range of topics • To promote access to systematic reviews • To develop an efficient, transparent organisational structure and management system for the Collaboration • To achieve sustainability of the Collaboration Principles Collaboration Building on the enthusiasm of individuals Avoiding duplication Minimising bias Keeping up to date Striving for relevance Promoting access Ensuring quality Continuity
PART 4: SYSTEMATIC REVIEWS WHAT IS THEIR (POTENTIAL) IMPACT?
Evaluate a solution •
Situation: 1 mj. people die each year due to traffic accidents
•
Solution: Educate the drivers
•
Systematic Review: 24 studies on education, no evidence that it reduces traffic accidents
•
Recommendations: Stimulate teleconferences and the use of safe, public transport “Driver education is big business – our results show that it is also a big con” (Prof. Roberts)
Identify gaps in „evidence‟ •
Situation: anti-psychotics are subscribed for pregnant women with a psychotic disease
•
Systematic Review: No literature on adverse effects of anti-psychotics on the mother or the development of the (unborn) baby – –
•
RCT‟s with pregnant women are considered unethical Medicals‟ referals are based on habits and opinions
Recommendation: Research on the effects of anti-psychotics in pregnant women is necessary
“The continued use of antipsychotic drugs in women during pregnancy and lactation without sound evidence raises serious clinical and ethical concerns.” (Webb)
Identify solutions that cause „harm‟ •
Situation: Illinois law: Chicago Public Schools mandated to identify children at-risk for future criminal behaviour.
•
Solution: Scaired straigth programs: take them on tours of adult prison facilities
•
Systematic Review: Not only does a scared straight program fail to deter crime, it actually leads to more offending behaviour. “Governments need to adopt rigorous evaluation to ensure that they are not causing more harm to the citizens they intend to protect” (Dr. Petrosino).
1.
Select a topic
2.
Make a review team
3.
Develop/Register your title
4.
Write a protocol
PART 5: SYSTEMATIC REVIEWS HOW CAN WE CONTRIBUTE TO THEM?
1. Select a topic Systematic reviews: • Commissioned
• Invited • Unsolicited Motivation to undertake a review: • Resolve conflicting evidence • Address questions when practice is uncertain • Explore variations in practice • Confirm appropriateness of current practice
• Highlight need for future research
1. Select a topic • Learn to ask questions (Treatment or Intervention): • • • •
P I C O
population intervention comparison outcome
Does providing information (I) at school have a preventive effect on the use of drugs (O) in high school students (P) compared to sanctions when someone is caught using drugs (C) ?
1. Select a topic “Information campagnes in the prevention of drug use”
“Does information provision at school decreases the risk of drug use in high school students (compared to a standard approach)?”
Scope of the question? BROAD (LUMP) Advan -tages
NARROW (SPLIT)
Comprehensive
Manageable for review team
Generalizibility
Easy to read
Effect modifiers (why different effect between apples and oranges?)
Dis
More resources
Evidence sparse?
advan -tages
“mixing apples and oranges”
Lack of generalizibility
Choice of scope “biased” by authors
Excercise 1: developing a question
2. The review team Why more than 1 person? • Detection of errors (selection of eligible studies, data extraction) • Reduces risk of bias • When more than one person / team is interested in topic
Who should be in the team? • Expertise in topic area • Expertise in systematic review methodology • Incorporate view of “users” (teachers, consumers, therapists) • Address questions that are important to people • Take account of outcomes that are important to those affected • Make it accessible to people making decisions • Reflects variability in populations, settings…
3. Registering a title Contact a Cochrane Review Group (CRG) or Campbell Coordinating Group (CCG) and fill in a title form Cochrane Review Group:
Campbell Coordinating Group:
• > 52
• 4
• Specific condition (eg
• Crime & Justice CG
incontinence, learning & developmental disorder group)
• Group of diseases (eg ear nose and throat disorders group)
• Education CG • Social welfare CG • International Development
• Organ (eg renal group, heart group) • Patient (eg neonatal group, pregnancy and childbirth group, pain and palliative care)
There is a potential to co-register reviews!
4. Write a protocol • A priori statement of aims and methods of the review • Research question(s), aims, methods are considered in advance of identifying the relevant literature – Conduct review with minimal bias – Access to peer review – Greater efficiency in review process (Torgerson, 2003)
ADVANTAGES: • Clear research question before the review Avoid retrieving irrelevant papers • A priori inclusion and exclusion criteria avoid changing criteria as review progresses or studies may be included on basis of their results • If decisions are explicit it enables them to be justified • Develop protocol as independently as possible from the literature avoid influence by one or two major studies, less bias (Torgerson, 2003)
Review Protocol Anatomy • • • •
Title and authors Background Objectives Methods – – – –
Searching for studies Selecting studies Data extraction (including assessment study quality) Data analysis
• Acknowledgements / conflicts of interests • References • Tables / Figures
Review Protocol Contents BACKGROUND AND OBJECTIVES • Describes the context of the review: why is it important? – – – –
Description of the problem Description of intervention How the intervention might work Description potential subgroups
• Clarify conceptual issues central to the review • Precise statement of objective and research question, using PICO(S) – – – – –
Which population? Which intervention(s)? Which comparison(s)? Which outcome(s)? (Which study design?)
To assess the effects of home-based programmes aimed at improving developmental outcomes for pre-school children from disadvantaged families
A well formulated objective focuses the review
Review Protocol Contents METHODS • How to select studies? • How to search for studies? • Assessment quality of included studies • How to extract data? • How to analyse results?
Review Protocol Contents SELECTING STUDIES • •
•
Define inclusion and exclusion criteria Key features: – Established a priori – Explicit (explain all terms) – Based on PICO – Include study design, if applicable All retrieved studies are listed in the review either under included or excluded together with a justification
Review Protocol Contents Types of studies Randomised controlled trials. The control group will either receive no intervention or standard care. Studies comparing two different types of home based programme without a control group will be excluded. Types of participants: Home based child development interventions for pre-school children from socially disadvantaged families & Parents with children up to the age of school entry and who are socially disadvantaged in respect of poverty, lone parenthood or ethnic minority status. Age of school entry can vary between countries (4 to 7 years) and so the upper age range for this review will be the school entry age for the country in which the trial took place. Studies will be excluded if they aim to recruit particular clinical subgroups of parents. Types of outcome measures Primary outcomes: Cognitive development (including language development and attention) & Socio-emotional development (including self regulation and behavioural development) + Adverse outcomes: Parents feel disempowered. Types of interventions Home based interventions designed to improve child intellectual and socio-emotional development through the provision of relevant knowledge and skills to the parent. The intervention is delivered by trained lay or professional family visitors.
Review Protocol Contents SEARCH FOR STUDIES • Process needs to be reproducible! • Describe search terms + search strings (BOOLEAN LOGIC) • Describe sources (electronic databases, other sources, etc)
Review Protocol Contents ASSESSMENT OF STUDY QUALITY • Study quality may affect conclusions • A systematic review always includes an assessment of individual study quality • Different tools available • Describe how to use this information – Description methodological quality – Exclude studies of poor quality? – To include in conclusions (strong versus weak evidence)
Review Protocol Contents DATA EXTRACTION • Process of reading through a study and extracting the relevant information from each study, preferably by two independent reviewers • The reviewer fills out an data extraction sheet with the appropriate information taken from that study • Data extraction sheet may be on paper or electronically Extraction form consists of two different parts 1. characteristics that apply to the entire study 2. results: studies typically have multiple outcomes and therefore different effect sizes
Descriptive
Statistical part
Summary • Review protocol important first step when undertaking systematic review • Helps to focus and structure the review • Limits the scope for bias • In addition to how you plan to deal with the different steps in a review, include information on: – How many reviewers? – How are reviewers trained? – How to resolve disagreements?
Check the resource centre at the Campbell Website for specific guidance from methods groups and coordinating groups http://www.campbellcollaboration.or g/resources/research.php
References / more reading •
• • •
• • • •
Egger M & Davey-Smith G. (2001). Principles of and procedures for systematic reviews. In M Egger G Smith & D Altman (Eds.), Systematic Reviews in Health Care: Meta-analysis in Context (2nd ed., pp. 23-42 ). London, UK: BMJ Books. Lipsey MW & Wilson DB. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE. Miller S & Eakin A. Home based child development interventions for preschool children from socially disadvantaged families (Protocol). The Campbell Library, 2011. Petticrew M & Roberts H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA: Blackwell Publishing. Torgerson C. (2003). Systematic Reviews. London, UK: Continuum. Resources Campbell Collaboration: http://www.campbellcollaboration.org/ education_articles/ECG_Resources_for_Reviewers.php Resources Cochrane Collaboration: http://www.cochrane.org/training/cochrane-handbook
Searching the literature
Outline • The Information Retrieval Process – Search strategies: Decisions and challenges
• Searching the Main Databases – – – –
Selection and Types of databases Preparing a search strategy Implementing a search Saving & Managing the results
• Additional Databases & Retrieval Methods • Wrap Up • Resources & Additional Readings
Information Retrieval: A Continuous Process – Preliminary (scoping) Searches • Supports beginning steps: Definition of key concepts & research question • Use of standard reference tools and broad searches for review articles and key primary studies
– Main Searches • Identification of primary studies through searches of databases, Web, branching, manual searches • Most difficult given a number of challenges
Main Searches: Decisions Exclusion criteria:
• Selection of Information Retrieval Tools – Scope of search: Which disciplines or subject fields should be searched (including all related fields)? – Availability of indexing tools & expertise: Which tools do we have access to at our institution? Are there others who can perform searches for us? – Format of indexing tools: What format are they in (e.g. online, print, web-based)?
– Dates: How far back does the indexing go for each tool? – Language: What is the language of the material that is indexed? How can we locate non-English material? – Unpublished work: How can we access dissertations, reports, and other grey literature?
Selection of Databases • Consult your academic library‟s website to learn what databases are accessible from your institution! – Education: ERIC, British Education Index, Australian Education Index, CBCA Education, Education: A SAGE Fulltext Collection; Education Full text, Linguistics and Language Behavior Abstracts, … – Psychology: PsycINFO, PubMed (Medline), Ageline,
Psychology: A SAGE Full-Text Collection,Criminology: A SAGE Full-Text Collection, …
– Sociology: Sociological Abstracts, Contemporary Women‟s Issues. Sociology: A SAGE Full-text Collection, … – Health Care: Medline, CINAHL, EMBASE, AMED, Cochrane Trial Register,… – Multidisciplinary: Academic Search Premier, ProQuest Dissertations and Theses, FRANCIS, Social Sciences Index, SCOPUS, Web of Science
Main Searches: Decisions – Preparation of Search Strategies • What are the key concepts to be searched? • How are these represented in each discipline? • What are their related terms? • How are these key concepts represented in the controlled vocabulary within each database to be searched?
Approaches to Parent Involvement for Improving the Academic Performance of Elementary School Age Children Chad Nye, Herb Turner, Jamie Schwartz The purpose of this review is to determine the effectiveness of parental involvement in improving the academic performance of school age children in grades K-6.
Using a Thesaurus 1. From the research question, determine the main concepts to be searched (usually there are three): – – –
Intervention: Parental involvement Outcome: Academic performance Population: Kindergarten or Elementary students
2. Consult the main database to be searched. 3. Look up each concept in the thesaurus for this database. – –
A thesaurus is an alphabetical listing of the controlled vocabulary (or descriptors) used within a subject database A hierarchical arrangement is used so that Broader, Narrower and Related headings may be discovered
Check definition of terms. Keep those from which the definition matches your inclusion criteria.
Example: ERIC • Selecting the ERIC Descriptors Intervention
– Descriptors: Parental Involvement See: Parent participation – Related descriptors: Family involvement, Parent-school relationship, Parent role, Parents as teachers – Related keywords: parent* involvement, parent* effectiveness, parent* support, family support
Outcome
– Descriptors: Academic Performance See: Academic achievement – Related descriptors: Science achievement, Reading achievement, Writing achievement, Achievement gains
Population
– Descriptors: Elementary School Children See: Elementary school students – Related descriptors: Elementary education, Primary education, Kindergarten
Main Searches: Decisions • Construction of the Search Statements – What terms should be searched as descriptors or as “keywords”? – What Boolean operators should be used? AND / OR / NOT / NEXT / NEAR? – Where should truncation characters be used? (e.g. parent* will retrieve parent, parents, parental) – What limiting features are available to narrow results? (e.g. use of Publication Type codes, time period, language)? ERIC Example
Boolean Operators AND: Both terms must be present in order for a record to be retrieved. Used to combine different concepts. e.g parent participation AND achievement OR: Either term may be present in order for a record to be retrieved. Used to search for related terms or synonyms. e.g. parent OR family
NOT: Used between two terms to ensure that the second term will not appear in any of the results. e.g. literacy NOT adult (Parental involvement OR parent participation) AND academic achievement AND (elementary OR primary education)
Example: ERIC, cont‟d •
Combining Keywords/Descriptors using Boolean operators:
1. DE=(Parent participation OR Family involvement OR Intervention Parent role OR Parent-school relationship OR Parents as teachers) 2. “Parent* involvement” OR “Parent* effectiveness” OR “Parent* support” OR “family support” 3. #1 OR #2 = 28,958 records 4. DE=(Academic achievement OR Science achievement Outcome OR Reading achievement OR Achievement gains) – 46,574 records 5. DE=(Elementary school students OR elementary OR elementary schools OR primary education Population education OR kindergarten) - 291997 recs 6. #3 AND #4 AND #5 = 1,669 records
Limiting Your Results Using the Limiting Commands: •
Limiting fields contain information that is common to a large number of records within a database. These include language, document type, publication year and so.
•
Some limiting fields will vary across databases (e.g. Classification Code, Age Group)
•
Decisions about whether you are going to restrict: •
The language of the document
•
The search to a certain time period
•
The results to empirical studies only
ERIC record High quality search strategies deliver relevant results! AUTHOR : Polovina,-Nada; Stanisic,-Jelena TITLE: A Study on Family-School Cooperation Based on an Analysis of School Documentation PUBLICATION YEAR: 2007 SOURCE: Online Submission. Journal of Educational Research (Belgrade), v39 n1 p115-133 ERIC DOCUMENT LINK: http://www.eric.ed.gov/contentdelivery/servlet/ERICServlet?accno=ED499159 DOCUMENT TYPE: Journal-Articles; Reports-Research LANGUAGE: English DESCRIPTORS: Student-Behavior; Attendance-Patterns; Child-Development; Parents-; Family-School-Relationship; Parent-TeacherCooperation; Foreign-Countries; Elementary-Schools; Parent-Participation; Grades-Scholastic; Academic-Achievement; ParentInfluence; Parent-Child-Relationship IDENTIFIERS: SerbiaABSTRACT: Family-school cooperation is a very complex process that can be studied at different levels in a number of different ways. This study has covered only some aspects of cooperation between parents and teachers, based on school documentation of a Belgrade elementary school. The study covered analyses of 60 Attendance Registers pertaining to 60 classes with 1289 students from Grade 1 through Grade 8 during an academic year. The unit of analysis included: parents attendance at PTA meetings and individual meetings between parents and teachers. In addition to the frequency of parents' visits to school, the relationship between such registered parents' visits and overall academic performance, grades in conduct, excused and unexcused absence from classes were also considered. The research findings indicated interference between development factors (attitude change in parent-child relationship and growing-up) and parents' informal "theory of critical grades" i.e. transitional processes in schooling. The findings confirmed that parents' individual visits to school were mainly meant to offer an excuse for the student's absence from school, while attendance at PTA meetings was linked to poor grades in conduct and missed classes (both excused and unexcused). The findings also showed that parents pursued visiting strategies which were pragmatic, less time-consuming and less emotionally draining ones. The closing part refers to discussions on practical use of the study and possible further research. (Contains 4 graphs.) [This article is the result of the project "Education for Knowledge-Based Society" No. 149001 (2006-2010), financially supported by the Ministry for Science and Environmental Protection of the Republic of Serbia.] (Author)
Next Steps
Repeat these steps for each database to be searched.
Additional Retrieval Methods • We haven‟t talked about…. – The web and other resources…to locate grey literature • Use Advanced Search screens on large engines (eg. Google, Altavista, AlltheWeb, MSN Live) • Consult specific sites • Screen conference abstracts and databasis storing dissertations
– Manual searches • Browse the Table of Contents of key journals for current years
– Reference Searching • Look for relevant references in key-articles retrieved and conduct a forward reference search in Web of Science.
Managing Your Results •
Export the results –
•
Import into a bibliographic management software: – – –
•
•
Save as a Text file RefWorks, Reference Manager, EndNote
Edit your inhouse database –
Add Source code for each database searched (e.g ERIC1,
–
Add notes to the records (e.g.includes vs excludes)
PsycINFO1…)
Compile a Search History document listing the original search strategies
Information Retrieval: Wrap Up • Importance of information retrieval process – Not a “one-shot”deal – Requires expertise in the planning and implementation of searches – Consulting with the Trials Search Advisor or an Information Specialist is highly recommended
• Use a bibliographic management software – Store, manage and organize results
• Must have ability to replicate review – Documentation of entire process, including search strategies used for each database, decisions taken, etc.
Resources C2. Education Coordinating Group. (2009). Information Retrieval Methods Group Systematic Review Checklist and Database Worksheet. Available: http://www.campbellcollaboration.org/education_articles/ECG_Resources_for_Reviewers.php Designed to help both reviewers and those reviewing C2 protocols. – –
The Checklist itemizes the recommended steps for information retrieval during the protocol and review stages. A good resource for new C2 reviewers as it provides an indication of the criteria that will be used to evaluate the information retrieval component of a protocol and a review; The Database worksheet provides a useable, expandable template for the documentation of database searches.
Lefebvre, C., Manheimer, E., & Glanville, J. (2008, Feb.). Chap. 6: Searching for studies. In J.P.T. Higgins & S. Green (Eds.). Cochrane handbook for systematic reviews of interventions. London: Wiley. Provides a “how to” guide for Cochrane Trial Search Coordinators. Clearly written and easy to understand for those who have no experience with searching. Focus is on the retrieval of information in the health sciences. Information Retrieval Methods Group (2009). Searching for studies. Campbell Collaboration. This comprehensive document provides the background on the C2 policy related to information retrieval and will be useful to Trials Search Co-ordinators (or Advisors) who are new to their post, as well those who are conducting reviews. This document outlines some general issues in searching for studies; describes the main sources of potential studies; and discusses how to plan the search process, design and carry out search strategies, manage references found during the search process and correctly document and report the search process. It is currently being revised and updated.
Readings • •
Gomersall, A. (2007). Literature searching: Waste of time of essential skill? Evidence & Policy: A Journal of Research, Debate and Practice, 3(2), 8301-308. Available: Hopewell, S., Clarke, M., & Mallett, S. (2006). Grey literature and systematic reviews. In Dr. H.R. Rothstein, A.J. Sutton, & M. Borenstein (Eds.). Publication bias in meta-analysis (pp.49-72). New York: John Wiley & Sons.
•
McGowan, J. & Sampson, M. (2005). Systematic reviews need systematic searchers. Journal of the Medical Library Association, 93(1), 74-80.
•
Wade, A., Turner, H. M., Rothstein, H. R., & Lavenberg, J. (2006). Information retrieval and the role of the information specialist in producing high-quality systematic reviews in the social, behavioral, and education sciences. Evidence & Policy: A Journal of Research, Debate and Practice, 2(1), 89-108
Critical appraisal
Why is it so important? • Meta-analysis aims to increase precision • Meta-analysis of studies with bias in results gives very precise but wrong results • Garbage in, garbage out
Methodological rigor relates directly to the interpretation and generalization of findings (Troia, 1999).
Bias versus imprecision Bias: A systematic error in the results or the inferences Methodological flaw Overestimation or underestimation
Ideal study
BIAS
Bias versus imprecision Imprecision: • A random error in the results • Sample variation • Direction of error is random
Ideal study
IMPRECISION
Risk of Bias versus Bias •Clear empirical evidence that particular flaws in study design can lead to bias. •Usually impossible to know to what extent biases have affected the results. •Key consideration = should the results be believed
BIAS + IMPRECISION
http://www.sign.ac. uk/methodology/ checklists.html
Online guideance: Chapter Critical Appraisal http://www.cochrane .org/contact/method s-groups
Domaine
Cochrane Tool
Description
Judgement
Sequence generation: Did the study use a randomized sequence of assignments (centralized)?
QUOTE: “patients were randomly allocated” COMMENT: probably done, since earlier reports of this study describe use of random sequences.
YES (low risk of bias)
Allocation concealment:Did the study use any mechanism that shields those who enroll patients in a study from knowing the next assignment?
QUOTE: “We used sealed enveloppes to allocate the students to either the intervention or the control group”. COMMENT: Risk unclear since there is no mention of any safeguards used, such as opaque or sequentially numbered enveloppes.
YES (low risk) NO (high risk) UNCLEAR (uncertain)
Lack of blinding could affect actual outcomes •Differential drop-out •Differential cross-over to alternative intervention •Differential administration of co-interventions Lack of blinding could affect the outcome asssessment
YES NO UNCLEAR
Blinding: Participants, Providers, Outcome assessors
Incomplete outcome data: Data available, but excluded from analysis: Withdrawal, Do not attend follow-up appointment, Failure to complete questionnaire / diaries; Cannot be located (lost to follow-up), Decision by investigator to cease followup, Data or records are lost
QUOTE: “Participants found to be ineligible after enrolment”. COMMENT: Justifiable, so not considered as leading to missing outcome data. VERSUS: Difference in proportion of incomplete data across groups and related to outcomes (e.g. adverse effects in experimental group)
Selective outcome reporting: Selection of a subset of the variables recorded for inclusion in publication, on the basis of the results
Omission of non-significant outcomes Choice of data for an outcome (e.g. competences) Choice of analysis (e.g. scores on a test for skills) Reporting of subsets of data (e.g. no attitudes, little correlation) Under-reporting of data (e.g. only “not significant”)
YES NO UNCLEAR
YES NO UNCLEAR
“Risk of bias summary”
Tool for Nonrandomized studies Downs and Black instrument (J Epidemiol Community Health 1998;52:377-84) http://www.nccmt.ca/ registry/view/eng/9.html
Focus on confounding issues in studies!
In summary • Risk of bias assessment is a very important part of your review
• Think about it when you write your protocol, describe the methodology • For RCT: use the Cochrane tool
• For NRS: • Potential biases are likely to be greater (selection bias & reporting bias) • Use the appropriate tool to assess risk of bias • Consider how potential confounders are addressed
Excercise: critical appraisal
“Schrijven van reviews”: Een praktijkvoorbeeld Mieke Heyvaert Ph. D. Fellow of the Research Foundation Flanders (FWO) Methodology of Educational Sciences Research Group, Faculty of Psychology and Educational Sciences, Katholieke Universiteit Leuven, Belgium
“Schrijven van reviews” Een praktijkvoorbeeld: A meta-analysis of intervention effects on challenging behaviour (CB) among persons with intellectual disabilities (ID)
Background Interventions for CB among persons with ID Examples of CB: Verbal and physical aggression, Property damage and destructiveness, Disruptive and antisocial behaviour, Overactivity, Temper tantrums, Screaming, Stereotyped and repetitive behaviour, General delinquency, and Self-injurious behaviour (e.g. head punching, self-biting, skin picking, and hitting against hard objects or other body-parts)
Background • High prevalence of CB among persons with ID • Negative impact of CB for persons with ID and their family/context
Intervention research Bio–Psycho–Social intervention components
Study aim • Systematically review biological, psychotherapeutic, and contextual intervention components applied to treat CB among persons with ID, & analyse intervention effects and moderating variables • Method: Meta-analysis Boost the total sample size and effect precision by combining „sufficiently‟ homogeneous results across studies
Method – Literature search • Idealiter: – Systematic search of Electronic databases; Grey literature; Journals; Reference lists; Citation indexes; Contact authors & experts • For this study: – Systematic search of 4 electronic databases: ERIC, PsycINFO, Web of Science, Medline
Method – Literature search Systematic search of 4 electronic databases: ERIC, PsycINFO, Web of Science, Medline
How would you conduct this systematic search?
Method – Literature search • Articles reporting on intervention effects for CB among persons with ID by combining key terms describing the target group and outcome variables • Key terms Target group: intellectual disability, mental retardation, learning difficulty, complex/high support needs • Key terms Outcome variables: challenging behaviour, problem behaviour, behavioural problems, aggression, selfinjury, self-injurious behaviour
• Terms in singular and plural, in US- and UK-terminology
Method – Literature search • • • •
Included: English-language articles Included: Empirical studies Included: Published in peer-reviewed journals Included: Published between January 2000 - November 2008
• Excluded: Articles that only reported on diagnostic instruments / only presented review material • Excluded: Articles had to contain data or sufficient statistical information to make quantitative analysis feasible
Method – Data-analysis • • • • • •
Coding Meta-analysis Sensitivity analysis Subgroup analysis Meta-regression analysis Publication bias analysis
Coding Intervention effects Intervention characteristics • Intervention type • Intervention combination Methodological features • Study quality • Study design • Data collection • Reliability
Study characteristics • Research period • Continent Participant features • Gender • Age • Degree of ID
Method – Meta-analysis Statistical analyses: Comprehensive Meta-Analysis Version 2.0 (Borenstein et al. 2009, http://www.metaanalysis.com)
Method – Meta-analysis Fixed-effect vs. Random-effects meta-analysis: • Fixed-effect meta-analysis: • We assume that there is one true effect size („fixed effect‟) which underlies all the studies in the analysis • The goal is to compute the common effect size for the identified population, and NOT to generalize to other populations Fixed-effect meta-analysis: Rather rare !
Method – Meta-analysis Fixed-effect vs. Random-effects meta-analysis: • Random-effects meta-analysis: • We allow that the true effect could vary from study to study. There may be different effect sizes underlying different studies – E.g. the effect size might be higher in studies where a more intensive variant of an intervention is used, with older participants, with higher educated participants,… • The goal is to generalize to a range of scenarios Random-effects meta-analysis: Often done ! When accumulating data from studies written by different independent researchers, it would be unlikely that all the studies are functionally equivalent
Method – Meta-analysis Random-effects meta-analysis: + Between-study variance into account + Study weights are more balanced: Less relative weight to large studies + Inferential results referring to a universe of more diverse studies than fixed-effect analyses + Width of the confidence intervals: Generally more „conservative‟ and „realistic‟
Method – Meta-analysis Random-effects meta-analysis: • Assuming that the studies in the analysis only represent a random sample of effect sizes that could have been observed • The summary effect is the estimate of the mean of these effects • Studies weighted by study precision: Studies that yield more precise estimates of the effect size are assigned more weight
Method – Meta-analysis • (1) Calculate effect sizes (standardised mean difference; SMD) and variances for all included studies‟ intervention effects • (2) Calculate the summary effect (effect size and variance) with a 95% confidence interval (CI)
Results – Meta-analysis (1) Calculate effect sizes (SMD) and variances for all included studies‟ intervention effects: • Combined effect sizes and their standard errors computed for all 30 articles are presented in Table 1 • The applied interventions made an improvement in each study: All calculated effect sizes are positive
Method – Meta-analysis (2) Calculate the summary effect (effect size and variance) with a 95% confidence interval (CI) – Random effects model: • SMD is 0.671 with a 95% CI of 0.570 to 0.771 • Medium (effect size around 0.5) to large (effect size around 0.8) effect • The null hypothesis that the mean of these effects is zero, can be rejected, Z(N = 30) = 13.070, p < 0.001
Results – Meta-analysis Comparison: For the Fixed-effect model: • Standardized mean difference is 0.670 • 95% CI of 0.572 to 0.768 • The null hypothesis that the mean of these effects is zero, can be rejected, Z(N = 30) = 13.452, p < 0.001
The results for the fixed- and random-effects analysis are analogous
Method – Sensitivity analysis = Determine the robustness of our results by examining whether our conclusions might differ substantially if a study was dropped • Running the analysis 30 times, each time removing one study, in order to show each study‟s impact on the combined effect
Results – Sensitivity analysis • Calculate for each study the overall standardized mean difference when that study was hypothetically removed from the metaanalysis • Since the 30 overall effect sizes formed by omitting each study separately varied between 0.649 and 0.689 (see next Table), our results look very robust
Method – Subgroup & meta-regression analysis To assess the impact of moderating variables: • Subgroup analysis: Compare treatment effects across groups concerning categorical variables • Meta-regression analysis: Explores the impact of continuous moderators
Results – Subgroup analysis • Analyses of variance to compare treatment effects across groups for the categorical variables: intervention type, intervention combination, quality assessment, study design, data collection, reliability, length of the research period, continent, gender, age, degree of ID • For all these variables the differences between the groups are not significantly related to the effect size
Results – Meta-regression analysis • One continuous moderator: Publication year
• This variable is not significantly related to the effect size • Possible explanation of „not significantly related to the effect size‟: Relatively small number of studies included in the metaanalysis
Method – Publication bias analysis We refer to publication bias when studies included in an analysis differ systematically from all studies that could be included: Particularly, studies with larger effects are more likely to be published, and this can lead to an upward bias in the summary effect
Method – Publication bias analysis • (A) Funnel plot analysis: Plot the included studies • (B) Classic fail-safe N analysis: Do we need to be concerned that the entire observed effect is an artifact of bias?
• (C) Duval‟s and Tweedie‟s trim and fill-analysis: How would the effect size shift if the apparent bias were to be removed?
Results – (A) Funnel plot analysis
Results – (A) Funnel plot analysis • In the Figure a measure of study size (precision, the inverse of standard error) is plotted on the vertical axis as a function of effect size on the horizontal axis • In the absence of publication bias, we would expect the included studies (white circles) to be distributed symmetrically around the combined effect size - OK!
Results – (B) Classic fail-safe N analysis
Results – (B) Classic fail-safe N analysis • Quantify the possible publication bias effect • The fail-safe N is 1336 We would need to locate & include 1336 'null' studies in order for the combined two-tailed p-value to exceed 0.05 • Because this number is large, we can be relatively confident that the treatment effect, while possibly inflated by the exclusion of some studies, is real
Results – (C) Trim and fill-analysis
Trim and fill plot of precision by standardised difference in means: Plot with observed (white) and imputed (black circles) studies
Results – (C) Trim and fill-analysis Left side of the mean effect: • Duval and Tweedie's trim and fill analysis suggests that four studies are missing • Random-effects model: SMD & 95% CI for the combined studies is 0.671 (0.570, 0.771) • Using trim and fill: Imputed SMD & 95% CI is 0.634 (0.525, 0.744) - Little bit lower than the original SMD
Results – (C) Trim and fill-analysis Right side of the mean effect: • Duval and Tweedie's trim and fill analysis suggests that no studies are missing
Conclusions • We found effect sizes ranging from 0.223 to 1.411 The effect sizes vary between a small and a very large effect, all indicating positive treatment effects • The combined effect size over all studies is 0.671, with a 95% CI of 0.570 to 0.771, which is a medium to large effect • The implemented sensitivity analysis revealed that this effect is robust
Conclusions • Analyses of variance showed no significant different treatment effects for biological, psychotherapeutic, and contextual interventions • Differences between unimodal and multimodal treatments turned out to be not significant as well • The calculated means and standard errors for the moderators intervention type and intervention combination show that there are only small differences between the mean effects for biological, psychotherapeutic, and contextual interventions, and for unimodal and multimodal treatments
Conclusions • Some authors claim that there is evidence for the effectiveness of pharmacological treatments, although many authors assert that they lack empirical effectiveness. Additionally, there are many concerns regarding their adverse effects • Concerning psychotherapeutic and contextual interventions too, there are authors advocating their effectiveness, while some say that the evidence is rather limited • Our meta-analysis shows that there is evidence for the effectiveness of pharmacological, psychotherapeutic and contextual interventions, used alone or in combination • Furthermore, we found no indications for the superiority of one of the treatment approaches or combination types
Conclusions • Treatment effects also did not vary much across groups for the categorical moderators included in our meta-analysis: quality assessment, study design, data collection, reliability, length of the research period, continent, and gender, age, & level of ID of participants • We also did not find any effect of the continuous moderator publication year on the treatment effect • The nonsignificance of moderator effects is not uncommon, due to low statistical power for detecting interaction between a moderator variable and the independent variable • There exist only small differences between the mean effects for the tested categorical moderating variables, so even if we would increase the power, we would most likely not detect clinically important differences
Conclusions • A funnel plot-, a fail-safe N-, and Duval‟s and Tweedie‟s trim and fill-analysis demonstrate that our meta-analysis does not seem to suffer much from publication bias effects
Conclusions • In summary, interventions for challenging behaviour among persons with ID described in the 30 included articles were effective, with only small differences between the mean effects for biological, psychotherapeutic, and contextual treatments, and for unimodal and multimodal interventions • In contrast to claims in the literature that the evidence for one or another intervention is still rather limited, the effects in our meta-analysis were robust and convincing
Bibliography • Borenstein M., Hedges L. V., Higgins J., & Rothstein H. (2009). Introduction to meta-analysis. Wiley, Chichester. • Comprehensive Meta-Analysis http://www.meta-analysis.com
Version
2.0
(2009).
• Heyvaert, M., Maes, B., & Onghena, P. (2010). A metaanalysis of intervention effects on challenging behaviour among persons with intellectual disabilities. Journal of Intellectual Disability Research, 54, 634-649.
Data-analysis: narrative approaches
Synthesis examples Preliminary synthesis examples – Textual summaries – Tabulation – Grouping and clustering
• Exploring relationships examples – Idea webbing – Synthesis tabulation – Concept mapping
•
Preliminary synthesis Textual summary example: Young et al (1999) and Camit (2002) report on the effectiveness and implementation of a smoke alarm prevention campaign in NSW Australia for Asian foreigners, including info sessions by skilled workers. Qualitative data on barriers to purchase were also collected in focus groups. The main observation is that….
•
Tabulation example
Author /year
Location/ Setting
Target population Method
Main findings
Roberts London, et al UK, urban (2004)
58 adults and 41 children in qual study, 2145 households exposed to intervention
Focus groups and interviews Before and After study design
Problems with smoke alarms identified as major barrier
Camit (1998)
Chinese, vietnamese, arabic-speaking (numbers not given)
Focus groups
Implementa tion succesful using language appropriate approach.
NSW, Australia, mixed
Preliminary synthesis • Grouping example Grouping according to Location
Focus of report
Population
UK DiGiuseppe et al (1999), Roberts et al (2004)
Broad, general factors affecting program: DiGiuseppe (1999), Camit (2002)
Ethnically mixed: Camit (2002), Young et al (1999), Roberts et al (2004)
Australia Camit et al (2002), Young et al (1999)
Individual factors affecting program: Roberts (2004), Young et al (1999)
Ethnically mixed & low income: DiGiuseppe (1999)
Preliminary synthesis: thematic • Within case Study
Themes
Study 1
Involvement of fire service False alarm issues Fear from installers Testing of knowledge
Study 2
Lack of awareness of danger Overestimation of cost Frequent moving Lack of knowledge on detectors
• Cross-case Barriers
Facilitators
Barriers and facilitators to installation of smoke alarms General
Anxiety of property damage
Landlord approval for installation
Specific to alarm campaign
Lack of knowledge on installation Unwillingness to install Anxiety to letting installer in the house
Installation of alarm by skilled worker from community
Barriers and facilitators to continued use of smoke alarms
Exploring relationships: Web of Ideas
Exploring relationships: Synthesis Tabulation
Exploring relationships: Concept mapping
http://www.lancs.ac.uk/shm/research/nssr/res earch/dissemination/publications.php
Systematic Review Summary • • • • •
Establish objectives, selection criteria Search for eligible studies Apply selection criteria Assess study quality, extract data Analyse results using meta-analysis, where appropriate and possible • Perform sensitivity analyses • Prepare report (keep it up-to-date!)
Acknowledgments: Staff members of CMPO: Wim Van den Noortgate Staff members of CEBAM: Bert Aertgeerts, Filip Cools, Trudy Bekkering Cochrane and Campbell Colleagues International
Contact:
[email protected] Tel: 016/326220
ANNEX: Anatomy of a systematic review
Anatomy of a Systematic Review •
Background/Introduction – – –
•
Establish need (Distinguish from previous review efforts) State objectives and review questions
Methods –
Criteria for inclusion and exclusion • • • •
Type of population Type of studies Type of intervention (+ comparison) Type of outcomes
Anatomy of a Systematic Review (cont.) •
Methods – Locating studies: Consulted data sources (databases, grey literature, reference searches, expert consulting etc., – Search strategy (final)
Anatomy of a Systematic Review (cont.) Inclusion criteria review on QES in the literature (update Dixon-Woods)
• Methods – Proces of selecting studies
1. Published between 2005 (jan) and 2008 (dec)
Already filtered out.
Conducted within health care or a health care context
Include: Syntheses of qualitative with quantitative research) by synthesis methods other than informal review.
2.
• Include screening instrument in annex
Possible screening criteria • Timespan • Language restrictions • Discipline / Scientific field • INCLUSION and EXCLUSION CRITERIA
Exclude: Papers commenting on methodological issues but without including details of the outcomes of the synthesis. Papers that do not explicitly describe or name a method for synthesis. Reviews on concepts/definitions used within health care or research issues 3.
Published in English language
4.
Published in a peer-reviewed journal
Consider a flowchart of results from screening
Anatomy of a Systematic Review (cont.) •
Methods: –
Data Extraction • Introduce coding form • Describe and define coding categories • Describe process of data extraction „at least two independent reviewers‟
Do not re-invent the fire!
„We used the EPOC guidance on dataextraction (reference)‟…
Anatomy of a Systematic Review (cont.) • • • • •
•
Results
– –
Descriptive results Inferential results (if applicable)
Discussion Conclusions
– –
Implications for practice Implications for research
References Appendix: search strings, critical appraisal checklist, list with excluded studies (usually a flow chart), coding/extraction sheets, outcomes of meta-synthesis exercise etc.
Consider developing a user sheet (short summary avoiding scientific jargon)
