Software and model selection challenges in meta

Meta-analysis overview Two-stage meta-analysis One-stage meta-analysis Summary

Software and model selection challenges in meta-analysis MetaEasy, model assumptions, homogeneity and IPD

Evan Kontopantelis

David Reeves

Centre for Primary Care Institute of Population Health University of Manchester

Health Methodology Research Seminar Manchester, 6 Nov 2012 [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]

Kontopantelis, Reeves

Software and model selection challenges in meta-analysis


Outline

1

Meta-analysis overview

2

Two-stage meta-analysis

3

One-stage meta-analysis

4

Summary

[Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]




The heterogeneity issue

Outline

1


2


3


4

Summary






Timeline ‘Meta’ is a Greek preposition meaning ‘after’, so meta-analysis =⇒ post-analysis Efforts to pool results from individual studies back as far as 1904 The first attempt that assessed a therapeutic intervention was published in 1955 In 1976 Glass first used the term to describe a "statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings" [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Meta-analysing reported study results A two-stage process the relevant summary effect statistics are extracted from published papers on the included studies these are then combined into an overall effect estimate using a suitable meta-analysis model

However, problems often arise papers do not report all the statistical information required as input papers report a statistic other than the effect size which needs to be transformed with a loss of precision a study might be too different to be included (population clinically heterogeneous) [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Individual Patient Data IPD

These problems can be avoided when IPD from each study are available outcomes can be easily standardised clinical heterogeneity can be addressed with subgroup analyses and patient-level covariate controlling

Can be analysed in a single- or two-stage process mixed-effects regression models can be used to combine information across studies in a single stage this is currently the best approach, with the two-stage method being at best equivalent in certain scenarios






Outline

1

Meta-analysis overview The heterogeneity issue

2


3


4

Summary






When Robin ruins the party...






Heterogeneity tread lightly!

When the effect of the intervention varies significantly from one study to another It can be attributed to clinical and/or methodological diversity Clinical: variability that arises from different populations, interventions, outcomes and follow-up times Methodological: relates to differences in trial design and quality

Detecting quantifying and dealing with heterogeneity can be very hard [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Absence of heterogeneity Assumes that the true effects of the studies are all equal and deviations occur because of imprecision of results Analysed with the fixed-effects method [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Presence of heterogeneity

Assumes that there is variation in the size of the true effect among studies (in addition to the imprecision of results) Analysed with random-effects methods [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]




the MetaEasy add-in metaeff & metaan Methods and performance τˆ 2 = 0

Outline

1


2


3


4

Summary






Challenges with two-stage meta-analysis Heterogeneity is common and the fixed-effect model is under fire Methods are asymptotic: accuracy improves as studies increase. But what if we only have a handful, as is usually the case? Almost all random-effects models (except Profile Likelihood) do not take into account the uncertainty in τˆ2 . Is this, practically, a problem? DerSimonian-Laird is the most common method of analysis, since it is easy to implement and widely available, but is it the best? [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Challenges with two-stage meta-analysis ...continued

Can be difficult to organise since... outcomes likely to have been disseminated using a variety of statistical parameters appropriate transformations to a common format required tedious task, requiring at least some statistical adeptness

Parametric random-effects models assume that both the effects and errors are normally distributed. Are methods robust? Sometimes heterogeneity is estimated to be zero, especially when the number of studies is small. Good news? [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Outline 1


2

Two-stage meta-analysis the MetaEasy add-in metaeff & metaan Methods and performance τˆ2 = 0

3


4

Summary [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Organising

Data initially collected using data extraction forms A spreadsheet is the next logical step to summarise the reported study outcomes and identify missing data Since in most cases MS Excel will be used we developed an add-in that can help with most processes involved in meta-analysis More useful when the need to combine differently reported outcomes arises






What it can do

Help with the data collection using pre-formatted worksheets. Its unique feature, which can be supplementary to other meta-analysis software, is implementation of methods for calculating effect sizes (& SEs) from different input types For each outcome of each study... it identifies which methods can be used calculates an effect size and its standard error selects the most precise method for each outcome






What it can do ...continued

Creates a forest plot that summarises all the outcomes, organised by study Uses a variety of standard and advanced meta-analysis methods to calculate an overall effect a variety of options is available for selecting which outcome(s) are to be meta-analysed from each study

Plots the results in a second forest plot Reports a variety of heterogeneity measures, including Cochran’s Q, I 2 , HM 2 and τˆ2 (and its estimated confidence interval under the Profile Likelihood method) [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Advantages Free (provided Microsoft Excel is available) Easy to use and time saving Extracted data from each study are easily accessible, can be quickly edited or corrected and analysis repeated Choice of many meta-analysis models, including some advanced methods not currently available in other software packages (e.g. Permutations, Profile Likelihood, REML) Unique forest plot that allows multiple outcomes per study Effect sizes and standard errors can be exported for use in other meta-analysis software packages [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Installing

Latest version available from www.statanalysis.co.uk Compatible with Excel 2003, 2007 and 2010 Manual provided but also described in: Kontopantelis E and Reeves D MetaEasy: A Meta-Analysis Add-In for Microsoft Excel Journal of Statistical Software, 30(7):1-25, 2009 Play video clip






Outline 1


2


3


4






Stata implementation MetaEasy methods implemented in Stata under: metaeff, which uses the different study input to provide effect sizes and SEs metaan, which meta-analyses the study effects with a fixed-effect or one of five available random-effects models

To install, type in Stata: ssc install help

Described in: Kontopantelis E and Reeves D metaan: Random-effects meta-analysis The Stata Journal, 10(3):395-407, 2010 [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Outline 1


2


3


4






Many random-effects methods which to use?

DerSimonian-Laird (DL): Moment-based estimator of both within and between-study variance Maximum Likelihood (ML): Improves the variance estimate using iteration Restricted Maximum Likelihood (REML): an ML variation that uses a likelihood function calculated from a transformed set of data Profile Likelihood (PL): A more advanced version of ML that uses nested iterations for converging Permutations method (PE): Simulates the distribution of the overall effect using the observed data [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Performance evaluation our approach

Simulated various distributions for the true effects: Normal Skew-Normal Uniform Bimodal

Created datasets of 10,000 meta-analyses for various numbers of studies and different degrees of heterogeneity, for each distribution [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Performance evaluation our approach

Compared all methods in terms of: Coverage, the rate of true negatives when the overall true effect is zero Power, the rate of true positives when the true overall effect is non-zero Confidence Interval performance, a measure of how wide the (estimated around the effect) CI is, compared to its true width.






Homogeneity Zero between study variance

Power (25th centile)

Coverage

zero between study variance 1.00

zero between study variance

0.90

1.00 %

0.80

0.95

0.70 0.60

0.90

%

0.50 2

5

8

11

14

17

20

23

26

29

32

35

number of studies

0.85

FE REML

DL PL

ML PE

0.80 Overall estimation performance zero between study variance

0.75

2.00

2

5

8

11

14

17

20

23

26

29

32

DL PL

1.80 1.60

number of studies FE REML

35

1.40

ML PE

1.20 1.00 0.80 0.60 0.40 2

5

8

11

14

17

20

23

26

29

32

35

number of studies

[Poster title]

FE REML

DL PL

ML PE

[Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Coverage performance Small and large heterogeneity under various distributional assumptions Coverage

Coverage

H²=1.18 − I²=15% (normal distribution)

Coverage

H²=1.18 − I²=15% (skew−normal distribution)

Coverage

H=1.18 − I=15% (bimodal distribution)

H=1.18 − I=15% (uniform distribution)

1.00

1.00

0.95

0.95

0.95

0.95

0.90

0.90

0.90

0.90

0.85

0.85

0.85

0.80

0.80

0.80

0.75

0.75

0.75

%

%

%

1.00

%

1.00

0.85 0.80

2

5

8

11

14

17

20

23

26

29

32

35

2

5

8

11


14

17

20

23

26

29

32

35

2

5

8

11

number of studies

DL PL

ML PE

FE REML

sk=0, ku=3

14

DL PL

ML PE

20

FE REML

sk=2, ku=9

23

26

29

32

0.75

35

2

DL PL

Coverage

8

11

14

Coverage


0.95

0.95

0.95

0.95

0.90

0.90

0.90

0.80

0.75

0.75

0.75

29

32

35

ML PE

%

0.85

0.80

26

0.90

%

%

1.00

0.85

23

DL PL


1.00

0.80

20

Coverage


1.00

0.85

17


1.00

%

5

ML PE

sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1

Coverage H²=2.78 − I²=64% (normal distribution)

17

number of studies

0.85 0.80

2

5

8

11

14

17

20

23

26

29

32

35

2

5

number of studies FE REML sk=0, ku=3

DL PL

8

11

14

17

20

23

26

29

32

number of studies ML PE

FE REML

DL PL

35

2

5

8

11

14

17

20

23

26

29

32


sk=2, ku=9

FE REML

DL PL

sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1

35

0.75 2

ML PE

5

8

11

14

17

20

23

26

29

DL PL



32



ML PE

35



Power performance Small and large heterogeneity under various distributional assumptions Power (25th centile)








1.00

1.00

0.90

0.90

0.90

0.90

0.80

0.80

0.80

0.80

0.70

0.70

0.70

0.60

0.60

0.60

0.50

0.50

0.50

%

%

%

1.00

%

1.00

0.70 0.60

2

5

8

11

14

17

20

23

26

29

32

35

2

5

8

11


14

17

20

23

26

29

32

35

2

5

8

11

number of studies

DL PL

ML PE

FE REML

sk=0, ku=3

DL PL

ML PE

17

20

FE REML

sk=2, ku=9

23

26

29

32

0.50

35

2

DL PL


8

0.90

0.90

0.90

0.90

0.80

0.80

0.80

0.60

0.50

0.50

0.50

26

29

32

35

ML PE

%

0.70

0.60

23

DL PL

0.80

%

%

1.00

0.70

20


1.00

0.60

17



1.00

0.70

14

number of studies



11

FE REML

1.00

%

5

ML PE

sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1

Power (25th centile) H²=2.78 − I²=64% (normal distribution)

14

number of studies

0.70 0.60

2

5

8

11

14

17

20

23

26

29

32

35

2

5


DL PL

8

11

14

17

20

23

26

29

32


FE REML

DL PL

35

2

5

8

11

14

17

20

23

26

29

32


sk=2, ku=9

FE REML

DL PL

sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1

35

0.50 2

ML PE

5

8

11

14

17

20

23

26

29

DL PL



32



ML PE

35



CI performance Small and large heterogeneity under various distributional assumptions Confidence Interval performance

Confidence Interval performance


Overall estimation performance (medians)


Overall estimation performance



2.00

2.00

2.00

2.00

1.80

1.80

1.80

1.80

1.60

1.60

1.60

1.60

1.40

1.40

1.40

1.20

1.20

1.20

1.00

1.00

1.00

0.80

0.80

0.80

0.60

0.60

0.60

0.40

0.40

0.40

2

5

8

11

14

17

20

23

26

29

32

35

2

5

8

11


14

17

20

23

26

29

32

35

1.40 1.20 1.00 0.80 0.60 2

5

8

11

number of studies

DL PL

ML PE

FE REML

sk=0, ku=3

DL PL

ML PE

20

23

26

29

32

35

0.40 2

DL PL

Overall estimation performance (medians)


2.00

1.80

1.80

1.60

1.60

1.60

1.60

1.40

1.40

1.40

1.20

1.20

1.20

1.00

1.00

1.00

0.80

0.80

0.80

0.60

0.60

0.60

0.40

0.40

0.40

14

17

20

23

26

29

32

35

2

5


DL PL

8

11

14

17

20

23

26

29

32


FE REML

DL PL

35

sk=2, ku=9

20

23

26

29

DL PL

35

1.20 1.00 0.80 0.60 2

5

8

11

14

17

20

23

26

29

32

FE REML

DL PL

sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1

35

0.40 2

ML PE

5

8

11

14

17

20

23

26

29

32


DL PL



32

ML PE

1.40


17


2.00

1.80

11

14

Overall estimation performance


2.00

8

11

number of studies

1.80

5

8

FE REML

2.00

2

5

ML PE

sk1=0, ku1=3, var1=.1, sk2=0, ku2=3, var2=.1



17

FE REML

sk=2, ku=9


14

number of studies


ML PE

35



Coverage by method Large heterogeneity across various between-study variance distributions FE − Coverage

ML − Coverage

H=2.78 − I=64%

PL − Coverage

H=2.78 − I=64%

1.00

H=2.78 − I=64%

1.00

1.00

0.95 0.95

0.90

0.95

0.85

0.90

0.75

%

%

%

0.80 0.85

0.90

0.70 0.80

0.65

0.85

0.60

0.75

0.55 0.50

0.70 2

5

8

11

14

17

20

23

26

29

32

35

0.80 2

5

8

11

number of studies Zero variance

Normal

Bimodal

Uniform

17

20

23

Zero variance

Normal

Bimodal

Uniform

DL − Coverage

26

29

32

35

2

5

8

11

0.95

0.90

0.85

20

23

Normal

Bimodal

Uniform

26

29

32

35

Skew−normal

PE − Coverage H=2.78 − I=64% 1.00

%

0.95

%

1.00

17

Zero variance

H=2.78 − I=64%

1.00

14

number of studies Skew−normal

REML − Coverage

H=2.78 − I=64%

%

14


0.90

0.95

0.85

0.80

0.80 2

5

8

11

14

17

20

23

26

29

32

35

0.90 2

5

8

11


Normal

Bimodal

Uniform

14

17

20

23

26

29

32

35

2

5

8

11


Zero variance

Normal

Bimodal

Uniform

14

17

20

23

26

29

32

35


Zero variance

Normal

Bimodal

Uniform

Skew−normal






Power by method Large heterogeneity across various between-study variance distributions FE − Power (25th centile)

ML − Power (25th centile)

H=2.78 − I=64% 1.00

0.90

0.90

0.90

0.80

0.80

0.70

0.70

0.80 0.70 0.60

%

0.60

0.60

0.50

0.50

0.40

0.40

0.30

0.30

0.20 5

8

11

14

17

20

23

26

29

32

35

0.30 0.20 0.10 0.00 2

5

8

11


Normal

Bimodal

Uniform

0.50 0.40

0.20 2

14

17

20

23

26

29

32

35

2

Skew−normal

Zero variance

Normal

Bimodal

Uniform

REML − Power (25th centile)

H=2.78 − I=64%

0.80

0.80

0.80

0.70

0.70

0.70

0.60

0.60

0.60

%

0.90

%

0.90

0.50 0.40

0.40

0.30

0.30

0.30

0.20

0.20

0.10

0.10 14

17

20

23

26

29

32

35

Normal

Bimodal

Uniform

Uniform

26

29

32

35

Skew−normal

0.10 5

8

11

14

17

20

23

26

29

32

35

2

5

8

11


23

Normal

0.20 2


20

0.50

0.40

11

17

H=2.78 − I=64%

0.90

0.50

14

Bimodal

PE − Power (25th centile) 1.00

8

11

Zero variance

H=2.78 − I=64% 1.00

5

8


1.00

2

5

number of studies

DL − Power (25th centile)

%

PL − Power (25th centile)

H=2.78 − I=64% 1.00

%

%

H=2.78 − I=64% 1.00

Zero variance

Normal

Bimodal

Uniform

14

17

20

23

26

29

32

35


Zero variance

Normal

Bimodal

Uniform

Skew−normal






CI performance by method Large heterogeneity across various between-study variance distributions FE − Overall estimation performance

ML − Overall estimation performance

H=2.78 − I=64%

PL − Overall estimation performance

H=2.78 − I=64%

H=2.78 − I=64%

2.00

2.00

2.00

1.80

1.80

1.80

1.60

1.60

1.60

1.40

1.40

1.40

1.20

1.20

1.20

1.00

1.00

1.00

0.80

0.80

0.80

0.60

0.60

0.40

0.40 2

5

8

11

14

17

20

23

26

29

32

35

0.60 0.40 2

5

8

11


Normal

Bimodal

Uniform

14

17

20

23

26

29

32

35

2

DL − Overall estimation performance

Zero variance

Normal

Bimodal

Uniform

REML − Overall estimation performance

H=2.78 − I=64% 1.80

1.80

1.60

1.60

1.60

1.40

1.40

1.40

1.20

1.20

1.20

1.00

1.00

1.00

0.80

0.80

0.80

0.60

0.60

0.40

0.40 14

17

20

23

26

29

32

35

Normal

Bimodal

Uniform

23

Normal Uniform

26

29

32

35

Skew−normal

0.40 5

8

11

14

17

20

23

26

29

32

35

2

5

8

11


20

0.60

2


17

H=2.78 − I=64%

1.80

11

14

Bimodal

PE − Overall estimation performance 2.00

8

11

Zero variance

H=2.78 − I=64% 2.00

5

8


2.00

2

5


Zero variance

Normal

Bimodal

Uniform

14

17

20

23

26

29

32

35


Zero variance

Normal

Bimodal

Uniform

Skew−normal






Which method then?

Within any given method, the results were consistent across all types of distribution shape Therefore methods are highly robust against even severe violations of the assumption of normality Choose PE if the priority is an accurate Type I error rate (false positive) But low power makes it a poor choice when control of the Type II error rate (false negative) is also important and it cannot be used with less than 6 studies






Which method then?

For very small study numbers (≤5) only PL gives coverage >90% and an acccurate CI PL has a ‘reasonable’ coverage in most situations, especially for moderate and large heterogeneiry, giving it an edge over other methods REML and DL perform similarly and better than PL only when heterogeneity is low (I 2 < 15%) The computational complexity of REML is not justified






Outline 1


2


3


4






Bring on the champagne?

Does not necessarily mean homogeneity Most methods use biased estimators and not uncommon to get a negative τˆ2 which is set to 0 by the model We identified a large percentage of cases where the estimators failed to identify existing heterogeneity In our simulations, for 5 studies and I 2 ≈ 29%: 30% of the meta-analyses were erroneously estimated to be homogeneous under the DL method 32% for REML and 48% for ML-PL






What does it mean?

In these cases coverage was substandard and was over 10% lower than in cases where τˆ2 > 0, on average The problem becomes less profound as the number of studies and the level of heterogeneity increase Better estimators are needed There might be a large number of meta-analyses of ‘homogeneous’ studies which have reached a wrong conclusion





A practical guide ipdforest methods ipdforest examples Power

Outline

1


2


3


4

Summary






Forest plot One advantage of two-stage meta-analysis is the ability to convey information graphically through a forest plot study effects available after the first stage of the process, and can be used to demonstrate the relative strength of the intervention in each study and across all informative, easy to follow and particularly useful for readers with little or no methodological experience key feature of meta-analysis and always presented when two-stage meta-analyses are performed

In one-stage meta-analysis, only the overall effect is calculated and creating a forest-plot is not straightforward Enter ipdforest [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Outline 1


2


3

One-stage meta-analysis A practical guide ipdforest methods ipdforest examples Power

4






The hypothetical study Individual patient data from randomised controlled trials For each trial we have a binary control/intervention membership variable baseline and follow-up data for the continuous outcome covariates

Assume measurements consistent across trials and standardisation is not required We will explore linear random-effects models with the xtmixed command; application to the logistic case using xtmelogit should be straightforward In the models that follow, in general, we denote fixed effects with ‘γ’s and random effects with ‘β’s [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Model 1 fixed common intercept; random treatment effect; fixed effect for baseline

Yi j = γ0 + β1 j groupi j + γ2 Yb i j + i j β1 j = γ1 + u1j i: the patient j: the trial Yi j : the outcome γ0 : fixed common intercept β1 j : random treatment effect for trial j γ1 : mean treatment effect groupi j : group membership

i j ∼ N(0, σj2 ) u1j ∼ N(0, τ1 2 )

γ2 : fixed baseline effect Yb i j : baseline score u1j : random treatment effect for trial j τ1 2 : between trial variance i j : error term σj2 : within trial variance for trial j






Model 1 fixed common intercept; random treatment effect; fixed effect for baseline

Possibly the simplest approach In Stata it can be expressed as xtmixed Y i.group Yb || studyid:group, nocons where studyid, the trial identifier group, control/intervention membership Y and Yb, endpoint and baseline scores note that the nocons option suppresses estimation of the intercept as a random effect






Model 2 fixed trial specific intercepts; random treatment effect; fixed trial-specific effects for baseline

Common intercept & fixed baseline difficult to justify A more accepted model allows for different fixed intercepts and fixed baseline effects for each trial: Yi j = γ0 j + β1 j groupi j + γ2 j Yb i j + i j β1 j = γ1 + u1j where γ0 j the fixed intercept for trial j γ2 j the fixed baseline effect for trial j

In Stata expressed as: xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4 || studyid:group, nocons where Yb‘i’=Yb if studyid=‘i’ and zero otherwise [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Model 3 random trial intercept; random treatment effect; fixed trial-specific effects for baseline

Another possibility, althought contentious, is to assume trial intercepts are random (e.g. multi-centre trial): Yi j = β0 j + β1 j groupi j + γ2 j Yb i j + i j β0 j = γ0 + u0 j β1 j = γ1 + u1j wiser to assume random effects correlation ρ 6= 0: i j ∼ N(0, σj2 ) u0 j ∼ N(0, τ02 ) u1j ∼ N(0, τ12 ) cov (u0 j , u1j ) = ρτ0 τ1

In Stata expressed as: xtmixed Y i.group Yb1 Yb2 Yb3 Yb4 || studyid:group, cov(uns) cov(uns): allows for distinct estimation of all RE variance-covariance components [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Model 4 random trial intercept; random treatment effect; random effects for baseline

The baseline could also have been modelled as a random-effect: Yi j = β0 j + β1 j groupi j + β2 j Yb i j + i j β0 j = γ0 + u0 j β1 j = γ1 + u1j β2 j = γ2 + u2j as before, non-zero random effects correlations: u0 j ∼ N(0, τ02 ) u1j ∼ N(0, τ12 ) 2 u2j ∼ N(0, τ2 ) cov (u0 j , u1j ) = ρ1 τ0 τ1 cov (u0 j , u2j ) = ρ2 τ0 τ2 cov (u1j , u2j ) = ρ3 τ1 τ2

In Stata expressed as: xtmixed Y i.group Yb || studyid:group Yb, cov(uns) [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Model 5 Interactions and covariates

A covariate or an interaction term can be modelled as a fixed or random effect Assuming continuous and standardised variable age we can expand Model 2 to include fixed effects for both age and its interaction with the treatment: Yi j = γ0 j + β1 j groupi j + γ2 j Yb i j + γ3 agei j + γ4 groupi j agei j + i j β1 j = γ1 + u1j

In Stata expressed as: xtmixed Y i.group i.studyid Yb1 Yb2 Yb3 Yb4 age i.group#c.age || studyid:group, nocons

If modelled as a random effect, non-convergence issues more likely to be encountered [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Outline 1


2


3


4






General ipdforest is issued following an IPD meta-analysis that uses mixed effects two-level regression, with patients nested within trials and a linear model (xtmixed) or logistic model (xtmelogit)

Provides a meta-analysis summary table and a forest plot Trial effects are calculated within ipdforest Can calculate and report both main and interaction effects Overall effect(s) and variance estimates are extracted from the preceding regression [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Process ipdforest estimates individual trial effects and their standard errors using one-level linear or logistic regressions Following xtmixed, regress is used and following xtmelogit, logit is used, for each trial ipdforest controls these regressions for fixed- or random-effects covariates that were specified in the preceding two-level regression User has full control over included covariates in the command (e.g. specification as fixed- or random-effects) But we strongly recommend using the same specifications [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Estimation details In the estimation of individual trial effects, ipdforest controls for a random-effects covariate (i.e. allowing the regression coefficient to vary by trial) by including the covariate as an independent variable in each regression Control for a fixed-effect covariate (regression coefficient assumed constant across trials and given by the coefficient estimated under two-level model) is a little more complex. Not possible to specify a fixed value for a regression coefficient under regress and the continuous outcome variable is adjusted by subtracting the contribution of the fixed covariates to its values in a first step prior to analysis For a binary outcome the equivalent is achieved through use of the offset option in logit [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Reported heterogeneity 2, For continuous outcomes, ipdforest reports, I 2 and HM based on the xtmixed output For binary outcomes, an estimate of the within-trial variance is not reported under xtmelogit and hence heterogeneity measures cannot be computed Between-trial variability estimate τˆ2 and its confidence interval is reported under both models Random-effects model approach is more conservative, accounting for even low levels of τ 2 Described in:

Kontopantelis E and Reeves D A short guide and a forest plot command (ipdforest) for one-stage meta-analysis The Stata[Poster Journal, in print title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Outline 1


2


3


4






Depression intervention We apply the ipdforest command to a dataset of 4 depression intervention trials Complete information in terms of age, gender, control/intervention group membership, continuous outcome baseline and endpoint values for 518 patients Results not published yet; we use fake author names and generated random continuous & binary outcome variables, while keeping the covariates at their actual values Introduced correlation between baseline and endpoint scores and between-trial variability Logistic IPD meta-analysis, followed by ipdforest [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Dataset . use ipdforest_example.dta, . describe Contains data from ipdforest_example.dta obs: 518 vars: 17 size: 20,202

variable name studyid patid group sex age depB depBbas depBbas1 depBbas2 depBbas5 depBbas9 depC depCbas depCbas1 depCbas2 depCbas5 depCbas9 Sorted by:

storage type byte int byte byte float byte byte byte byte byte byte float float float float float float

studyid

display format

value label

%22.0g %8.0g %20.0g %10.0g %10.0g %9.0g %9.0g %9.0g %9.0g %9.0g %9.0g %9.0g %9.0g %9.0g %9.0g %9.0g %9.0g

stid grplbl sexlbl

6 Feb 2012 11:14

variable label Study identifier Patient identifier Intervention/control group Gender Age in years Binary outcome, endpoint Binary outcome, baseline Bin outcome baseline, trial 1 Bin outcome baseline, trial 2 Bin outcome baseline, trial 5 Bin outcome baseline, trial 9 Continuous outcome, endpoint Continuous outcome, baseline Cont outcome baseline, trial 1 Cont outcome baseline, trial 2 Cont outcome baseline, trial 5 Cont outcome baseline, trial 9

patid [Poster title]






ME logistic regression model - continuous interaction fixed trial intercepts; fixed trial effects for baseline; random treatment and age effects . xtmelogit depB group agec sex i.studyid depBbas1 depBbas2 depBbas5 depBbas9 i > .group#c.agec || studyid:group agec, var nocons or Mixed-effects logistic regression Group variable: studyid

Number of obs Number of groups Obs per group: min avg max Wald chi2(11) Prob > chi2

Integration points = 7 Log likelihood = -326.55747 Std. Err.

z

= = = = = = =

518 4 42 129.5 214 42.06 0.0000

depB

Odds Ratio

P>|z|

[95% Conf. Interval]

group agec sex

1.840804 .9867902 .7117592

.3666167 .0119059 .1540753

3.06 -1.10 -1.57

0.002 0.270 0.116

1.245894 .9637288 .4656639

2.71978 1.010403 1.087912

studyid 2 5 9

1.050007 .8014551 1.281413

.5725516 .5894511 .6886057

0.09 -0.30 0.46

0.929 0.763 0.644

.3606166 .189601 .4469619

3.057303 3.387799 3.673735

depBbas1 depBbas2 depBbas5 depBbas9

3.152908 4.480302 2.387336 1.881203

1.495281 1.863908 1.722993 .7086507

2.42 3.60 1.21 1.68

0.015 0.000 0.228 0.093

1.244587 1.982385 .5802064 .8990569

7.987253 10.12574 9.823007 3.936262

group#c.agec 1

1.011776

.0163748

0.72

0.469

.9801858

1.044385

_cons

.5533714

.2398342

-1.37

0.172

.2366472

1.293993

Random-effects Parameters

Estimate

Std. Err.

studyid: Independent var(group) var(agec)

8.86e-21 5.99e-18

2.43e-11 4.40e-11


0 0

. .






ipdforest modelling main effect and interaction . ipdforest group, fe(sex) re(agec) ia(agec) or One-stage meta-analysis results using xtmelogit (ML method) and ipdforest Main effect (group) Study

Effect

Hart 2005 Richards 2004 Silva 2008 Kompany 2009

2.118 2.722 2.690 1.895

[95% Conf. Interval] 0.942 1.336 0.748 0.969

4.765 5.545 9.676 3.707

% Weight 19.88 30.69 8.11 41.31

Overall effect

1.841

1.246

2.720

100.00

One-stage meta-analysis results using xtmelogit (ML method) and ipdforest Interaction effect (group x agec) Study

Effect


0.972 0.995 0.987 1.077

[95% Conf. Interval] 0.901 0.937 0.888 1.015

1.049 1.055 1.098 1.144

% Weight 19.88 30.69 8.11 41.31

Overall effect

1.012

0.980

1.044

100.00

Heterogeneity Measures value I^2 (%) H^2 tau^2 est

. . 0.000


0.000

.






Forest plots

Hart 2005

Hart 2005

Richards 2004

Richards 2004

Studies

Studies

main effect and interaction

Silva 2008

Silva 2008

Kompany 2009

Kompany 2009

Overall effect

Overall effect

0

1

2

3

4

5

6

7

8

9

10

Effect sizes and CIs (ORs) Main effect (group)

0

.2

.4

.6

.8

1

1.2

1.4

Effect sizes and CIs (ORs) Interaction effect (group x agec)






ME logistic regression model - binary interaction fixed trial intercepts; fixed trial effects for baseline & age cat; random treatment effect . xtmelogit depB group i.agecat sex i.studyid depBbas1 depBbas2 depBbas5 depBba > s9 i.group#i.agecat || studyid:group, var nocons or Mixed-effects logistic regression Group variable: studyid

Number of obs Number of groups Obs per group: min avg max Wald chi2(11) Prob > chi2

Integration points = 7 Log likelihood = -326.24961 Std. Err.

518 4 42 129.5 214 42.67 0.0000

depB

Odds Ratio

P>|z|


group 1.agecat sex

1.931763 .7389353 .7173044

.5702338 .213031 .154959

2.23 -1.05 -1.54

0.026 0.294 0.124

1.083151 .4199616 .469698

3.445233 1.300179 1.095439

studyid 2 5 9

1.029638 .828577 1.266728

.5608887 .6082301 .6812765

0.05 -0.26 0.44

0.957 0.798 0.660

.3539959 .1965599 .4414556

2.994823 3.492777 3.634794

depBbas1 depBbas2 depBbas5 depBbas9

3.196139 4.625802 2.354493 1.902359

1.515334 1.923028 1.692149 .7183054

2.45 3.68 1.19 1.70

0.014 0.000 0.233 0.089

1.261999 2.047989 .5756364 .9075907

8.094542 10.44832 9.630454 3.987446

group#agecat 1 0 1 1

.9277371 1

.3558053 (omitted)

-0.20

0.845

.4374944

1.967331

_cons

.6175744

.2800575

-1.06

0.288

.253914

1.502076

Random-effects Parameters

z

= = = = = = =

Estimate

Std. Err.

2.24e-19

1.24e-10


studyid: Identity var(group)

0

.






ipdforest modelling main effect and interaction . ipdforest group, fe(sex i.agecat) ia(i.agecat) or One-stage meta-analysis results using xtmelogit (ML method) and ipdforest Main effect (group), agecat=0 Study

Effect


3.594 2.814 3.750 1.010

[95% Conf. Interval] 1.133 1.034 0.585 0.475

11.402 7.657 24.038 2.147

% Weight 19.88 30.69 8.11 41.31

Overall effect

1.792

1.079

2.976

100.00

One-stage meta-analysis results using xtmelogit (ML method) and ipdforest Main effect (group), agecat=1 Study

Effect


1.166 2.566 1.935 3.551

[95% Conf. Interval] 0.376 1.007 0.340 1.134

3.617 6.543 11.003 11.126

% Weight 19.88 30.69 8.11 41.31

Overall effect

1.932

1.083

3.445

100.00

Heterogeneity Measures value I^2 (%) H^2 tau^2 est

. . 0.000


0.000

.






Forest plots

Mead 2005

Mead 2005

Willemse 2004

Willemse 2004

Studies

Studies

main effect for each age group

Lovell 2008

Lovell 2008

Meyer 2009

Meyer 2009

Overall effect

Overall effect

0

1

2

4

6

8

10

12

14

16

18

20

22

24

26

Effect sizes and CIs (ORs) Main effect (group), agecat=0

0

1

2

4

6

8

10

Effect sizes and CIs (ORs) Main effect (group), agecat=1




12



Outline 1


2


3


4






Power calculations to detect a moderator effect

The best approach is through simulations As always, numerous assumptions need to be made effect sizes (main and interaction) exposure and covariate distributions correlation between variables within-study (error) variance between-study (error) variance - ICC

Generate 100s of data sets using the assumed model(s) Estimate what % of these give a significant p-value for the interaction Can be trial and error till desired power level achieved [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]





Research on power calculations with simulations Arnold et al. BMC Medical Research Methodology 2011, 11:94 http://www.biomedcentral.com/1471-2288/11/94

The Stata Journal (2002) 2, Number 2, pp. 107–124

Power by simulation A. H. Feiveson NASA Johnson Space Center [email protected] Abstract. This paper describes how to write Stata programs to estimate the power of virtually any statistical test that Stata can perform. Examples given include the t test, Poisson regression, Cox regression, and the nonparametric rank-sum test.

COMMENTARY

Benjamin F Arnold1*†, Daniel R Hogan2†, John M Colford Jr1 and Alan E Hubbard3

Abstract Background: Estimating the required sample size and statistical power for a study is an integral part of study design. For standard designs, power equations provide an efficient solution to the problem, but they are unavailable for many complex study designs that arise in practice. For such complex study designs, computer simulation is a useful alternative for estimating study power. Although this approach is well known among statisticians, in our experience many epidemiologists and social scientists are unfamiliar with the technique. This article aims to address this knowledge gap. Methods: We review an approach to estimate study power for individual- or cluster-randomized designs using computer simulation. This flexible approach arises naturally from the model used to derive conventional power equations, but extends those methods to accommodate arbitrarily complex designs. The method is universally applicable to a broad range of designs and outcomes, and we present the material in a way that is approachable for quantitative, applied researchers. We illustrate the method using two examples (one simple, one complex) based on sanitation and nutritional interventions to improve child growth. Results: We first show how simulation reproduces conventional power estimates for simple randomized designs over a broad range of sample scenarios to familiarize the reader with the approach. We then demonstrate how to extend the simulation approach to more complex designs. Finally, we discuss extensions to the examples in the article, and provide computer code to efficiently run the example simulations in both R and Stata. Conclusions: Simulation methods offer a flexible option to estimate statistical power for standard and nontraditional study designs and parameters of interest. The approach we have described is universally applicable for evaluating study designs used in epidemiologic and social science research.

Keywords: st0010, power, simulation, random number generation, postfile, copula, sample size

1

Introduction

Statisticians know that in order to properly design a study producing experimental data, one needs to have some idea of whether the scope of the study is sufficient to give a reasonable expectation that hypothesized effects will be detectable over experimental error. Most of us have at some time used published procedures or canned software to obtain sample sizes for studies intended to be analyzed by t tests, or even analysis of variance. However, with more complex methods for describing and analyzing both continuous and discrete data (for example, generalized linear models, survival models, selection models), possibly with provisions for random effects and robust variance estimation, the closed-form expressions for power or for sample sizes needed to achieve a certain power do not exist. Nevertheless, Stata users with moderate programming ability can write their own routines to estimate the power of virtually any statistical test that Stata can perform.

2 2.1

General approach to estimating power Statistical inference

Before describing methodology for estimating power, we first define the term “power” and illustrate the quantities that can affect it. In this article, we restrict ourselves to classical (as opposed to Bayesian) methodology for performing statistical inference on the effect of experimental variable(s) X on a response Y . This is accomplished by calculating a p-value that attempts to probabilistically[Poster describetitle] the extent to which the data are consistent with a null hypothesis, H0 , that X has no effect on Y . We would then reject H0 if the p-value is less than a predesignated threshold α. It should be pointed out that considerable criticism of the use of p-values for statistical inference has appeared in the literature; for example, Berger and Sellke (1987), Loftus (1993), and

Keywords: Computer Simulation, Power, Research Design, Sample Size

Background Estimating the sample size and statistical power for a study is an integral part of study design and has profound consequences for the cost and statistical precision of a study. There exist analytic (closed-form) power equations for simple designs such as parallel randomized trials with treatment assigned at the individual level or cluster (group) level [1]. Statisticians have also derived equations to estimate power for more complex designs, such as designs with two levels of correlation [2] or designs with two levels of correlation, multiple treatments and



Open Access

Simulation methods to estimate design power: an overview for applied research

attrition [3]. The advantage of using an equation to estimate power for study designs is that the approach is fast and easy to implement using existing software. For this reason, power equations are used to inform most study designs. However, in our applied research we have routinely encountered study designs that do not conform to conventional power equations (e.g. multiple treatment interventions, where one treatment is deployed at the group level and a second at the individual level). In these situations, simulation techniques offer a flexible alternative that is easy to implement in modern statistical software. Here, we provide an overview of a general method to estimate study power for randomized trials based on a

* Correspondence: [email protected] Software and model selection challenges in meta-analysis † Contributed equally 1


Outline

1


2


3


4

Summary





Two-stage meta-analysis MetaEasy can help you organise your meta-analysis and can be especially useful if you need to combine continuous and binary outcome. Methods implemented in Stata under metaeff and metaan A zero τˆ2 is a reason to worry; heterogeneity might be there but we cannot measure or account for in the model If τˆ2 > 0, even if very small, use a random-effects model The DL method works reasonably well, under all distributions, especially for low levels of heterogeneity Profile likelihood, which takes into account the uncertaintly in τˆ2 , works better when I 2 ≥ 15% [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]




One-stage meta-analysis A few different approaches exist for conducting one-stage IPD meta-analysis Stata can cope through the xtmixed and the xtmelogit commands The ipdforest command aims to help meta-analysts calculate trial effects display results in standard meta-analysis tables produce familiar and ‘expected’ forest-plots

The easiest way to calculate power to detect complex effects is through simulations [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]



Appendix

Thank you! Key references

Comments, suggestions: [email protected] [Poster title] [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]



Appendix

Thank you! Key references

References Kontopantelis E, Reeves D. A short guide and a forest plot command (ipdforest) for one-stage meta-analysis. The Stata Journal, in print. Kontopantelis E, Reeves D. Performance of statistical methods for meta-analysis when true study effects are non-normally distributed: A simulation study. Stat Methods Med Res, published online Dec 9 2010. Kontopantelis E, Reeves D. metaan: Random-effects meta-analysis. The Stata Journal, 10(3):395-407, 2010. Kontopantelis E, Reeves D. MetaEasy: A Meta-Analysis Add-In for Microsoft Excel. [Poster title] Journal of Statistical Software, 30(7):1-25, 2009. [Replace the following names and titles with those of the actual contributors: Helge Hoeing, PhD1; Carol Philips, PhD2; Jonathan Haas, RN, BSN, MHA3, and Kimberly B. Zimmerman, MD4 1[Add affiliation for first contributor], 2[Add affiliation for second contributor], 3[Add affiliation for third contributor], 4[Add affiliation for fourth contributor]

