Comparative Effectiveness Research - Semantic Scholar

RESEARCH TOOLS AND ISSUES

Comparative Effectiveness Research: What Kind of Studies Do We Need? John Concato, MD, MS, MPH,*þ Peter Peduzzi, PhD,Þ§ Grant D. Huang, PhD,||¶ Timothy J. O’Leary, MD, PhD,||¶ and Joel Kupersmith, MD¶

Abstract: Comparative effectiveness research (CER) is increasingly popular, yet discussions of its conduct and consequences often overlook the extensive history of comparing different therapeutic options in patient-oriented research. In particular, research in the Department of Veterans Affairs (VA) has included a decades-long focus on generating information that can enhance medical decision making and improve health outcomes. Categories of such research include multisite randomized controlled trials (conducted by the Cooperative Studies Program) and observational studies involving either primary or secondary data collection. As representative examples from cardiology, a landmark VA clinical trial published in the 1970s evaluated the benefits of coronary artery bypass grafting surgery among patients with angina; a VA trial initiated in the 1990s, and identified formally as CER, demonstrated that percutaneous coronary intervention is not superior to optimal medical therapy; and a database investigation using information from the VA electronic medical record system in the 2000s found that use of proton pump inhibitor medication is associated with the attenuation of the benefits of clopidogrel among patients hospitalized for acute coronary syndrome. A review of these (and other) selected projects, based on their type of study design, serves to highlight the strengths, limitations, and potential of CER. Key Words: comparative effectiveness research, randomized controlled trials, observational studies, database research, meta-analysis, US veterans (J Investig Med 2010;58: 764Y769)

C

omparative effectiveness research (CER) evaluates the relative effectiveness of different options for treating a specific medical condition in a selected population.1Y4 Comparative effectiveness research is not a single entity; it can focus on patientlevel or system-level elements of care, involving medications or procedures, with or without considering costs. Regardless of its scope, CER has the potential to generate information that can

From the *Clinical Epidemiology Research Center, and †Cooperative Studies Program Coordinating Center, Veterans Affairs (VA) Connecticut Healthcare System, West Haven; ‡Schools of Medicine, and §Public Health, Yale University, New Haven, CT; ||VA Cooperative Studies Program Central Office; and ¶Office of Research and Development, Veterans Health Administration, Washington, DC. Received January 11, 2010, and in revised form March 29, 2010. Accepted for publication March 29, 2010. Reprints: John Concato, MD, MS, MPH, Clinical Epidemiology Research Center, VA Connecticut Healthcare System, 950 Campbell Ave, Mailcode 151B, West Haven, CT 06516. E-mail: [email protected]; [email protected]. The authors have no financial disclosures to report. The views in this article are those of the authors and do not reflect an official position or policy of the Department of Veteran Affairs. Dr. Kupersmith serves on the Federal Coordinating Council for Comparative Effectiveness Research. Copyright * 2010 by The American Federation for Medical Research ISSN: 1081-5589 DOI: 10.231/JIM.0b013e3181e3d2af

764

enhance medical decision making and improve health outcomes, but concerns exist regarding its conduct and consequences.5,6 Recent attention to CER has discussed mechanisms to implement findings into practice,7,8 potential impacts on personalized medicine,9 and disagreements regarding funding.5 Less attention has been given to strategies of the design and conduct of such research. Whereas discussions of nonscientific issues are ongoing, an internal look at the methods of CER is also warranted. In addition, selected examples from the Department of Veterans Affairs (VA) can illustrate one organization’s experience in conducting CER.

HISTORICAL OVERVIEW Comparative effectiveness research can be considered a subset of patient-oriented research that examines available therapeutic options among particular patients to determine relevant health outcomes. As such, CER is a relatively recent term, but the idea of evaluating Breal world[ treatment options dates back to antiquity. As a centuries-old example, James Lind compared various treatments for scurvy in the mid-1700s10; providing citrus fruits led to resolution of symptoms, whereas vinegar, cider, and other remedies did not. (This example also demonstrates that valid associations can be identified before a full explanation of underlying mechanisms; the isolation of vitamin C and a biological understanding of scurvy did not occur until almost 200 years later.) In the 20th century, the so-called comparative effectiveness of 2 penicillin treatment schedules in pneumococcal infections of mice11 was evaluated in 1948, but the phrase Bdosing study[ might be applied to the same research today. In the same era, the first rigorous studies of the treatment of tuberculosis in humans involved the US Veterans Administration, Army, and Navy collaborating to evaluate streptomycin among veterans and soldiers whose condition was diagnosed after World War II.12 This observational study12 predated, and found results similar to, a randomized trial of streptomycin conducted in the United Kingdom by the Medical Research Council.13 (The latter study13 is acknowledged as the first randomized trial reported in modern clinical medicine.) In the last half-century, terminology for research design has evolved, based specifically on the meaning and application of the words Befficacy[ and Beffectiveness.[ Attributable in large part to the passage (in 1962) of the Kefauver-Harris Amendment14 to the Federal Food, Drug, and Cosmetic Act and developed in the wake of thalidomide causing phocomelia and other birth defects, this amendment required proof of efficacy as part of the drug approval process in the United States. The subsequent adoption of phase I to III (and later phase 0 and phase IV) schema for clinical trials emphasizes the evaluation of new drugs versus placebo rather than with existing agents. This system also places more emphasis on internal validityVbased on Befficacy trials[ that tend to have restrictive inclusion, exclusion, and treatment criteriaVthan on external validity derived from Beffectiveness trials[ that tend to have patient populations and (nonplacebo)

Journal of Investigative Medicine

&

Volume 58, Number 6, August 2010

Copyright © 2010 American Federation for Medical Research. Unauthorized reproduction of this article is prohibited.


&


interventions more representative of clinical practice. Implementation of research results after the publication of trials (or other types of studies) is another relatively recent consideration. By any reasonable definition, numerous studies of comparative effectiveness have been done in recent decades, but usually without being identified as such. Interestingly, the formal term Bcomparative effectiveness research[ has been popularized only in the past several years, related more to policy15 and economic16 considerations than to scientific issues themselves. Whether an article is classified as comparative effectiveness is variable, depending on the authors and the viewpoint of reviewers and journal editors. A recent review4 classified 689 studies as CER based on a search of clinicaltrials.gov,17 a database of clinical trials, and HSRProj,18 a database of health services research projects. The visibility of CER was increased by an appropriation of $1.1 billion in the American Recovery and Reinvestment Act of 2009.19 Although this amount is less than 1% of the $150 billion directed toward health care overall, the support represents a major infusion of funding for CER.19 Even before this new initiative, however, CER had become a prominent component of research on health care. As the purposes and implications of CER are debated in medical and nonmedical venues, a major unanswered question is what kind of CER will be considered worthwhile? For example, in the current era of evidence-based medicine, a hierarchy of research designs might be promoted for CER that favors randomized trials. Conversely, the relatively short timeframe to spend Bstimulus funds[ encourages the use of secondary sources of data that can be analyzed promptly.

CATEGORIES OF RESEARCH DESIGN From a methodological perspective, randomized controlled trials can address comparative effectiveness and are arguably underused for this purpose. Specifically, studies funded by the pharmaceutical industry represent most trials in the United States but are usually designed to evaluate efficacy and not effectiveness. As mentioned, randomized trials evaluating efficacy (or explanatory clinical trials) often compare an intervention to placebo among patients with little or no comorbid illness to address whether an intervention can work under ideal circumstances. Early-phase clinical trials and applications to the Food and Drug Administration for licensure, including studies conducted or supported directly by the pharmaceutical industry, tend to fall into this category. Randomized trials evaluating effectiveness (or Bpractical clinical trials[) determine whether an intervention actually works under more representative, or real-world, conditions. These studies compare an intervention to a standard-of-care among patients representative of a target population. Randomized trials have often been criticized as lacking generalizability,20 but effectiveness trials are more likely to apply to general medical practice than efficacy trials. Specific methodological strategies, such as adaptive designs, represent novel features of trials that can improve their utility. Random allocation promotes a balance of prognostic factors among groups being comparedVprecluding a major source of biasVbut randomized trials are not always feasible, ethical, or necessary. Well-designed observational cohort studies and casecontrol studies collecting primary data constitute another approach to CER. These studies have enhanced generalizability in representative patient populations (especially compared with efficacy trials), yet can produce results similar to randomized trials when sufficient attention is given to sources of bias.21,22 In this context, an often-cited example of hormone replacement * 2010 The American Federation for Medical Research

Comparative Effectiveness Research

therapy and subsequent health outcomes has generated misconceptions regarding the limitations of observational studies. In reality, when differences in patient populations, treatment regimens, and duration of follow-up are taken into account, data from randomized and observational studies on this topic (and others) are quite consistent.23Y25 So-called database research, involving information obtained for clinical or administrative purposes during health care delivery, is another (observational) method for evaluating comparative effectiveness. The generalizability of results can become unrivaled, given assessments of large and representative populations of health care recipients. Conversely, problems include the lack of randomization and concerns with data quality (such as BAre diagnoses listed in an administrative database accurate?[). Carefully validated analytic strategies are therefore critically important; comparison of results with randomized trials and other observational studies can help to enhance interpretation. These categories of research design26 are not exhaustive, but they do account for common, primary strategies of CER. Secondary strategies include systematic reviews and statistical modeling. Rather than obtaining data as observations on individual participants, systematic reviews (including metaanalyses) use information from previously completed studiesV but not without limitations.27 BSecondary[ statistical modeling uses novel techniques to assess cause-effect associations, typically with a data set that had already been collected and analyzed. Although most patient-oriented research investigations use a statistical model per se, this type of statistical modeling goes beyond original hypotheses and planned calculations to explore associations based on statistical theory and corresponding assumptionsVthat may or may not be justified.

EXAMPLES FROM THE DEPARTMENT OF VETERANS AFFAIRS (VA) The VA has conducted CER using various methodologies for many years. For example, the VA Cooperative Studies Program has been recognized for producing Bnumerous studies designed to support clinical and policy decisions.[20 The Cooperative Studies Program28,29 has generated more than 175 large-scale, multisite investigations, mostly as randomized trials of comparative effectiveness. Key elements of the Program include methodological, technical, and administrative personnel in a national infrastructure of specialized centers for conducting collaborative research. The Program also has established guidelines30 for the design, conduct, and analysis of its multisite studies. Selected trials over 4 decades have shown (Table 1) that coronary artery bypass surgery (vs medical therapy) improves survival among patients with left main coronary artery disease,31 steroid therapy (added to usual care) does not reduce mortality from septic shock,32 watchful waiting (vs transurethral resection) is a safe alternative among men less bothered by symptoms from benign prostatic hyperplasia,33 and percutaneous coronary intervention (vs optimal medical therapy) does not reduce the risk of cardiovascular events or death in stable angina.34 These trials were selected with the main goal of highlighting both a longstanding BVA experience[ and a broad scope of topics and methods, such as medical versus surgical interventions and usual care versus optimal therapy as a comparator. In conducting CER, and looking beyond the strengths or limitations of any individual study, the Cooperative Studies Program has been described by non-VA authors as Ban important source of practical clinical trials in the United States.[20

765


Concato et al


&


TABLE 1. Representative Examples of CER Conducted by VA, Organized According to Type of Study Design Study Design and Clinical Context Randomized controlled trials: Patients with angina pectoris found to have a significant lesion in the left main coronary artery Patients with clinical signs of systemic sepsis receiving intensive care Ambulatory men with symptoms of benign prostatic hyperplasia Patients with evidence of myocardial ischemia and stable coronary artery disease Observational studies involving primary data collection: Older patients undergoing major noncardiac surgery Database (observational) investigations of archival information: Patients after discharge from a hospital admission for acute coronary syndrome Systematic reviews (including meta-analyses): Patients with coronary artery disease considered eligible for coronary revascularization Secondary statistical modeling: Ambulatory patients receiving treatment for alcohol dependence

Observational studies with primary data collection also contribute to CER in the VA, originating from the Cooperative Studies Program (via epidemiology centers), individual investigators, and other sources. An example is the National Surgical Quality Improvement Program (NSQIP) that systematically measures and seeks to improve risk-adjusted surgical outcomes using validated methods. The NSQIP35 is based in the VA Health Services Research and Development Service and was expanded to the private sector by the American College of Surgeons.36 In a collaborative NSQIP project,37 abnormally low or high preoperative hematocrit levels were found to be associated with increased 30-day mortality and cardiac events after noncardiac surgery among elderly patients (Table 1). These results inform CER by identifying a potentially modifiable risk factor for developing targeted treatments, with a goal of improving outcomes. Health services research projects frequently use archived database information only, as in a recent evaluation38 of an adverse interaction between proton-pump inhibitor medications and clopidogrel after hospitalization for acute coronary syndrome (Table 1). This study is pertinent in merging and analyzing national-level VA data collected for other purposes, with careful attention given to possible sources of bias and with multiple sensitivity analyses. The Department of Veterans Affairs Health Services Research also supports the Quality Enhancement Research Initiative,39 designed to translate study findings into clinical practice via implementation research.7,8 As an example of a systematic review (Table 1), a report40 evaluated the comparative effectiveness of percutaneous coronary interventions and coronary artery bypass graft surgery (CABG); the focus was not limited to VA-based studies. After a structured literature search, results from 23 randomized trials were identified, reviewed, and analyzed. The authors concluded that compared with percutaneous coronary interventions, CABG was more effective in relieving angina, and led to fewer revascularizations, but had a higher risk for procedural stroke; Bsurvival to 10 years was similar[ for both procedures.40

766

Information to Improve Patient Care (Year) Coronary artery bypass grafting surgery improves survival compared with medical therapy (1976).30 Steroid therapy added to usual care is not effective in reducing mortality among patients with septic shock (1987).31 Watchful waiting is a safe alternative to surgery among men less bothered by urinary symptoms (1995).32 Percutaneous coronary intervention is not superior to optimal medical therapy (2007).33 Abnormally low or high preoperative hematocrit levels increase mortality after surgery (2007).36 Proton pump inhibitor medications are associated with attenuation of benefits of clopidogrel (2009).37 Coronary artery bypass surgery improves angina and decreases revascularizations versus percutaneous intervention, but with higher risk of stroke and similar survival (2007).39 Naltrexone may reduce levels of Bconsistent[ drinking even if time to first drink is not improved (2007).42

As an example of Bsecondary[ statistical modeling, data were obtained from a VA Cooperative Study41 and a similar trial conducted among nonveterans,42 evaluating naltrexone in the treatment of alcohol dependence (Table 1). Although the original trials were null in terms of not finding an improvement in outcomes, such as the time to the first day of drinking a threshold amount of alcohol, questions arose as to whether the intervention might have benefited certain participants. A followup study43 used longitudinal models with Bfixed polynomial trends over time[ and other statistical methods; the analyses generated estimates of the proportion of participants classified (on a given day) as abstaining, drinking sporadically, or drinking consistently. The authors reported that Bnaltrexone may have a clinically meaningful effect for alcohol-dependent patients[ regarding consistent drinking.43 These VA-based (and other) examples serve collectively to characterize a methodological spectrum of CER. Within the Veterans Health Administration, such research has changed health policy and influenced clinical care,44 as with recommendations and treatment patterns for anemia after a study comparing subcutaneous and intravenous administration of erythropoietin.45,46 Despite such examples to support an argument that research can and does directly change practice patterns, a counterargument would include the previously cited example31 of a randomized trial evaluating CABG surgery for patients with angina. Approximately 10 years after publication of the main report, questions still lingered as to whether the study was valid and should influence the care of patients47; the answers eventually emerged as Byes[ and Byes,[ respectively.48 Ultimately, the complex issue of how to change physicians’ behavior is beyond the scope of this report, but conducting and understanding rigorous and relevant CER is an important component of the process. A related issue involves the patient population receiving health care at VA medical facilities, representing the source population for VA-based research. For example, although the proportion of women in the VA is increasing, their representation * 2010 The American Federation for Medical Research



&


still falls far short of the one-to-one balance in the community. Similarly, veterans who receive VA health care tend to come from lower socioeconomic strata, often linked to private health insurance being less available or affordable to them. Thus, a question of generalizability (external validity) does exist when VA studies are considered, yet this concern is sometimes extrapolated inappropriately to cast doubt on the internal validity of a study, without giving due consideration to the methods involved. In other situations, this concern is not applicableYfor example, when VA studies are done in collaboration with the National Institutes of Health49 or as international collaborations.34 Overall, if examined as a body of knowledge, VA studies in CER have been considered reliable and relevant in improving care delivered to the general (nonveteran) population.20 In addition, an informal review (data not shown) indicates that VA research has contributed to a sizeable majority of the 100 categories listed in the Institute of Medicine’s recent report on national priorities for CER.50

GENERAL CONSIDERATIONS The VA is distinctive in supporting virtually all of the types of research design mentioned in this report and in promoting implementation of research results. Other organizations and agencies might embrace a hierarchy of research design so that randomized trials and meta-analyses are emphasized as the best forms of evidence. (Under such circumstances, the VA Cooperative Studies Program should be considered an important repository of randomized studies.) At the same time, however, any approach that imposes a research design hierarchy should be understood as placing a Bfilter[ between the generation of evidence and its interpretation, bypassing the question of whether such filtering is appropriate or how well it is done. This problem can be subtle, as when Bdefinitive randomized controlled trials[ versus Bweaker forms of evidence, such as nested case-control or well-designed cohort studies[ are presented as self-evident truths in discussions of CER.51 Conversely, other organizations and agencies might use only secondary sources of information, perhaps underestimating the corresponding challenges, for reasons including database research can be done more readily. A major take-home message of this report is that types of study design are an important feature of CER, but hierarchies of research design should not become a distraction in the process of scientific inquiry. A concurring opinion was articulated as: BThis is not a plea to abandon RCTs and replace them with observational studies. [I] Rather, it is a plea to investigators to continue to develop and improve their methods; to decision makers to avoid adopting entrenched positions about the nature of evidence; and for both to accept that the interpretation of evidence requires judgment.[52 A similar viewpoint appeared in a report on CER,53 with statements including Btraditional Fhierarchies of evidence_ are overly simplistic and should not necessarily guide the implementation of CER[; Ba range of methods are important in CER[; and Bthe right approach to a given CER study depends on the circumstances.[ (Of note, a recent descriptive report54 suggests that randomized trials constitute most of the published CER studies.) Another take-home message involves the attributes needed for an organization to conduct CER. The VA is certainly not alone in its capabilities and commitment to CER,1Y4,55 and it also is not the largest health research organization in the United States. Yet, the unique position of VA research in the national health care environment includes at least 4 elements that have sustained CER: a comprehensive and relatively stable infrastructure, support for methodological advances, research ideas * 2010 The American Federation for Medical Research


arising from clinical and administrative questions, and a commitment to excellence in research on clinical care. Studies of comparative effectiveness in the VA are also facilitated by an electronic medical record system, a uniform reimbursement mechanism, and a national health care delivery network. In particular, few national-level organizations have a similar ability to integrate clinical, laboratory, pharmacy, service use, and outcome data in one health care context. The VA also serves an example of an organization that can collaborate with the private sector and yet avoid common conflicts of interest, using a philosophy of Bscience comes first.[ As exemplified by the Cooperative Studies Program, strict policies govern the (intramural) design, conduct, interpretation, and regulatory aspects of a study, whereas unrestricted research grants (or in-kind support, such as medications) from industry are acceptable. Cooperative Research and Development Agreements (CRADAs), authorized under the Federal Technology Transfer Act, represent one strategy used to safeguard the interests of the VA and veterans while being sensitive to the perspective of industry regarding commercially valuable information. Research conducted under a Cooperative Research and Development Agreement concurs with federal statutes, regulations, and policies and, at the same time, protects background inventions, trade secrets, and confidential information. A representative statement would mention that B[the industry sponsor] had no role in the design of the study, collection, and analysis of data, or preparation and approval of the manuscript.[ Given the paucity of incentives for the pharmaceutical or medical device industry to conduct CER on their own, this category of research represents an important opportunity to advance knowledge. Finally, details pertaining to cost-effectiveness analysis are beyond the scope of this paper, but a decision not to focus on costs in studies of CER is related more to policy than to science. Interestingly, the integrated nature of the VA health care system is a potential advantage in terms of the ability to assess the costs of treatment options accurately on a head-to-head basis.

CONCLUSIONS If a national-level entity dedicated to CER1,5,15,51,56 is created, either as a public or private (or hybrid) agency, the culture of such an organization would likely influence the selection of types of study design, but the methodological issues raised in this report should be considered. Rather than adhering to a hierarchy of research design, comparative effectiveness research should be conducted and evaluated using sound clinical, epidemiological, and statistical judgment. Beyond these attributes of CER, and in the broader context of health care reform,57,58 careful attention should be given to methods of patient-oriented research. Although not unique to CER, important steps include identifying an important research question; developing a cogent study design among feasible options; collecting relevant, high-quality data; and performing appropriate statistical analyses. Ultimately, precise ratings of interventions based on CER are arguably neither feasible nor desirable, and the source of CER is less pertinent than creating and disseminating rigorous research. The utility of a particular CER study should be judged in a peer-reviewed, scientific context and applied to individual patients by competent and caring clinicians. REFERENCES 1. IOM (Institute of Medicine). 2007. Learning What Works Best: The Nations Need for Evidence on Comparative Effectiveness in Health Care. Available at: http://www.iom.edu/ebm-effectiveness. Accessed March 24, 2010.

767


Concato et al


&


2. Kupersmith J, Sung N, Genel M, et al. Creating a new structure for research on health care effectiveness. J Investig Med. 2005;53:67Y72.

28. Department of Veterans Affairs Cooperative Studies Program. Available at: www.csp.research.va.gov/. Accessed March 24, 2010.

3. Lauer MS. Comparative effectiveness research: the view from the NHLBI. J Am Coll Cardiol. 2009;53:1084Y1086.

29. Henderson WG, Lavori PW, Peduzzi P, et al. Biostatistics in clinical trials. In: Redmond CK, Colton T, eds. Cooperative Studies Program, US Department of Veterans Affairs. New York: John Wiley & Sons, Ltd; 2001:99Y115.

4. Hove E, Pittman P. A First Look at the Volume and Cost of Comparative Effectiveness Research in the United States. Washington, DC: AcademyHealth; June 2009. 5. Avorn J. Debate about funding comparative-effectiveness research. N Engl J Med. 2009;360:1927Y1929. 6. Selker HP. Comparative effectiveness research: medical practice, payments, and politics: the need to retain standards of medical research. J Gen Int Med. 2009;24:776Y778. 7. Naik AD, Petersen LA. The neglected purpose of comparative-effectiveness research. N Engl J Med. 2009;360:1929Y1931. 8. Alexander GC, Stafford RS. Does comparative effectiveness have a comparative edge? JAMA. 2009;310:2488Y2490. 9. Garber AM, Tunis SR. Does comparative-effectiveness research threaten personalized medicine? N Engl J Med. 2009;360:1925Y1927. 10. Lind J. A Treatise of the Scurvy in Three Parts. London: A. Millar; 1753. 11. Gibson CD. Comparative effectiveness of two penicillin treatment schedules in pneumococcal infections of mice. Proc Soc Exp Biol Med. 1948;67:278Y280. 12. Streptomycin Committee. The effects of streptomycin on tuberculosis in man. JAMA. 1947;5135:634Y641. 13. Medical Research Council. Streptomycin treatment of pulmonary tuberculosis. Br Med J. 1948;2:769Y782. 14. Kefauver-Harris Amendments, 87 U.S. Constitution, Section 2 (1962). 15. Wilensky GR. Developing a center for comparative effectiveness information. Health Aff. 2006;25:w572Yw585. 16. Neumann PJ, Rosen AB, Weinstein MC. Medicare and cost-effectiveness analysis. N Engl J Med. 2005;353:1516Y1522. 17. ClinicalTrials.gov: a service of the U.S. National Institutes of Health. Available at: http://www.clinicaltrials.gov/. Accessed March 24, 2010. 18. HSRProj (Health Services Research Projects in Progress), National Information Center on Health Services Research and Health Care Technology (NICHSR). Available at www.nlm.nih.gov/hsrproj/. Accessed March 24, 2010. 19. Steinbrook R. Health care and the American Recovery and Reinvestment Act. N Engl J Med. 2009;360:1057Y1060. 20. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA. 2003;290:1624Y1632. 21. Concato J, Shaw N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342:1887Y1892. 22. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342:1878Y1886. 23. Concato J, Horwitz RI. Beyond randomised versus observational studies. Lancet. 2004;363:1660Y1661. 24. Grodstein F, Clarkson TB, Manson JE. Understanding the divergent data on postmenopausal hormone therapy. N Engl J Med. 2003;348:648Y650. 25. Garbe E, Suissa S. Hormone replacement therapy and acute coronary outcomes: methodological issues between randomized and observational studies. Hum Reprod. 2004;19:8Y13. 26. Concato J. Overview of research design in epidemiology. Journal of Law and Policy. 2004;12:489Y507. 27. Feinstein AR. Meta-analysis: statistical alchemy for the 21st century. J Clin Epidemiol. 1995;48:71Y79.

768

30. Guidelines for the planning and conduct of cooperative studies. Washington, DC: Office of Research and Development, Department of Veterans Affairs. May 2007. Available at: http://www.csp.research.va.gov/update/guide.pdf. Accessed March 24, 2010. 31. Takaro T, Hultgren HN, Lipton MJ, et al. The VA cooperative randomized study of surgery for coronary arterial occlusive disease II. Subgroup with significant left main lesions. Circulation. 1976;54(suppl 6):III107YIII117. 32. The Veterans Administration Systemic Sepsis Cooperative Study Group. Effect of high-dose glucocorticoid therapy on mortality in patients with clinical signs of systemic sepsis. N Engl J Med. 1987;317:659Y665. 33. Wasson JH, Reda DJ, Bruskewitz RC, et al. A comparison of transurethral surgery with watchful waiting for moderate symptoms of benign prostatic hyperplasia. The Veterans Affairs Cooperative Study Group on Transurethral Resection of the Prostate. N Engl J Med. 1995;332:75Y79. 34. Boden WE, O’Rourke RA, Teo KK, et al. Optimal medical therapy with or without PCI for stable coronary disease. N Engl J Med. 2007;356:1503Y1516. 35. Khuri SF, Daley J, Henderson W, et al. The Department of Veterans Affairs’ NSQIP: the first national, validated, outcome-based, risk-adjusted, and peer-controlled program for the measurement and enhancement of the quality of surgical care. National VA Surgical Quality Improvement Program. Ann Surg. 1998;228:491Y507. 36. History of the ACS NSQIP (American College of Surgeons, National Surgical Quality Improvement Program). Available at: http://www.acsnsqip.org/main/about_history.asp. Accessed March 24, 2010. 37. Wu WC, Schifftner TL, Henderson WG, et al. Preoperative hematocrit levels and postoperative outcomes in older patients undergoing noncardiac surgery. JAMA. 2007;297:2481Y2488. 38. Ho PM, Maddox TM, Wang L, et al. Risk of adverse outcomes associated with concomitant use of clopidogrel and proton pump inhibitors following acute coronary syndrome. JAMA. 2009;301:937Y944. 39. VA Quality Enhancement Research Initiative. Available at: http://www.queri.research.va.gov/. Accessed March 24, 2010. 40. Bravata DM, Gienger AL, McDonald KM, et al. Systematic review: the comparative effectiveness of percutaneous coronary interventions and coronary artery bypass graft surgery. Ann Intern Med. 2007;147:703Y716. 41. Krystal JH, Cramer JA, Krol WF, et al. Naltrexone in the treatment of alcohol dependence. N Engl J Med. 2001;345:1734Y1739. 42. O’Malley SS, Sinha R, Grilo CM, et al. Naltrexone and cognitive behavioral coping skills therapy for the treatment of alcohol drinking and eating disorder features in alcohol-dependent women: a randomized controlled trial. Alcohol Clin Exp Res. 2007;31:625Y634. 43. Gueorguieva R, Wu R, Pittman B, et al. New insights into the efficacy of naltrexone based on trajectory-based reanalyses of two negative clinical trials. Biol Psychiatry. 2007;61:1290Y1295. 44. Kizer KW, Demakis JG, Feussner JR. Reinventing VA Health Care: systematizing quality improvement and quality innovation. Med Care. 2000;38(suppl I):7Y16. 45. Kaufman JS, Reda DJ, Fye CL, et al. Subcutaneous compared with intravenous epoetin in patients receiving hemodialysis. N Engl J Med. 1998;339:578Y583.

* 2010 The American Federation for Medical Research



&


46. Hynes DM, Stroupe KT, Kaufman JS, et al. Adherence to guidelines for ESRD anemia management. Am J Kidney Dis. 2006;47:455Y461. 47. Hultgren H, Takaro T, Detre K, et al. Criticisms of the study. Prog Cardiovasc Dis. 1986;28:391Y396. 48. Yusuf S, Zucker D, Peduzzi P, et al. Effect of coronary artery bypass graft surgery on survival: overview of 10-year results from randomised trials by the Coronary Artery Bypass Graft Surgery Trialists Collaboration. Lancet. 1994;334:563Y570. 49. Palevsky PM, Zhang JH, O’Connor TZ, et al. Intensity of renal support in critically ill patients with acute kidney injury. N Engl J Med. 2008;359:7Y20. 50. IOM (Institute of Medicine). Initial National Priorities for Comparative Effectiveness Research. Washington, DC: The National Academies Press; 2009. 51. VanLare JM, Conway PH, Sox HC. Five next steps for a new national program for comparative-effectiveness research. N Engl J Med. 2010;362:970Y973.

* 2010 The American Federation for Medical Research


52. Rawlins M. De testimonio: on the evidence for decisions about the use of therapeutic interventions. Lancet. 2008;372:2152Y2161. 53. Tunis ST. Strategies to improve comparative effectiveness research methods and data infrastructure. In: Implementing Comparative Effectiveness Research: Priorities, Methods, And Impact. Washington, DC: Brookings Institution; 2009:35Y53. 54. Hochman M, McCormick D. Characteristics of published comparative effectiveness studies of medications. JAMA. 2010;303:951Y958. 55. Steinbrook R. Saying no isn’t NICEVthe travails of Britain’s National Institute for Health and Clinical Excellence. N Engl J Med. 2008;359:1977Y1981. 56. American College of Physicians [Position Paper]. Information on cost-effectiveness: an essential product of a national comparative effectiveness program. Ann Intern Med. 2008;148:956Y961. 57. Brook RH. The science of health care reform. JAMA. 2009;301:2486Y2487. 58. Wilensky GR. Health care reformVwhere do we go from here? N Engl J Med. 2010;362:e36(1Y3).

769
