STATE
nature publishing group
ART
Novel Statistical Tools for Monitoring the Safety of Marketed Drugs JS Almenoff1, EN Pattishall1, TG Gibbs2, W DuMouchel3, SJW Evans4 and N Yuen1 Robust tools for monitoring the safety of marketed therapeutic products are of paramount importance to public health. In recent years, innovative statistical approaches have been developed to screen large post-marketing safety databases for adverse events (AEs) that occur with disproportionate frequency. These methods, known variously as quantitative signal detection, disproportionality analysis, or safety data mining, facilitate the identification of new safety issues or possible harmful effects of a product. In this article, we describe the statistical concepts behind these methods, as well as their practical application to monitoring the safety of pharmaceutical products using spontaneous AE reports. We also provide examples of how these tools can be used to identify novel drug interactions and demographic risk factors for adverse drug reactions. Challenges, controversies, and frontiers for future research are discussed.
In the wake of the thalidomide tragedy in the early 1960s, many countries introduced pharmacovigilance systems for marketed products, which encompassed the systematic collection of reports of suspected adverse events (AEs) and dissemination of information relating to suspected adverse effects of drugs.1 It is generally accepted that it is not possible to identify all of the safety issues associated with a medicine during the premarketing clinical trials, because of a number of factors. Firstly, the number of human subjects or patients that will have been exposed to a drug at the time of first product registration, particularly for a new chemical entity, is typically too small to detect uncommon (incidence of 1 in 1,000) or rare (incidence of 1 in 10,000) AEs.2 Secondly, once marketed, the actual use of the medicine may not precisely mirror the clinical trials experience in terms of patient population, dosing regimen, duration of therapy, or concomitant therapies. In 1970, Dunlop observed that ‘‘No drug, which is pharmacologically effective, is without hazard. Furthermore, not all hazards can be known before a drug is marketed’’.3 It is, therefore, vital to monitor the safety of medicines, as used in routine clinical practice, throughout their marketed life. The currently established way to do this is to collect and analyze reports of suspected AEs by means of spontaneous
reporting systems. A spontaneous reporting system is a voluntary, passive surveillance system that collects reports of suspected AEs from both health-care professionals and product consumers. Despite the limitations of spontaneous reporting,4–6 most notably underreporting, uneven quality of information provided, and influences by media/publicity and litigation, many important safety ‘‘signals’’ have been identified via these systems.7 ‘‘Signals’’ are defined in CIOMS VI as: a report or reports of an event with an unknown causal relationship to treatment that is recognized as worthy of further exploration and continued surveillance.8 In some situations, spontaneous AE reports provide a high level of suspicions, which may be considered a sufficient basis for regulatory decisions.9 However, proof of causality requires additional evidence such as temporal and dose– response relationships between the drug and event, biological plausibility, and corroborating experimental evidence.10 AEs that are thought to be causally associated with the use of a drug are sometimes referred to as adverse drug reactions. Davis et al.7 provide examples from the past decade of situations where spontaneous reports were considered a sufficient basis for regulatory action in the UK. Wysowski and Swartz11 provide a list of 22 drug withdrawals from the US market during the period 1980–2005. For all but two of
1 Department of Epidemiology and Population Health, Safety Evaluation and Risk Management, Global Clinical Safety and Pharmacovigilance, GlaxoSmithKline, Research Triangle Park, North Carolina, USA; 2Department of Epidemiology and Population Health, Safety Evaluation and Risk Management, Global Clinical Safety and Pharmacovigilance, GlaxoSmithKline, Greenford, UK; 3Department of Epidemiology and Population Health, Lincoln Technologies (a subsidiary of Phase Forward Inc.), Waltham, Massachusetts, USA; 4Department of Epidemiology and Population Health, London School of Hygiene & Tropical Medicine, London, UK. Correspondence: JS Almenoff (
[email protected])
Published online 30 May 2007. doi:10.1038/sj.clpt.6100258 CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 82 NUMBER 2 | AUGUST 2007
157
STATE
ART
the 22 withdrawals, the analysis of spontaneous reports was a critical informational component contributing to the withdrawal decision. The two exceptions, encainide and flosequinan, were situations where the suspected AE was the same as the disease being treated, such that only a well-conducted randomized trial could convincingly establish a causal relationship between the drug and event. Troglitazone, terfenadine, and temafloxacin are notable examples where spontaneous reports provided crucial evidence to support drug withdrawal.12,13 Traditional work in pharmacovigilance has focused on the medical evaluation of individual case reports.14 Spontaneous AE reporting has steadily risen since its inception in the 1960s such that many organizations (pharmaceutical companies, regulators, and the World Health Organization (WHO)), receive tens of thousands of spontaneous reports annually.7,15 Therefore, methods involving traditional ‘‘case-by-case’’ assessment of reports have become challenged by the need to manually observe and analyze the vast quantities of data contained in AE databases. ‘‘Data mining’’ or disproportionality analysis (DPA) is a technique for extracting meaningful, organized information from large complex databases. These tools identify combinations of drugs and AEs that are reported more frequently than expected by using information on all drugs and all events in the database to estimate expected reporting frequencies. This approach facilitates the identification of patterns in the data that might be difficult to see with individual case review, and more importantly, avoids preconceptions about the data. The value of DPA as a tool for post-marketing signal detection has been described.16 Evidence is emerging that the use of data mining tools for proactive detection and prioritization of drug safety signals allows drug safety evaluators to focus attention on issues that are important to public health.16–18 The potential limitations of data mining include those inherent to spontaneous reporting databases. However, early warning about safety issues is paramount; therefore, early alerts from spontaneous data may generate important safety hypotheses that can be investigated and corroborated with other knowledge. For this reason, data mining is being integrated into routine pharmacovigilance practice to support signal detection and decision-making at a number of companies, regulatory agencies, and pharmacovigilance centers.19–25 DATABASES USED FOR SAFETY DATA MINING
There are a number of databases containing post-marketing AE reports that are utilized for drug safety data mining, including databases maintained by pharmaceutical manufacturers, regulatory agencies, and public health monitoring centers. Vaccine safety databases have also been used.26 All of these databases acquire AE reports through voluntary, passive surveillance known as the spontaneous reporting system. Each database has its own data entry rules, structure, and reporting criteria. All are subject to limitations: underreporting of events, no clear link to drug exposure data, lack 158
of control group, potential biases in reporting, missing and incomplete data, and unknown causality.15,18 Despite these complexities, spontaneous AE databases have been used to successfully identify numerous safety issues in marketed products.16,18 Safety databases maintained by pharmaceutical companies are used by those companies to monitor the safety of their own products and to facilitate the compulsory reporting of AEs to global regulatory authorities. The size and diversity of a company’s database is generally related to the size and diversity of the company’s product portfolio. The main disadvantages of company databases are that they are proprietary, and their size and diversity may be too small for meaningful analysis. As data mining techniques utilize comparisons of reporting rates to a background of reporting rates across many products, the database must contain enough cases and enough drugs to generate the background for comparison. If a database is dominated by certain therapeutic areas or certain AEs, these dominant events may mask this same event for another drug.18,27 Although there are no guidelines for size or diversity of databases, it is often useful to compare results of analyses from smaller databases with those from larger, more diverse public databases. If the results are generally comparable, then it may be reasonable to use a company database for routine signal detection.28,29 There are numerous single-country databases, typically maintained by individual country regulatory agencies. One large single-country database that is often utilized for data mining is the public-release version of the United States Food and Drug Administration (FDA) Adverse Event Reporting System (AERS) for pharmaceutical products. Although still limited by a voluntary reporting scheme and associated biases, the advantages are that it is available to the public and has a broad diversity of products; it is large (43 million reports collected since 1968), and in addition to spontaneous domestic reports, contains reports of serious, unlabeled AEs from non-US sources and serious, unlabeled, and attributable post-marketing clinical trial reports.15 The FDA has done pioneering work in quantitative signal detection, and has used these methods to support decision-making for several marketed products.16,30,31 The major limitations of AERS are the time-lag in data release (generally 3–6 months) and the absence of case narratives in the public-release version. Case reports with narratives can be requested; however, delivery takes several months. Some safety databases combine data from multiple countries. For example, the WHO safety database is large, with 43.7 million AE reports spanning 430 years and is global in scope, utilizing data from 82 countries participating in the WHO International Drug Monitoring Program.32 The countries contributing the largest amount of data include the US, UK, Germany, Australia, and Canada. Strengths of the database include the potential ability to identify countryspecific differences, to evaluate drugs by generic or trade name, the presence of a case-quality grading system, and public availability. The database is dominated by data from VOLUME 82 NUMBER 2 | AUGUST 2007 | www.nature.com/cpt
STATE
the US (almost 60% of reports), and case narratives are not included in the public-release version. The WHO database has many of the same advantages and disadvantages as AERS; it is an excellent resource for understanding the safety profile of drugs that have never been marketed in the US. One caveat to consider is that the regulatory reporting practices of countries that submit data to WHO may differ, which may create some inconsistencies in the characteristics of the data across these countries. The European Agency for the Evaluation of Medicinal Products (EMEA) established the EudraVigilance database in December 2001 for reporting and evaluating AEs for drugs in development and on the market in the European Economic Area. Reports are received from regulatory agencies of the EU member states and from pharmaceutical companies.33 As of 24 April 2007, the EudraVigilance database currently contains approximately 630,000 individual cases (Sabine Brosch, EMEA, personal communication). The EMEA is beginning to develop systems for signal detection using the EudraVigilance database.34 However, the usefulness of this data set to the public is limited because the EMEA restricts access to the data (i.e., companies can only view data that they have submitted). STATISTICAL APPROACHES
In contrast to clinical trials, where the incidence of AEs is easily computed (because numbers of patients exposed to drugs are known, and AE reporting is compulsory), counts of spontaneously reported drug–event combinations cannot be assessed as ‘‘large’’ or ‘‘small’’ without some comparison value to serve as background for each combination. Although exposure denominators can, in principle, be estimated from prescription data, these estimates have significant limitations because they are based on the amount of product sold and do not capture the dose or duration of therapy. Moreover, they apply to the entire treated population rather than to the limited population reporting AEs.35 Moreover, interpretation of ratios of AE counts to prescription counts depends on knowing AE reporting rates, which are notoriously hard to estimate. An alternate strategy is to gauge drug–event associations within the spontaneous reports database without reference to external data. The goal is to measure whether and by what factor particular AEs tend to occur disproportionally in the same report with particular drugs. The simplest approach to quantitative signal detection using DPA involves tabulating, for each combination of drug ingredient (indexed by i) and AE code (indexed by j), a twoby-two table of counts as shown in Table 1. Table 1 Two-by-two tables are the framework for disproportionality analysis Counts of reports With event j Without event j Total
With drug i
Without drug i
Total
nij=a
b
a+b
c
d
c+d
a+c
b+d
a+b+c+d
CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 82 NUMBER 2 | AUGUST 2007
ART
For example, if there are 3,000 drugs and 5,000 AE codes in the database, there are 15,000,000 possible two-by-two tables to tabulate and process. In practice, only drug–event combinations that are actually present in the database will be tabulated, but this may still involve hundreds of thousands of tables. For each such table, the corresponding drug–event combination is called disproportionally present in the database if nij ¼ a is larger than a baseline or expected value computed as if the drug and event occur independently. The overall proportion of reports having event j is (a þ b)/(a þ b þ c þ d), and so, if there are (a þ c) reports involving drug j and there is no association of report i with event j, then the expected number of reports of drug i with event j would be the product e ¼ (a þ c)(a þ b)/(a þ b þ c þ d). This leads to the definition of a ‘‘relative reporting ratio’’ as RRR ¼ a=e ¼ ½a=ða þ bÞ=½ða þ cÞ=ða þ b þ c þ dÞ A closely related quantity that is commonly used in pharmacovigilance is the ‘‘proportional reporting ratio’’ PRR ¼ ½a=ða þ cÞ=½b=ðb þ dÞ which is intended to be analogous to relative risk in a cohort study, namely the ratio of incidence of the AE in the ‘‘exposed reports’’ to incidence in the ‘‘unexposed reports.’’ Finally, a third measure of DPA is the ‘‘reporting odds ratio’’ ROR ¼ ad=bc As each individual drug and event occurs in a small proportion of the reports, it is usually true that a 5(b, c)5d so that relative reporting ratio (RRR), proportional reporting ratio (PRR), and reporting odds ratio (ROR) are nearly identical and their values interpreted similarly. A value of 1 for any of these measures means that there is no association between the reporting of drug i and event j in the database.36 A value of 5 means that there are five times as many reports of this drug–event combination in the database than would be expected if drugs and events were reported independently of each other. The idea of computing n/e ratios for all or some drug–event combinations is simple and was described several decades ago.37,38 However, recent computing and database advances have facilitated the routine use of DPA.19,22,36 Traditionally, biostatisticians have been uncomfortable performing formal analyses on tabulations of spontaneous reports for several reasons: data quality issues, unknown reporting mechanism can lead to reporting biases, frequent non-causal associations with indications, and comorbidities. For this reason, large values of n/e cannot always be assumed to represent robust signals; they require further evaluation. Drug–event pairs involving small values of n and/or e require statistical sophistication because the ratios n/e have very large sampling variation. For example, a count of n ¼ 3 is likely to change to any value from 0 through 10 if a database of the same size was to be resampled from the same 159
STATE
ART
population. The value of e is statistically more stable, but the variability of n propagates to variability in n/e, and more so when e is small. The usual application of PRR is in conjunction with computing the two-by-two table w2 values for association; with this approach, large ratios with nonsignificant w2, which are typically seen with small n and even smaller e, are ignored.19 Alternative approaches to the frequentist methods described above are Bayesian methods. Bayesian approaches account for the uncertainty in the disproportionality ratio when the counts are small is to use Bayesian statistical methods that produce ‘‘shrinkage’’ values of n/e, so that the raw value of RRR is moved toward the null hypothesis value of 1 by an amount that depends on the variability of the disproportionality statistic.39,40 The raw values of RRR are transformed using the Bayesian theory to a basic estimate called EBGM (Empirical Bayes Geometric Mean of the relative reporting ratio) and a 90% confidence interval (EB05, EB95). Bate et al.22 at WHO Uppsala Monitoring Center have described a similar procedure, which calculates a statistic known as the Information Component. One advantage of using the Bayesian estimates over the combination of PRR and w2 statistic is the convenience of being able to sort many different drug–event combinations in a single dimension for rankings and comparisons. A single number like EBGM incorporates information both about the value of n/e and its variability. Another important statistical issue is the problem of multiple comparisons. When scanning the tens of thousands of computed n/e values in a large database, it is natural to focus on the largest ratios. This is an example of post hoc selection, which is likely to select ratios biased toward large values, based on counts that happen to be large because of sampling variation. The Bayesian shrinkage methods mentioned above are designed to correct for this bias. For example, how should one estimate and compare the disproportionality of one drug–event combination having (n ¼ 3, e ¼ 0.03, n/e ¼ 100) with that of another combination having (n ¼ 50, e ¼ 5, n/e ¼ 10). Both ratios are likely to be statistically larger than their ‘‘true values;’’ the computation of how much to shrink their estimates depends on fitting a Bayesian model to the entire ensemble of (n, e) pairs in the database. Depending on the results of the fit, it might be that the first estimate shrinks from 100 down to 5, whereas the more reliable second estimate only shrinks from 10 to 9. More recently, logistic regression has been used to compute the strength of a mathematical association between reports of an event and a drug after adjusting for the effects of other potentially contributing or confounding factors such as other drugs, age group, or gender.41 These methods are computationally more intensive than the aforementioned approaches; for this reason, they have not yet been implemented for routine screening of safety databases. The data mining literature also describes Bayesian versions of logistic regression and related methods, which are useful when very many regression coefficients are being computed 160
and when issues of multiple comparisons and the biasvariance tradeoff need to be addressed.42,43 DATA MINING IN ROUTINE PHARMACOVIGILANCE ACTIVITIES
Quantitative signal detection using DPA can quickly and efficiently screen large data sets for AEs that are reported at greater than expected frequency. Signals of disproportionate reporting derived from these analyses are viewed as hypotheses, which are further investigated (through review of individual cases and other relevant information such as literature or prior clinical trials) to determine if they represent credible safety issues.20 DPA should not be used in isolation but rather integrated into comprehensive systems, which include traditional, case-based causality assessments.18,44,45 Data mining tools are generally perceived to be valuable in helping to prioritize information in large noisy databases, and their use in pharmacovigilance has been endorsed by the Institute of Medicine.46 Published retrospective examples demonstrate their utility in detecting safety issues,14 although no prospective studies have compared their effectiveness vis-a-vis traditional case-driven methods. The challenge of doing such studies is lack of a gold standard for assessing the validity and magnitude of suspected safety issues.18,47 Prioritization of signals is particularly important when reviewing large volumes of data for a large number of drugs. There are several approaches to help focus on areas of greatest importance. Impact analysis, which combines the strength of evidence and the potential public health impact, has been used to prioritize signals for further evaluation.48 Szarfman et al.49 suggest using a lower signal alert threshold for medically severe events. Triage logic has also been used to focus on signals of greatest importance.50 It is generally advised that DPA methods be used in parallel with triage or alerting systems that do highlight single reports of medically serious events that are likely to be drug-induced (e.g., Stevens-Johnson syndrome, QT prolongation, extrapyramidal reactions, etc.), because disproportionality methods require more than a single report to flag such issues.18,29 All of these approaches require medical and scientific interpretation of the data by knowledgeable, well-qualified persons, who have been trained on the best practices for use of these tools. Systems need to be carefully implemented, with efficient processes that have appropriate governance. User feedback and metrics should be obtained periodically, and this information should be used to drive the iterative enhancement of both process and system improvements. Data mining can be integrated with traditional methods for signal detection; the most efficient approach is to design systems to take advantage of the electronic nature of AE data. One such system, Online Signal Management, does this by coupling a set of statistical, graphical, case ‘‘drill-down,’’ and work-flow management tools to yield an integrated, highproductivity environment for detection, prioritization, evaluation, and tracking of safety signals from aggregate data.25 VOLUME 82 NUMBER 2 | AUGUST 2007 | www.nature.com/cpt
STATE
DETECTION OF POTENTIAL DRUG INTERACTIONS AND DEMOGRAPHIC FACTORS ASSOCIATED WITH ADVERSE EVENTS
Clinical trials are often conducted under restricted conditions with respect to patient population (e.g., age limits, exclusion of patients with certain medical conditions) and concurrent therapies. Consequently, at the time of first marketing of a drug, information regarding safety of the drug in certain patient populations may be limited. Post-marketing safety databases can provide safety information from broad populations of patients with respect to age, gender, medical history, and concurrent medications. With traditional, casebased approaches to post-marketing pharmacovigilance, it is difficult to assess the importance of these demographic and treatment variables because there are no background rates against which individual AEs can be assessed. Using DPA, it is possible to partition (subset) an entire database into relevant demographic or treatment segments in order to analyze the impact of the variable of potential interest (i.e., age, gender, concurrent medication) on reporting frequency. DPA can be applied in comparative data subsets as a tool for evaluating patient demographic and also treatmentrelated factors associated with AEs. Because post-marketing data are less robust than clinical trials data, these types of analyses are most appropriate for developing hypotheses about risk factors associated with rare AEs, which were either not observed in, or were too infrequent to be assessed with controlled clinical trial data.51 For all of the examples shown below, the empirical Bayes algorithm, multi-item gamma Poisson shrinker,39,40 was applied to spontaneous non-vaccine reports in the GlaxoSmithKline (GSK) company database using WebVDME version 3.0, Lincoln Technologies, St Waltham, MA.52 Age as an associated factor for an AE
Elderly patients are generally prone to AEs,53 although some drugs are more likely than others to cause adverse drug reactions. DPA is a helpful tool to identify which reactions occur most frequently in elderly patients and also which drugs are associated with these reactions. In the example shown in Figure 1, age was analyzed as a possible associated factor for neurological AEs with an anti-viral agent. All 25
reports in the GSK database were subset into the following categories based on age: 0–16, 17–64, X65 years, and Unknown/Missing. The results were filtered for associations between the anti-viral drug and the following MedDRA preferred terms describing the neurologic events of interest: coma, dysarthria, sedation, and speech disorder. For each event, the relative reporting rates (EBGM values) among age groups were compared and were considered different if the two-sided 90% confidence intervals (EB05, EB95) around EBGM did not overlap. These neurological events, which were reversible in this patient population, were reported more frequently for this anti-viral drug in older patients than in younger patients. A qualitative, case-based review of these events before the availability of WebVDME had suggested that the elderly were more likely to report such events, but the relative frequencies could not be estimated because of lack of drug exposure data for discrete age groups. Therefore, this analytical approach helped to provide a quantitative approach to understanding whether age was associated with the occurrence of these effects. Dose as an associated factor for a rare AE
Figure 2 shows an example of the use of DPA to evaluate whether dose is an associated factor for three rare AEs that are associated with use of a corticosteroid. For this analysis, AE reports were grouped into one of three subsets, based on the dose formulation reported in each case: low, intermediate, or high (reports where dose information was not provided were excluded from the analysis). The strengths of associations between the dosage groups and the MedDRA preferred terms describing the AEs of interest were queried. The relative reporting rates of these events were lower for the two lower dose groups than for the high-dose group. These data suggest that the risk for these rare events may be less at the low or intermediate dose formulations. Drug interactions
It has been suggested by surveys that many of the adverse reactions resulting in hospital admission are caused by drug interactions, which may be avoidable in many instances.54 Many drug interactions are not recognized until drugs are marketed. Additionally, pharmacokinetic studies cannot
N =26 Age (years) 17–64
EBGM
20
N =57
65 and above
15 10 N=52
5
ART
N=23
N =51 N =47
N=41
N =39
0 Coma
Dysarthria
Sedation
Speech disorder
Figure 1 The relative reporting rates (EBGM with 90% confidence interval) for neurologic events are shown for the elderly and young adults. Total numbers of reports (N) for each event appear above the histogram bars. CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 82 NUMBER 2 | AUGUST 2007
161
STATE
ART
60 N=31
N =57
80
High dose Medium dose Low dose
50
N =41 N=34
60
30
Allopurinol+azathioprine Azathioprine
N =5
EBGM
EBGM
40
Allopurinol
40
20
10
N=2 N =20
N =12
20
N=2 N =23
N=224 N =130
0
Adverse event 1
Adverse event 2
Adverse event 3
Figure 2 The relative reporting rates (EBGM with 90% confidence interval) of three corticosteroid-associated events are shown for three dosage levels of the drug. Total numbers of reports (N) for each event appear above the histogram bars.
always predict whether pharmacokinetic findings are clinically relevant. The availability of real-world data from spontaneous AE databases provides a rich opportunity to interrogate the data for novel drug interactions, and to understand the clinical impact of pharmacokinetic studies. For the exploration of possible drug interactions, reports containing a drug of interest were grouped into subsets on the basis of whether the report also contained a second drug that was hypothesized to interact with the drug of interest. For each event, the relative reporting rate (EBGM) among reports containing both the drugs was compared to the relative reporting rates of the event based on containing just one or other of the two drugs. Reporting rates were considered different if the two-sided 90% confidence intervals (EB05, EB95) around EBGM did not overlap. We have previously described this strategy using well-known drug interactions.55,56 An example of using this approach is shown in Figure 3. The relative reporting rate of the bone marrow failure is higher for reports containing both allopurinol and azathioprine than for those containing either drug. One possible explanation for this observation may be that the event is more likely to be reported as a result of an interaction between allopurinol and azathioprine. This event is a consequence of a known interaction, in which allopurinol indirectly inhibits the metabolism of azathioprine.57 In addition to screening for possible drug interactions, this approach has been valuable for anticipating the safety profile of planned fixed-dose combination drug products. Despite the incremental value of this approach, it is limited to interactions between two drugs. Many patients, particularly the elderly, take more than two medications. Searching for, and analyzing, the effects of polypharmacy to find higher order interactions is a challenge that can be approached with regression-based strategies.58 162
0 Bone marrow failure
Figure 3 The relative reporting rates (EBGM with 90% confidence interval) for bone marrow failure among reports containing allopurinol and azathioprine are shown beside that of reports containing one of the drugs. Total numbers of reports (N) appear above the histogram bars.
Application of data subsetting in routine pharmacovigilance
The use of data subsetting in DPA provides a useful strategy for exploring patient- and treatment-related factors associated with AEs. Other uses of this technique may include exploration of associated factors such as indication for drug and gender. These kinds of analyses have been implemented for signal detection at GSK, through algorithms that routinely screen the entire GSK safety database. AEs identified by the use of these quantitative approaches are assessed by product specialists to determine whether the event is a signal that warrants further evaluation. Such determination is made on the basis of product domain knowledge, pharmacologic plausibility, and the careful interpretation of DPA results. CHALLENGES
Challenges in the use of quantitative signal detection can be divided into the following three main areas: (1) data, (2) methods, and (3) interpretation. Data
The reporting of suspected AEs requires that a reporter notices the effect, links it to a drug, and then reports it. All stages have to be completed and biases can occur according to drug, type of medical event, and many external factors such as media influence (publicity). With spontaneous data, not all events reported are necessarily due to the suspected drug, but could instead represent other comorbid conditions, or even diagnoses unrelated to either drug or disease, that occur just by chance with a temporal association to use of a drug. Reports often have missing data that create the potential to introduce biases, which may affect the results of DPA and can impact the interpretation of individual case causality assessments.59 VOLUME 82 NUMBER 2 | AUGUST 2007 | www.nature.com/cpt
STATE
The AEs used for quantitative signal detection are coded in structured data sets using highly granular hierarchical medical dictionaries; the most commonly used system is the Medical Dictionary for Regulatory Activities (MedDRA), the international medical terminology developed under the auspices of the International Conference on Harmonization of Technical Requirements for Registration of Pharmaceuticals for Human Use (ICH).60 Most signal detection activities, including DPAs, are conducted at a relatively detailed level in the hierarchy, as this is where most meaningful diagnoses can be identified. At the ‘‘preferred term’’ level in the hierarchy, this dictionary is highly granular (e.g., seizure, grand mal seizure, convulsion, and partial seizure are all independent preferred terms that describe a similar pathophysiological process), and thus there is a risk that a signal can get ‘‘diluted’’ across multiple similar AE terms. For this reason, analyses should include inspection of medically similar terms, and robust signals will typically be apparent in multiple terms.61,62 The MedDRA Standardized Medical Queries, which consist of medically similar terms within MedDRA, might serve as a tool for enhancing sensitivity.63 Finally, novel work by Berry and Berry64 in the
ART
analysis of clinical trials, which uses Bayesian methods to ‘‘borrow strength’’ from related terms, may be a useful approach to enhance the sensitivity of the analysis of postmarketing data. A simple approach to ‘‘borrowing strength’’ is to create graphical displays to represent all EBGM values for a drug in a single view, and to organize these displays so that medically related AEs are depicted in close proximity to each other. In Figure 4, each small rectangle represents an EBGM score for a drug–event pair, and the shading of each rectangle represents the magnitude of the score. The display is presented as a heat map, with higher EBGM scores shown in red. The MedDRA AE terms are arranged with medically similar terms adjacent to each other, thus providing a visual approach to ‘‘borrowing strength.’’ If a number of AEs with high EBGM values are localized to adjacent areas on the display, showing multiple similar AEs with high EBGM scores, there is less uncertainty about whether a particular AE with a high value is reliable. These areas of borrowed strength appear in the figure as ‘‘hot spots’’ in Figure 4, the two most prominent of which are seen in the cardiac (abbreviated as ‘‘card’’) and nervous (abbreviated as ‘‘nerv’’) body systems.
Figure 4 Heat-map for displaying values of the disproportionality score EBGM for all MedDRA events in association with a particular drug. Each large rectangle corresponds to a particular System Organ Class in the MedDRA hierarchy. Smaller rectangles correspond to finer-grained events in the hierarchy, with the shade of red denoting the estimated disproportionality ratio (truncated at EBGM44). (This image is an example from the Online Signal Management system, used with permission from Lincoln Technologies Inc.).25 CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 82 NUMBER 2 | AUGUST 2007
163
STATE
ART
An additional challenge with post-marketing safety databases is that there are data consistency issues around drug naming and duplicate reporting that can compromise the results if the data are not ‘‘cleaned’’ before analysis. Drug names entered into the system often have slight inconsistencies in spelling. In our experience, the public-release version of AERS has over 300,000 distinct ‘‘verbatim drug names,’’ which can be reduced to about 4,000 generic drug ingredients with both expert input and computer matching algorithms. Another data quality issue is the detection, elimination, or condensing of duplicate reports arising from situations where the same AE is reported via different channels (e.g., the same report may be submitted to regulatory agencies by multiple manufacturers if a patient was taking multiple drugs). Algorithms using ‘‘recordlinkage’’ strategies have been applied to remove duplicate reports from the AERS public release data; this work suggests that public-release versions of AERS (as of the 3rd quarter of 2006) contain approximately 100,000 duplicate reports (David Fram, Lincoln Technologies, personal communication). Statistical record linkage approaches have been used for duplicate elimination in the WHO database.65 Methods
As described above, there are two methodological approaches to DPA, one involving Bayesian methods, whereas the other relying on frequentist (non-Bayesian) methods (e.g., proportionality reporting ratios, reporting odds ratios, Poisson). There are differing views within the pharmacovigilance community as to which approach is most useful. As the number of reports of an AE with a particular drug increases, the methods tend to give similar results. With small numbers of reports, the frequentist methods are more sensitive to sampling variation and thus more prone to extreme results; the consequence of this is that frequentist methods signal more false-positive associations while, at the same time, having fewer false-negative associations. For a given amount of data, it is clear from information theory that there will be a tradeoff between false-positive and false-negative findings. Without extra information, you cannot reduce both simultaneously. Simulation studies by Roux et al.66 have verified that this is the case. Bayesian methods provide additional value by adjusting for the multiplicity (‘‘multiple comparisons’’), associated with simultaneous analysis of a large number of drug–event combinations, which occurs with the multiple comparisons of all drug–AE pairs in the database, thereby helping to minimize false-positive alerts. The likelihood method, using a sequential probability ratio test, provides an intermediate approach between Bayesian and non-Bayesian methods.67 The earliest simple methodologies calculated disproportionality parameters for an entire database,19,37,38 and some experts currently prefer to use this approach.22 The methods described by DuMouchel, and used at FDA and elsewhere,16,29,40 stratify the data by age, sex, and calendar year.68 Stratification minimizes the detection of apparent drug–event associations 164
that may be due to independent relationships between a drug and a stratum variable and an event and the same stratum variable. (An example is the reporting association between a drug commonly used in the elderly and an event commonly reported in the elderly.) In this respect, stratification provides additional information that may reduce false positive and false negatives.18,29,69 There are other potential variables that could be taken into account in stratification procedures, although the impact of these variables has not been formally studied. These include the country of origin for a report, which adjust for region-specific in use and reporting, as well as whether the reporters were consumers or health-care professionals. Hauben and Reich70 have published examples of important safety signals that were identified earlier with traditional case-oriented methods (individual case literature reports) than with quantitative methods based on DPA of spontaneous reporting databases. However, these studies do not systematically track the validity of other safety signals highlighted in all literature reports for the drugs studied, nor do they examine the potential for important signals that may have been missed by the traditional methods. It is our belief that traditional and quantitative pharmacovigilance methods provide different and complementary types of pharmacovigilance alerts. Literature and individual case reports typically highlight a medical event that has occurred once or a few times, in temporal association with use of a drug. In contrast, DPA identifies medical events that are being reported on an aggregate basis, with greater relative frequency for a drug than would be expected based on overall reporting for all drugs and events. For these reasons, the parallel use of both approaches is both an efficient and effective pharmacovigilance strategy. Interpretation and caveats in the use of DPA
There are many caveats to be considered in the interpretation of DPA. These methods provide information about the relative reporting of AEs in spontaneous data; they do not provide estimates of the incidence of AEs or information about whether a statistical association between a drug and AE is causal. Signals detected through DPA indicate that there is a statistical association between a drug and an event. A causal relationship between the drug and the event is only one of the possible explanations for the association. High relative reporting ratios may be seen when events are related to underlying medical conditions or co-prescribed drugs, or when AE reporting has been stimulated in some way (e.g., publicity, Dear Doctor letters, litigation). For these reasons, high relative reporting ratios are interpreted as hypotheses regarding potential causal associations between drugs and events. These hypotheses are investigated by in-depth case review, literature review, and, in some cases, with additional studies. Factors that are helpful in evaluating the strength of association between a drug and a reported AE have been reviewed.44 Conversely, low relative reporting ratios do not guarantee the absence of a causal relationship between a drug VOLUME 82 NUMBER 2 | AUGUST 2007 | www.nature.com/cpt
STATE
and an event, particularly if there is a limited amount of data reported for the drug, as might be the case in the early postmarketing period. There is a controversy as to whether comparisons of relative reporting ratios between drugs should be made. If these comparisons are made, they should be made cautiously and should be interpreted as hypothesis-generating rather than confirmatory.18,23,71 Awareness of potential reporting biases is critical when making comparisons, as these frequencies may be affected by comorbidities that differ in patient populations (e.g., the newest drug may be given to individuals who are refractory to older treatments). FUTURE DIRECTIONS FOR SIGNAL DETECTION USING HEALTH-CARE RECORDS
There has been considerable interest in creating and using electronic surveillance networks of large patient-based cohorts to monitor the safety of marketed products.46 The data sources that could be used for this purpose include administrative claims data or electronic medical records, typically containing millions of health-care transactions including prescription, diagnostic, laboratory, and patient demographic data. These records provide the opportunity for longitudinal monitoring of large, relatively stable patient cohorts, and thus provide many advantages over spontaneous AE reporting. Large health-care databases offer the potential benefit of identifying AEs with long latency, which may not be suspected to be drug-related, particularly if patients have finished treatment. In contrast to spontaneous databases, medical record- or claims-based data provide numerator and denominator information on the populations; this can be used to estimate drug utilization and condition incidence, and to evaluate temporal relationships between the use of drugs and the occurrence of AEs. These data can be used to create comparable cohorts, which help to minimize confounding factors within the data. Epidemiological methods, such as for case–control or cohort analyses, are used to evaluate specific, previously suggested, safety concerns,72,73 but their automated use in signal detection poses many challenges. Investigators at the US Veterans’ Administration have been successful in using a medical record system to detect and characterize AEs in their large cohort of patients.74 These approaches are well suited to calculating and comparing incidence rates for relatively common AEs. Additional work needs to be performed to determine the effectiveness of these tools for identifying and characterizing rare, serious events, which are efficiently reported through spontaneous reporting systems. ACKNOWLEDGMENTS MedDRAs is a registered trademark of the International Federation of Pharmaceutical Manufacturers and Associations. We appreciate the help of David Fram (Lincoln Technologies) and Patrick Ryan of GSK for critical reading and editorial advice this paper. CONFLICT OF INTEREST The authors declared no conflict of interest. CLINICAL PHARMACOLOGY & THERAPEUTICS | VOLUME 82 NUMBER 2 | AUGUST 2007
ART
& 2007 American Society for Clinical Pharmacology and Therapeutics
1. 2. 3. 4.
5. 6.
7.
8. 9. 10. 11.
12. 13.
14.
15.
16.
17.
18. 19.
20.
21. 22. 23.
24.
25.
26.
Griffin, J.P. Pharmacovigilance. J. R. Coll. Physicians London 26, 197–198 (1992). Amery, W.K. Why there is a need for pharmacovigilance. Pharmacoepidemiol. Drug Saf. 8, 61–64 (1999). Committee on Safety of Drugs Report for 1969, (HMSO, London, 1971). Rawlins, M.D., Fracchia, G.N. & Rodriguez-Farre, E. EURO-ADR: pharmacovigilance and research. A European perspective. Pharmacoepidemiol. Drug Saf. 1, 261–268 (1992). Meyboom, R.H.B. et al. Principles of signal detection in pharmacovigilance. Drug Saf. 16, 355–365 (1997a). Meyboom, R.H.B., Hekster, Y.A., Egberts, A.C.G., Gribnau, F.W.J. & Edwards, I.R. Causal or casual? The role of causality assessment in pharmacovigilance. Drug Saf. 17, 374–389 (1997b). Davis, S., King, B. & Raine, J.M. Spontaneous reporting—UK. In Pharmacovigilance, 2nd edn. (eds. Mann, R.D. & Andrews, E.B.) Ch. 15 (John Wiley & Sons Ltd, West Sussex, 2007). CIOMS Working Group VI. Management of Safety Information from Clinical Trials, (WHO, Geneva, 2005). Auriche, M. & Loupi, E. Does proof of causality ever exist in pharmacovigilance? Drug Saf. 9, 230–235 (1993). Hill, A.B. The environment and disease: association or causation? Proc. R. Soc. Med. 58, 295–300 (1965). Wysowski, D.K. & Swartz, L. Adverse drug event surveillance and drug withdrawals in the United States, 1969–2002: the importance of reporting suspected reactions. Arch. Int. Med. 165, 1363–1369 (2005). Moore, T.J., Psaty, B.M. & Furberg, C.D. Time to act on drug safety. JAMA 279, 1571–1573 (1998). Food and Drug Administration. FDA proposes to withdraw Seldane approval. FDA Talk Paper, (13 January 1997), http://www.fda.gov/ bbs/topics/ANSWERS/ANS00780.html (accessed 18 May 2007). Venulet, J. Possible strategies for early recognition of potential drug safety problems. Adverse Drug React. Acute Poisoning Rev. 1, 39–47 (1988). Kennedy, D.L., Goldman, S.A. & Lillie, R.B. Spontaneous reporting in the United States. In Pharmacoepidemiology 3rd edn. (ed. Strom, B.L.) 151–174 (John Wiley & Sons Ltd, West Sussex, 2000). Szarfman, A., Machado, S.G. & O’Neill, R.T. Use of screening algorithms and computer systems to efficiently signal higher than expected combinations of drugs and events in the US FDA’s spontaneous reports database. Drug Saf. 25, 381–392 (2002). Lindquist, M., Sta˚hl, M., Bate, A., Edwards, I.R. & Meyboom, R.H.B. A retrospective evaluation of a data mining approach to aid finding new adverse drug reaction signals in the WHO international database. Drug Saf. 23, 533–542 (2000). Almenoff, J. et al. Perspectives in the use of data mining in pharmacovigilance. Drug Saf. 28, 981–1007 (2005). Evans, S.J.W., Waller, P.C. & Davis, S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiol. Drug Saf. 10, 483–486 (2001). Van Puijenbroek, E.P., Diemont, W.L. & van Grootheest, K. Application of quantitative signal detection in the Dutch spontaneous reporting system for adverse drug reactions. Drug Saf. 26, 293–301 (2003). Purcell, P. & Barty, S. Statistical techniques for signal generation. The Australian experience. Drug Saf. 25, 415–421 (2002). Bate, A. et al. A Bayesian neural network method for adverse drug reaction signal generation. Eur. J. Clin. Pharmacol. 54, 315–321 (1998). Goldkind, L. Hepatotoxicity: Leflunomide. FDA Arthritis Advisory Committee 2003 March, http://www.fda.gov/ohrms/dockets/ac/03/ slides/3930S2_05_FDA-Goldkind.ppt#5 (accessed 23 April 2007). Szarfman, A. & Levine, J. NDA 21-144, Ketek (telithromycin) 400 mg tabletss (Sanofi-Aventis): data mining analysis of AEs and outcomes reported for telithromycin and 15 comparator drugs (AERS data), http://www.fda.gov/ohrms/dockets/ac/06/briefing/2006-4266b1-00index.htm (accessed 3 April 2007). Almenoff, J.S. et al. Online signal management: a systems-based approach that delivers new analytical capabilities and operational efficiency to the practice of pharmacovigilance. Drug Info. J. (in press). Niu, M.T., Erwin, D.E. & Braun, M.M. Datamining in the US vaccine adverse event reporting system: early detection of intussusception and other events after rota virus vaccine. Vaccine 19, 4627–4634 (2001). 165
STATE 27. 28.
29.
30.
31.
32.
33.
34.
35.
36.
37. 38.
39.
40.
41.
42.
43. 44.
45.
46.
47. 48.
49.
50.
166
ART
Gould, A.L. Practical pharmacovigilance analysis strategies. Pharmacoepidemiol. Drug Saf. 12, 559–574 (2003). Lehman, H.P. et al. An evaluation of computer-aided disproportionality analysis for post-marketing signal detection. Clin. Pharmacol. Ther. 8 (2007) (in press). Almenoff, J., LaCroix, K., Yuen, N., Fram, D. & DuMouchel, W. Comparative performance of two quantitative signal detection methods: implications for use in a pharmacovigilance department. Drug Saf. 29, 875–887 (2006). Szarfman, A., Tonning, J.M. & Murali, D.P. Pharmacovigilance in the 21st century: new systematic tools for an old problem. Pharmacotherapy 24, 1099–1104 (2004). O’Neill, R. & Szarfman, A. Some US Food and Drug Administration perspectives on data mining for pediatric safety assessment. Curr. Ther. Res. 62, 650–663 (2001). Lindquist, M. The WHO Adverse Reaction Database Uppsala Monitoring Centre, May 2004: 1–9. http://www.who-umc.org/ DynPage.aspx?id=30480&mn=2126 (accessed 29 March 2007). Wagner, H.G., Simoen, O. & Buxton, T. The European System for Monitoring Drug Safety, EudraVigilance (version 1.0). www.lexjansen.com/phuse/2005/rc/rc07.pdf. (accessed 3 April 2007). Eudravigilance Expert Working Group. Guideline on the Use of Statistical Signal Detection Methods in the Eudravigilance Data Analysis System (2006). http://www.emea.europa.eu/pdfs/human/phvwp/ 10646406en.pdf (accessed 11 April 2007). Wiholm, B., Olsson, S., Moore, N. & Waller, P. Spontaneous reporting systems outside the US. In Pharmacoepidemiology (ed. Strom, B.L.) 175–192 (John Wiley & Sons, West Sussex, 2000). van Puijenbroek, E.P. et al. A comparison of measures of disproportionality for signal detection in spontaneous reporting systems for adverse drug reactions. Pharmacoepidemiol. Drug Saf. 11, 3–10 (2002). Finney, D.J. Systematic signalling of adverse reactions to drugs. Meth. Inf. Med. 13, 1–10 (1974). Patwary, K.W. Report on Statistical Aspects of the Pilot Research Project for International Drug Monitoring, 67–76, (WHO, Geneva, 1969). DuMouchel, W. Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system (with discussion). Am. Stat. 53, 177–202 (1999). DuMouchel, W. & Pregibon, D. Empirical Bayes screening for multi-item associations. In 7th ACM SigKDD International Conference on Knowledge Discovery and Data Mining, 67–76, (ACM Press, San Francisco, 2001). Solomon, R. & DuMouchel, W. Contrast media and nephropathy: findings from systematic analysis and food and drug administration reports of adverse effects. Invest. Radiol. 41, 651–660 (2006). Genkin, A., Lewis, D.D. & Madigan, D. BBR: Bayesian Logistic Regression Software, http://www.stat.rutgers.edu/Bmadigan/BBR/ (accessed 23 April 2007). Hastie, T., Tibshirini, R. & Friedman, J. The Elements of Statistical Learning, (Springer, New York, 2001). CIOMS Working Group III. Guidelines for preparing core safety clinical information on drugs. Report of CIOMS Working Group III, (WHO, Geneva, 1994). US Dept of Health & Human Services, Food & Drug Administration. Guidance for Industry Good Pharmacovigilance Practices and Pharmacoepidemiologic Assessment. http://www.fda.gov/Cder/ guidance/6359OCC.htm (accessed 10 April 2007). Committee on the Assessment of the US Drug Safety System Board on Population Health and Public Health Practice. The Future of Drug Safety Promoting and Protecting the Health of the Public (eds. Baciu, A., Stratton, K. & Burke, S.P.) 110–114 (The National Academies Press, Washington, DC, 2007). Hauben, M. A brief primer on automated signal detection. Ann. Pharmacother. 37, 1117–1123 (2005). Waller, P., Heeley, E. & Moseley, J. Impact analysis of signals detected from spontaneous adverse drug reaction reporting data. Drug Saf. 28, 843–850 (2005). Szarfman, A., Tonning, J.M. & Doraiswamy, P.M. Pharmacovigilance in the 21st century: new systematic tools for an old problem. Pharmacotherapy 24, 1099–1104 (2004). Stahl, M., Lindquist, M., Edwards, I.R. & Brown, E.G. Introducing triage logic as a new strategy for the detection of signals in the WHO drug
51.
52.
53. 54.
55.
56.
57. 58.
59. 60.
61. 62.
63.
64.
65.
66.
67.
68. 69.
70.
71.
72. 73.
74.
monitoring database. Pharmacoepidemiol. Drug Saf. 13, 355–363 (2004). Yuen, N., Almenoff, J., DuMouchel, W., Yang, X. & LaCroix, K.K. Disproportionality analysis to explore patient and treatment related factors associated with adverse events. Pharmacoepidemiol. Drug Saf. 13, S259–S260 (2004). Fram, D.M., Almenoff, J. & DuMouchel, W. Empirical Bayesian data mining for discovering patterns in post-marketing drug safety. Proc. Knowledge Discov. Data Proc. KDD 359–368 (2003). Martin, U. & George, C. Drugs and the elderly. In Pharmacovigilance, (eds. Mann R.D. & Andrews E.B.) 77–95 (John Wiley & Sons, West Sussex, 2002). Pirmohamed, M. et al. Adverse drug reactions as cause of admission to hospital: prospective analysis of 18820 patients. BMJ 329, 15–19 (2004). Almenoff, J.S., DuMouchel, W., Kindman, L.A., Yang, X. & Fram, D. Disproportionality analysis using empirical Bayes data mining: a tool for the evaluation of drug interactions in the post-marketing setting. Pharmacoepidemiol. Drug Saf. 12, 517–521 (2003). Yang, X. & Fram, D.M. Using disproportional analysis as a tool to explore severe drug–drug interactions in AERS database [abstract]. Pharmacoepidemiol. Drug Saf. 13, S1–S334 (2004). Murrell, G.A. & Rapeport, W.G. Clinical pharmacokinetics of allopurinol. Clin. Pharmacokinet. 11, 343–353 (1986). Noren, G.N., Bate, A., Orre, R. & Edwards, I.R. Extending the methods used to screen the WHO drug safety database towards analysis of complex associations and improved accuracy for rare events. Stat. Med. 25, 3740–3757 (2006). Lindquist, M. Data quality management in pharmacovigilance. Drug Saf. 27, 857–870 (2004). Maintenance and Support Services Organization (MSSO). Medical Dictionary for Regulatory Activities (MedDRA), http:// www.meddramsso.com. (accessed 12 April 2007). Brown, E.G. Using MedDRA. Implications for risk management. Drug Saf. 27, 591–602 (2004). Bousquet, C. et al. Appraisal of the MedDRA conceptual structure for describing and grouping adverse drug reactions. Drug Saf. 28, 19–34 (2005). Maintenance and Support Services Organization (MSSO). Standardized MedDRA Queries, http://www.meddramsso.com. (accessed 12 April 2007). Berry, S.M. & Berry, D.A. Accounting for multiplicities in assessing drug safety: a three-level hierarchical mixture model. Biometrics 60, 418–426 (2004). Noren, G.N., Orre, R., Bate, A. & Edwards, I.R. Duplicate detection in adverse drug reaction surveillance. Data Mining Knowledge Discov. 14, 305–328 (2007). Roux, E., Thiessard, F., Fourrier, A., Begaud, B. & Tubert-Bitter, P. Evaluation of statistical association measures for the automatic signal generation in pharmacovigilance. IEEE Transac. Info. Technol. Biomed. 9, 518–527 (2005). Evans, S.J.W. Statistics: analysis and presentation of safety data. In Stephens’ Detection of New Adverse Reactions, 5th edn. (eds. Talbot, J. & Waller, P.) 301–328 (John Wiley & Sons Ltd, West Sussex, 2002). Mantel, N. & Haenszel, W. Statistical aspects of the analysis of data from retrospective studies of disease. J. Nat. Cancer Inst. 22, 719–748 (1959). Wacholder, S., Chanock, S., Garcia-Closas, M., El Ghormli, L. & Rothman, N. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J. Nat. Cancer Inst. 96, 434–442 (2004). Hauben, M. & Reich, L. Potential utility of data-mining algorithms for early detection of potentially fatal/disabling adverse drug reactions: a retrospective evaluation. J. Clin. Pharmacol. 45, 378 (2005). Bate, A. et al. Data-mining analyses of pharmacovigilance signals in relation to relevant comparison drugs. Eur. J. Clin. Pharmacol. 58, 483–490 (2002). Wood, L. & Martinez, C. The general practice research database: role in pharmacovigilance. Drug Saf. 27, 871–881 (2004). Velentgas, P., West, W., Cannuscio, C.C., Watson, D.J. & Walker, A.M. Cardiovascular risk of selective cyclooxygenase-2 inhibitors and other non-asprin non-steroidal anti-inflammatory medications. Pharmacoepidemiol. Drug Saf. 15, 641–652 (2006). Berlowitz, D.R. et al. Differential associations of beta-blockers with hemorrhagic events for chronic heart failure patients on warfarin. Pharmacoepidemiol. Drug Saf. 15, 799–807 (2006).
VOLUME 82 NUMBER 2 | AUGUST 2007 | www.nature.com/cpt
