Critical appraisal of clinical practice guidelines ... - Wiley Online Library

Research Paper International Journal of

Pharmacy Practice International Journal of Pharmacy Practice 2018, 26, pp. 49–54

Critical appraisal of clinical practice guidelines for adult cancer patients with febrile neutropenia Ziad G. Nasra

, Safae Abu Yousefb, Farah Jibrilc and Kyle J. Wilbya

a

College of Pharmacy, Qatar University, bHeart Hospital and cNational Center for Cancer Care & Research, Doha, Qatar

Keywords adult; cancer; clinical practice guidelines; evidence-based medicine; febrile neutropenia Correspondence Dr Ziad G. Nasr, College of Pharmacy, Qatar University, PO Box 2713, Doha, Qatar. E-mail: [email protected] Received July 7, 2016 Accepted January 30, 2017 doi: 10.1111/ijpp.12357

Abstract Objectives To critically appraise published international clinical practice guidelines (CPGs) for management of febrile neutropenia in adult patients with cancer and to determine opportunities for improved development and reporting. Methods A literature search identified CPGs for adult cancer patients with febrile neutropenia. Four independent assessors evaluated each included CPG according to the Appraisal of Guidelines for Research and Evaluation II instrument. Standardized scores were calculated for each guideline and polled collectively. Reliability of assessment was determined using a two-way random model intraclass correlation coefficients. Key findings Eight CPGs were independently evaluated by four assessors. Collectively, the highest scoring domain was editorial independence (83.3), followed by clarity of presentation (55.4), scope and purpose (53.4), stakeholder involvement (53.1), rigour of development (52.7) and applicability (47.8). Overall assessments ranged from 28.6 to 96.4 of 100 possible points. Three (37.5%) guidelines were recommended for use without alterations, two (25%) guidelines were recommended with alterations, and three (37.5%) guidelines were not recommended for implementation into practice. Reliability varied between guidelines with intraclass correlation coefficients ranging from 0.41 to 0.82. Conclusions Clinical practice guidelines for febrile neutropenia in adult patients with cancer were moderately rated with a 37.5% of guidelines being recommended for use in practice. Guideline developers should focus on improving CPG applicability and rigour in the development and reporting processes. Critical appraisal of guidelines should become a standard practice prior to implementation into clinical settings.

Introduction Clinical practice guidelines (CPGs) are systematically established statements that typically summarize disease management recommendations that rely on evidencebased principles that have shown to optimize patient care.[1] CPGs have been widely adopted in the healthcare setting and primarily serve as secondary literature standard resources used frequently by healthcare professionals. As such, CPGs are commonly used for decision-making purposes during direct patient care activities.[2] To achieve the best possible patient outcomes, healthcare decision-making must be evidence based. Clinical studies are subjected to rigorous peer review prior to publication and also postpublication appraisal by clinicians and experts in the field.[3] In the past, CPGs did not necessarily undergo © 2017 Royal Pharmaceutical Society

the same extent of review before publication and implementation in practice. However, prior to endorsing or creating CPG’s recommendations, professional societies should determine that the recommendations are of high quality and reflect the best sources of evidence available.[3] Systematic assessment of CPGs should become standard practice for determining the utility of a specific CPG and also for identifying gaps in the CPG development and reporting processes. As shown in previous studies, this information can be relayed back to CPG creators for improvement in subsequent updates and revisions.[4–6] Febrile neutropenia is a disease state for which CPGs have a major role and many international guidelines currently exist. Patients with febrile neutropenia are also International Journal of Pharmacy Practice 2018, 26, pp. 49--54

50

considered a vulnerable population due to immunosuppression and neutropenia itself. Therefore, adherence to treatment becomes mandatory in order to ensure patients are optimally managed. The aim of this project was to use a systematic approach to critically appraise published international CPGs for management of febrile neutropenia in adult patients with cancer. The objectives were to determine individual CPG performance across appraisal domains, identify recommendations for improvement of development and reporting these CPGs and to evaluate reliability of a systematic approach for CPG assessment.

Methods Clinical practice guidelines identification and selection A literature search for febrile neutropenia CPGs was conducted using MEDLINE (1948–February 2016), EMBASE (1980–February 2016), National Guideline Clearinghouse (February 2016) and Google Scholar databases to identify original guidelines that target the prevention and/or management of febrile neutropenia in adults with cancer. Handsearch of references of retrieved articles was also performed. Review of infectious disease and oncology society web pages was also conducted to retrieve CPGs not accounted for in the database search. Supplementary documents, appendices and editorials were sought in addition to the primary published guideline for additional relevant data. Literature search was carried out by one investigator and repeated by another for validity and eligibility. Exclusions targeted guidelines that were not written in English language, did not report pharmacotherapy recommendations for febrile neutropenia prophylaxis and/or management in adult population and those related to the paediatric population, in addition previous versions and review papers that summarized already existing full-text CPGs.

Critical appraisal tool Guidelines were assessed using The Appraisal of Guidelines, Research and Evaluation II (AGREE II) tool that has shown to advance guideline development, reporting and evaluation in health care.[7,8] This instrument necessitates the appraisal of six domains summing up a total of 23 criteria to be systematically evaluated. The domains include the following: scope and purpose, stakeholder involvement, rigour of development, clarity and presentation, applicability and editorial independence. For a stronger evaluation, AGREE II tool recommends that at least two (preferably four) assessors are required to appraise the guidelines.[9] The 23 criteria are graded on a 7-point scale with 7 being ‘strongly agree’ and 1 being ‘strongly disagree’. For every © 2017 Royal Pharmaceutical Society

Appraisal of febrile neutropenia guidelines

CPG, each assessor endorsed the guideline to be deemed as one of the following: recommended, recommended with alterations, not recommended, all based on the overall domain scores determined by the authors themselves. A detailed AGREE II manual is available to explain in details the subsections of each domain.[9]

Study procedures A total of four assessors appraised each guideline. The assessors remained constant throughout the appraisal process. All assessors had a postgraduate doctor of pharmacy degree, received training for AGREE II tool in addition to critical appraisal. They all have experience with febrile neutropenia as part of their clinical practice. Assessors currently work in both academia and healthcare settings (one assistant professor, one clinical lecturer and two clinical pharmacists). The guidelines that met the inclusion criteria were disseminated among collaborators for independent evaluation. AGREE II training modules were used to standardize training among assessors. Critical appraisal was carried out over a period of 2 months (March–April 2016). The project leader pooled all individual appraisals into a master Microsoft Excel 2010 worksheet. Once all data were obtained, the project leader assessed each guideline for potential errors in assessment. An error was defined as a 5 or more point difference in rankings between at least two assessors. If this occurred, the guideline was sent back to assessors to re-evaluate their responses and state whether they wanted to keep ranking the same, or change to a different value. A 5-point difference was chosen based on limitations identified by a previous study which had set a wider margin with a 6-point difference considered as an inconsistency among assessors.[4]

Data analysis Scaled domain scores were calculated by summing the scores of individual statements in a domain and by standardizing the total as a percentage of the maximum possible score for that domain. This was carried out using the formulas outlined in the AGREE II manual.[9] Descriptive statistics (mean, standard deviation and 95% confidence intervals) were used to summarize the domain scores. IBM SPSS Statistics v. 22 (IBM Corporation, New York, USA) was used to assess inter-rater variability for each CPG that was calculated using the single measures intraclass coefficients (ICC) [two-way random model] for all numerical rankings, including the overall evaluation. ICC was interpreted as follows: 0–0.2 indicates ‘poor’ agreement; 0.3–0.4 indicates ‘fair’ agreement; 0.5–0.6 indicates ‘moderate’ agreement; 0.7–0.8 indicates ‘strong’ agreement; and >0.8 indicates ‘almost perfect’ agreement.[10] International Journal of Pharmacy Practice 2018, 26, pp. 49--54

Ziad G. Nasr et al.

51

Results Eight clinical practice guidelines were identified through the electronic and manual literature search and met the inclusion criteria for analysis.[11–21] The CPGs were all published between 2003 and 2013 and were largely endorsed by professional societies. The CPGs included in the study are briefly described in Table 1. The four appraisers reviewed each of the eight guidelines, which resulted in 32 independent AGREE II evaluations. After pooled results, 15 inconsistencies out of a total of 184 scorings were identified in individual items of different domains and were sent back to assessors for re-evaluation. Those domains included stakeholder involvement, rigour of development, applicability and editorial independence. Upon re-evaluation and review by assessors, no 5-point differences remained. The overall assessment standardized average AGREE II score for all the guidelines in this review was 63.8 of a possible 100 points (95% CI 42.5 to 85.2), with points ranging from 28.6 to a maximum of 96.4 points [SD 25.5]. The mean highest standardized domain score for all guidelines was for

‘editorial independence’ [83.3 (95% CI 59.6 to 100.0)], while the mean lowest standardized domain score was for ‘applicability’ [47.8 (95% CI 36.0 to 59.6)]. The average standardized scores for the other domains were as such: ‘scope and purpose’ 53.4; ‘stakeholder involvement’ 53.1; ‘rigour of development’ 52.7; and ‘clarity of presentation’ 55.4, all of possible 100 points. Three guidelines (37.5%) were endorsed by appraisers as ‘recommended without revision’,[11,13,14] and two guidelines (25.0%) were considered to be ‘recommended with alterations’,[15–18,21] while three others (37.5%) were endorsed to be ‘not recommended for adoption into clinical practice’.[12,19,20] One guideline showed ‘fair’ inter-rater reliability,[13] one showed ‘moderate’ reliability,[11] and four showed ‘strong’ reliability,[12,14,19,21] while two other guidelines showed ‘almost perfect’ agreement,[15– 18,20] with intraclass correlations of 0.41–0.82 before all items assessed for each individual guideline. The standardized mean scores by CPG and individual domain, and the appraisers’ recommendations in addition to the inter-rater reliability scores are presented in Table 2. The overall pooled domain scores are shown in Figure 1.

Table 1 Summary of included clinical practice guidelines Publication date

Journal

Affiliation and/or Endorsement organization

Clinical practice guideline for the use of antimicrobial agents in neutropenic patients with cancer: 2010 update by the Infectious Diseases Society of America[11] Management of febrile neutropenia: ESMO clinical practice guidelines[12] Antimicrobial prophylaxis and outpatient management of fever and neutropenia in adults treated for malignancy: American Society of Clinical Oncology clinical practice guideline[13] Neutropenic sepsis: prevention and management of neutropenic sepsis in patients with cancer[14] Australian consensus guidelines for the management of neutropenic fever in adult patients with cancer, 2010/2011[15–18] Clinical guidelines for the management of cancer patients with neutropenia and unexplained fever[19] Antimicrobial therapy of unexplained fever in neutropenic patients[20]

2011

Clinical Infectious Diseases

Infectious Disease Society of America

2010

Annals of Oncology

European Society for Medical Oncology

2013

Journal of Clinical Oncology

American Society of Clinical Oncology

2012

National Collaborating Centre for Cancer

Developed by the National Institute for Health and Care Excellence

2011

Internal Medicine Journal

Australian Consensus Group

2005

International Journal of Antimicrobial Agents

Unknown

2003

Annals of Hematology

Prevention and treatment of cancer-related infections: clinical practice guidelines in oncology[21]

2012

Journal of the National Comprehensive Cancer Network

Infectious Diseases Working Party (AGIHO) of the German Society of Hematology and Oncology (DGHO), Study Group Interventional Therapy of Unexplained Fever, Arbeitsgemeinschaft Supportivmassnahmen in der Onkologie (ASO) of the Deutsche Krebsgesellschaft (DKG-German Cancer Society) National Comprehensive Cancer Network

Clinical practice guideline

International Journal of Pharmacy Practice 2018, 26, pp. 49--54

© 2017 Royal Pharmaceutical Society

52



0.62 0.77 0.75 0.41 0.82 0.82 0.78 0.80 Yes No Yes Yes Yes with alterations No No Yes with alterations 78.6 42.9 82.1 96.4 75.0 28.6 32.1 75.0 63.8 (95% CI 42.5 to 85.2); [SD 25.5] 75.0 100.0 100.0 87.5 100.0 100.0 16.7 87.5 83.3 (95% CI 59.6 to 100.0) 62.5 37.5 42.5 71.9 53.1 47.9 29.7 37.5 47.8 (95% CI 36.0 to 59.6) 75.0 62.5 58.3 80.6 41.7 41.7 41.7 41.7 55.4 (95% CI 41.9 to 68.9) 71.4 56.3 67.7 90.6 51.0 12.5 17.2 55.2 52.7 (95% CI 30.6 to 74.9) 41.7 40.3 66.7 91.7 63.9 37.5 25.0 58.3 53.1 (95% CI 35.4 to 70.8) 77.1 38.9 80.6 80.6 50.0 30.6 20.8 48.3 53.4 (95% CI 33.7 to 73.0) [11]

Freifeld (2011) De Naurois (2010)[12] Flowers (2013)[13] NCCCN (2012)[14] Lingaratnam (2011)[15–18] Jun (2005)[19] Link (2003)[20] Baden (2012)[21] Overall

ICC Recommended endorsement Overall assessment Editorial independence Applicability Clarity of presentation Rigour of development Stakeholder involvement Scope and purpose Guideline

Table 2 Standardized domain scores and final endorsement for each guideline

This study demonstrates a systematic approach for critical appraisal of CPGs for adult patients with febrile neutropenia. A total of eight CPGs were assessed using AGREE II. Overall quality of CPGs was moderate with high variability between individual CPGs and associated domains. The highest ranked domain across all guidelines was editorial independence, and the lowest was applicability. Five guidelines were deemed usable in practice, with or without modifications to the current form.[11,13–18,21] The other three guidelines, which represent the oldest guidelines as per the date of publication, were not recommended for endorsement.[12,19,20] Some limitations of this study must be noted. First, guidelines varied greatly in domain scoring and this might have biased the results. For example, the poor performance of the CPG by Jun et al.[19] may have falsely lowered pooled domain scores. Therefore, relative comparisons between domains and CPGs are best for interpretation. Secondly, we used the same fixed assessors of the same profession (pharmacists only) for all guidelines, so it is unknown whether reliability would change sampling from a population of assessors. Moreover, it is possible other CPGs exist outside of the ones identified; however, we believe the search to be as comprehensive as possible for the purposes of this study. Lastly, the AGREE II tool is meant to support guidelines creators and all recommendations generated from this study are suggestions to help improve clinical practice. The CPG obtaining the highest score was the one developed by the National Institute for Health Care Excellence (NICE).[15] Upon closer examination, it is evident that this CPG excelled in most domains but specifically outperformed the other guidelines for stakeholder involvement and rigour of development. While it is possible the quality of this CPG is indeed better than the other CPGs, it is also possible that the NICE group has a better reporting process. Of course, specific recommendations must be carefully examined to ensure applicability to one’s practice setting before advocating for the use of any CPG. The lowest scoring CPG was the one by Jun et al.,[19] but this is largely due to a lack of reporting of methods and other key components throughout. Interestingly, the three guidelines that were not recommended[11,19,20] had major common flaws in several domains, specifically in the scope and purpose, stakeholder involvement, and applicability domains. Thus, CPG creators should pay close attention to such domains to enhance the quality and reporting of CPGs in febrile neutropenia. The high ranking of editorial independence was an interesting finding. In previous studies, clarity of

(0.42–0.79) (0.63–0.88) (0.60–0.87) (0.21–0.63) (0.69–0.91) (0.68–0.91) (0.65–0.89) (0.67–0.90)

Discussion


Ziad G. Nasr et al.

53

Standardized mean scores

90.00% 80.00% 70.00% 60.00% 50.00% 40.00% 30.00% 20.00% 10.00% 0.00%

Figure 1

Pooled domain scores for febrile neutropenia guidelines.

presentation was reported as the highest performing domain.[4–6] It is possible that this result differed for febrile neutropenia CPGs for a number of reasons. First, almost all identified CPGs were endorsed by an international professional society or organization, which could add credibility or provide an independent funding source. Alternatively, most guidelines were published in reputable journals that likely require strict disclosures of conflicts of interests. This finding is a strength of the febrile neutropenia literature, as editorial independence is a known limitation of CPG reporting.[22] Applicability scored as the lowest domain across all guidelines, which was directly in line with previous studies. However, the average score for this domain was higher than previous studies.[4–6] This is likely due to the inclusion of tools such as algorithms for clinician use. CPG creators can still aim to increase applicability scores by evaluating resource requirements and other facilitators and barriers to implementation. Inclusion of such information will help clinicians determine the feasibility of recommendations and whether or not institutional policies should be changed to reflect CPG updates. Additionally, providing specific monitoring or auditing criteria will allow for comprehensive assessment of implementation and associated impacts on practice. Other recommendations to improve the content and reporting of guidelines include ensuring all stakeholder perspectives are obtained in formation of recommendations (including patients), ensuring recommendation strength aligns with the quality of associated evidence, and providing a clear objective for the CPG in practice. These measures will help to optimize the impact of the CPG on patient outcomes. The systematic approach outlined in this article can be adapted by others to critically appraise CPGs for use in International Journal of Pharmacy Practice 2018, 26, pp. 49--54

practice or for determining quality as a whole. In line with other studies, inter-rater reliability in assessment was deemed to be acceptable. This suggests that AGREE II can be used with assurance that clinicians and academics rate items similarly. It also suggests that the user manual provided is sufficient for training purposes. As such, clinicians should be encouraged to complete their own critical appraisal of CPGs prior to using in practice. Interestingly, inter-rater reliability was the lowest for the highest scoring guideline in terms of overall assessment yet all assessors agreed it could be used in practice in current form.

Conclusions This study critically appraised CPGs for adult patients with febrile neutropenia. Overall quality was deemed to be moderate across all guidelines with the majority of guidelines being suitable for recommending in practice. Findings were generally in line with previous studies evaluating other CPGs yet identified considerations for developers of febrile neutropenia CPGs.[4,6] Specifically, CPG developers should continue to address and report funding sources and conflict of interests. They should also work to improve applicability of guidelines through greater discussion regarding feasibility of implementation of recommendations. Finally, this study showed that the application of a systematic approach for CPG assessment is reliable across evaluators and can be applied by clinicians to assess CPG’s quality in relation to their own practice settings. Future studies should attempt to determine the impact of critical appraisal on the guideline development process, as well as utility of the approach across other CPG groups. © 2017 Royal Pharmaceutical Society

54


Declarations

Authors’ contributions

Conflict of interest

ZN was the principal investigator on this study and was responsible for overall study design. All authors critically appraised the guidelines individually. ZN was responsible for data analysis and initial drafting of the manuscript. KW aided in interpreting findings and in the writing of the manuscript. All authors reviewed the final draft of the manuscript.

The Author(s) declare(s) that they have no conflicts of interest to disclose.

Funding This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors.

References 1. Gundersen L. The effect of clinical practice guidelines on variations in care. Ann Intern Med 2000; 133: 317–318. 2. Institute of Medicine et al. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press, 2011. 3. Gilli R et al. Practice guidelines developed by specialty societies: the need for critical appraisal. Lancet 2000; 355: 103–106. 4. Wilby KJ et al. Critical appraisal of clinical practice guidelines in pediatric infectious diseases. Int J Clin Pharm 2015; 37: 799–807. 5. Wilby KJ et al. Rigorous method to assess quality and generalizability of clinical practice guidelines. Can J Hosp Pharm 2014; 67: 397–398. 6. Gorman SK et al. A critical appraisal of the quality of critical care pharmacotherapy clinical practice guidelines and their strength of recommendations. Intensive Care Med 2010; 36: 1636–1643. 7. Brouwers M et al. AGREE II: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J 2010; 182: E839–E842. 8. Vlayen J et al. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care 2005; 17: 235–242. 9. The AGREE Next Steps Consortium: Appraisal of Guidelines for Research


10.

11.

12.

13.

14.

15.

16.

and Evaluation II (AGREE II) Instrument. May 2009. Portney LG, Watkins MP. Foundations of Clinical Research Applications to Practice. USA, NJ: Prentice Hall Inc. ISBN 0-8385-2695-0; p 560-567. 2000. Freifeld AG et al. Clinical practice guideline for the use of antimicrobial agents in neutropenic patients with cancer: 2010 update by the Infectious Diseases Society of America. Clin Infect Dis 2011; 52: e56–e93. de Naurois J et al. Management of febrile neutropenia: ESMO clinical practice guidelines. Ann Oncol 2010; 21: v252–v256. Flowers CR et al. Antimicrobial prophylaxis and outpatient management of fever and neutropenia in adults treated for malignancy: American Society of Clinical Oncology clinical practice guideline. J Clin Oncol 2013; 31: 794–810. National Collaborating Centre for Cancer (UK). Neutropenic Sepsis: Prevention and Management of Neutropenic Sepsis in Cancer Patient. UK, Manchester: National Institute for Health and Clinical Excellence: Guidance, 2012. Lingaratnam S et al. Introduction to the Australian consensus guidelines for the management of neutropenic fever in adult cancer patient: Australian Consensus Guidelines 2011 Steering Committee. Intern Med J 2011; 41: 75–81. Worth LJ et al. Use of risk stratification to guide ambulatory management

17.

18.

19.

20.

21.

22.

of neutropenic fever. Intern Med J 2011; 41: 82–89. Tam CS et al. Use of empiric antimicrobial therapy in neutropenic fever: Australian Consensus Guidelines 2011 Steering Committee. Intern Med J 2011; 41: 90–101. Salvin MA et al. Use of antibacterial prophylaxis for patients with neutropenia: Australian Consensus Guidelines 2011 Steering Committee. Intern Med J 2011; 4: 102–109. Jun HX et al. Clinical guidelines for the management of cancer patients with neutropenia and unexplained fever. Int J Antimicrob Agents 2005; 26(Suppl 2): S128–S132. Link H et al. Antimicrobial therapy of unexplained fever in neutropenic patients: Guidelines of the Infectious Diseases Working Party (AGIHO) of the GAERMAN Society of Hematology and Oncology (DGHO), Study Group Interventional Therapy of Unexplained Fever, Arbeitsgemeinschaft Supportivmassnahmen in der Onkologie (ASO) of the Deutsche Krebsgesellschaft (DKG-German Cancer Society). Ann Hematol 2003; 82(Suppl 2): S105–S117. Baden LR et al. Prevention and Treatment of Cancer Related Infections: NCCN Clinical Practice Guidelines in Oncology (version 2.2011). Fort Washington, PA: National Comprehensive Cancer Network, 2011. Detsky AS. Sources of bias for authors of clinical practice guidelines. Can Med Assoc J 2006; 175: 1033.
