Interrater Agreement between Nurses for the ... - Wiley Online Library

0 downloads 0 Views 66KB Size Report
ric EDs of Canada and in many pediatric EDs around the world.7–9. Staturg (Statdev, Montréal, Quebec, Canada) is a ..... Arch Pedi- atr Adolesc Med. 2005 ...
Interrater Agreement between Nurses for the Pediatric Canadian Triage and Acuity Scale in a Tertiary Care Center Jocelyn Gravel, MD, FRCPC, MSc, Serge Gouin, MDCM, FRCPC, ABP(EM), Sergio Manzano, MD, Michael Arsenault, MD, FRCPC, Devendra Amre, PhD

Abstract Objectives: The objective was to measure the interrater agreement between nurses assigning triage levels to children visiting a pediatric emergency departments (EDs) assisted by a computerized version of the Pediatric Canadian Triage and Acuity Scale (PedCTAS). Methods: This was a prospective cohort study evaluating children triaged from Level 2 (emergent) to Level 5 (nonurgent). A convenience sample of patients triaged during 38 shifts from April to September 2007 in a tertiary care pediatric ED was evaluated. All patients were initially triaged by regular triage nurses using a computerized version of the PedCTAS. Research nurses performed a second evaluation blinded to the first evaluation using the same triage tool. These research nurses were regular ED nurses performing extra hours for research purposes exclusively. The primary outcome measure was the interrater agreement between the two nurses as measured by the linear weighted kappa score. Secondary outcomes included the proportion of patient for which nurses did not apply the triage level suggested by Staturg (override) and agreement for these overrides. Results: A total of 499 patients were recruited. The overall interrater agreement was moderate (linear weighted kappa score of 0.55 [95% confidence interval {CI} = 0.48 to 0.61] and quadratic weighted kappa score of 0.61 [95% CI = 0.42 to 0.80]). There was a discrepancy of more than one level in only 10 patients (2% of the study population). Overrides occurred in 23.2 and 21.8% for regular and research triage nurses, respectively. These overrides were equally distributed between increase and decrease in triage level. Conclusions: Nurses using Staturg, which is a computerized version of the PedCTAS, demonstrated moderate interrater agreement for assignment of triage level to children presenting to a pediatric ED. ACADEMIC EMERGENCY MEDICINE 2008; 15:1262–1267 ª 2008 by the Society for Academic Emergency Medicine Keywords: triage, emergency department, children

T

riage is the safety system designed to minimize potential morbidity due to waiting time in the emergency department (ED). The importance of a

From the Division of Emergency Medicine, Department of Pediatrics (JC, AG, SG, MA), and the Research Institute (DA), CHU Sainte-Justine, Université de Montréal, Montréal, QC, Canada. Received June 14, 2008; revision received August 1, 2008; accepted August 4, 2008. Presented in part at the Pediatric Academic Societies (PAS) Conference, Honolulu, HI, May 2008; the Society for Academic Emergency Medicine (SAEM) Conference, Washington, DC, May 2008; and Canadian Association of Emergency Physicians (CAEP) Conference, Ottawa, Ontario, Canada, June 2008. This study was financially supported by a grant from the Research Institute of CHU Sainte-Justine. Address for correspondence and reprints: Jocelyn Gravel, MD, FRCP, MSc; e-mail: [email protected].

1262

ISSN 1069-6563 PII ISSN 1069-6563583

valid and reliable triage tool has been put forward with the increase in waiting time and overcrowding in EDs in North America.1–5 While many triage tools have been developed to assess patients’ urgency, their implementation in clinical practice is dependent on their validity and reliability. In 2001, the Canadian Association of Emergency Physicians in collaboration with the National Emergency Nurses’ Affiliation and the Canadian Pediatric Society proposed a five-level triage scale derived from the adult Canadian ED Triage and Acuity Scale (CTAS)1 for pediatric patients: the Pediatric Canadian Triage and Acuity Scale (PedCTAS).6 This triage tool is used in all pediatric EDs of Canada and in many pediatric EDs around the world.7–9 Staturg (Statdev, Montréal, Quebec, Canada) is a computerized adaptation of the PedCTAS.10 Using this tool, triage nurses choose from among 39 chief complaints. Staturg then displays key information needed to

ª 2008 by the Society for Academic Emergency Medicine doi: 10.1111/j.1553-2712.2008.00268.x

ACAD EMERG MED • December 2008, Vol. 15, No. 12



www.aemj.org

assign proper triage level with regard to the chief complaint. This information is derived from the guidelines of the PedCTAS. Once the information is gathered, a triage level is suggested by the computer. The nurse finalizes the triage level by approving the suggested triage level or by overriding it. Reliability refers to the degree to which a triage level assigned by an observer will be replicated by another. Low reliability is a major threat to the safety of patients because a measurement that is not reproducible may be inaccurate. A high reliability is also important for ethical reasons because an important difference in the time to be seen by a physician may exist between two triage levels during busy times in the ED.4,11,12 Few studies have evaluated the reliability of triage tools for children.10,13–16 Interrater agreements in these studies varied, with kappa scores ranging from 0.26 to 0.55. All these studies were based on case scenarios rather than a clinical setting. The PedCTAS is the most commonly evaluated triage tool and showed kappa scores ranging from 0.39 to 0.55. The interrater agreements reported with case scenarios may not represent the reality of patients seen in a busy clinical setting like the ED. We thus proposed to measure the interrater agreement between nurses who assigned triage levels to children visiting a pediatric ED assisted by a computerized version of the PedCTAS (Staturg from Statdev). METHODS Study Design This was a prospective cohort study. This study was approved by the institutional review board of our institution. Nurses and patients provided informed consent. Study Setting and Population The study population was composed of children triaged in the ED of a tertiary care pediatric hospital with an annual census of approximately 60,000 patients-visits. Nurses have been using a computerized version of the PedCTAS (Staturg from Statdev) since November 2005.10 The study was carried out from April to September 2007. For practical reasons, patients were recruited during a convenience sample of 38 research shifts distributed from 7:00 AM to 11:00 PM during weekdays. There were two levels of study participants. First, because we evaluated their work, regular triage nurses were considered participants. To be eligible for the study, a registered nurse had to be eligible to perform triage in our setting. To do so, she had to have more than 6 months of experience in the ED, have been trained in triage, and have succeeded in a local triage evaluation. The research nurses were five regular registered nurses eligible to perform triage in our setting, hired to do extra shifts for research purposes. Four of the five research nurses were also involved as regular triage nurses in the study. The second type of study participants were patients. Every patient younger than 18 years old who presented to the ED was eligible, except for the patients who needed immediate care. This was defined as patients needing to go directly to the resuscitation room (Triage

1263

Level 1), patients with very high pain necessitating fast intravenous access, or patients with significant wheezing (Triage Level 2). Study Protocol At the beginning of each research shift, regular triage nurses were invited to provide informed consent to participate in the study and to accept recruitment of their patients. Patients were initially seen by the regular triage nurse. The research nurse approached the patient to participate in the study. After patients’ assent (with the consent of a guardian), the research nurse performed a second triage evaluation within 15 minutes of the initial triage. The research nurse repeated all the necessary measurements but, for ethical reasons, did not repeat temperature measurement or draw blood tests if it was already performed by the first nurse. The research nurse was not informed of the triage level assigned by the regular nurse before assigning a triage score. The final triage level was the one assigned by the regular triage nurse (first evaluation). In the situations where the research nurse triaged the patients as a Level 2, and only these uncommon cases, the research nurse verified the initial triage level assigned by the regular nurse at the end of both evaluations. In case of discrepancy, both nurses discussed the case to assign a proper triage level. The triage level registered in the research data sheet was not modified if the blinding was broken. The research nurse followed every patient to record baseline demographics and final disposition of the participant. Information regarding children visiting the ED during the study periods was retrieved using the computerized database of the ED. Measurements The primary outcome measure was the interrater agreement between the two nurses. Secondary outcomes included the proportion of patients for which the triage nurse did not apply the triage level suggested by Staturg (override) and agreement for these overrides. Also, the correlation between independent variables and interrater agreement was evaluated. The independent variables were patients’ age, referral status, mode of arrival, chief complaint, and nurses’ experience. They were all decided a priori. Data Analysis The interrater agreement was measured using the linear weighted kappa17 as suggested by Dong et al.18 Quadratic weighted kappa scores were also calculated as a secondary analysis. The association between many potential risk factors and agreement between the two nurses was analyzed by calculating the odds ratio of association in a 2 · 2 table. An analysis was performed to evaluate specific kappa score for each chief complaint. The 95% confidence intervals (CI) were measured for every result. All data were entered in an Excel database (Microsoft Inc., Redmond, WA) and analyzed with SPSS v16 (SPSS Inc., Chicago, IL) and STATA (StataCorp, College Station, TX) software. The sample size was calculated based on a previous study evaluating the interrater agreement among nurses using the adult CTAS in a clinical setting.18

1264

Gravel et al.

Based on this, it was estimated that the evaluation of 500 patients would be sufficient to provide a CI of ±0.05 for the weighted kappa score. RESULTS From April 30, 2007, to September 14, 2007, five research nurses performed 38 shifts of approximately 8 hours each for recruitment. Thirty-five regular triage nurses were also involved in the triage of at least one patient and consented to participate in the study. Regular triage nurses seemed more experienced, as demonstrated by the fact that 10 (28.6%) had more than 10 years of experience in the ED. No research nurse had more than 10 years of experience, but the small number of research nurses (5) limited the possibility to find a statistically significant difference between research and regular triage nurses. During the study period, 19,265 children visited the ED. Because a research nurse skipped a data sheet form, a total of 499 patients were approached, and they all agreed to participate in the study. The baseline demographic data of the participants are presented in the Table 1. The most common chief complaints were related to trauma, fever, and digestive problems. Table 1 also shows that patients included in the study were of a lower severity than the general population of children visiting the ED during the study period. This is demonstrated by the lower proportion of arrival by ambulance, the smaller proportion of patients triaged

Table 1 Baseline Demographic Data of the Study Participants and All Patients Visiting the Emergency Department during the Study Period

Gender male Median age (months) Arrival by ambulance Referral Main chief complaints Trauma Vomiting ⁄ diarrhea Skin problem Fever without respiratory problem Abdominal problem Respiratory + fever Final triage level Level 1 Level 2 Level 3 Level 4 Level 5 Final disposition Home Hospitalization Day center* Left without being seen by a physician

Participants, n = 499 (%)

All Patients, N = 19,265 (%)

283 (56.7) 60 12 (2.4) 105 (21.0)

1,542 (8.0) 2,839 (14.7)

108 54 53 48

3,992 1,682 1,473 1,252

(21.6) (10.8) (10.8) (9.6)

40 (8.0) 36 (7.2) 0 4 98 301 96 436 18 5 28

(20.1) (8.7) (7.6) (6.5)



RELIABILITY OF THE PEDCTAS

Table 2 Final Triage Levels for Both Nurses (n = 499 Patients) Final Triage Level Assigned by Research Nurses, n

Final Triage Level Assigned by Triage Nurses

2

3

4

5

Total

2 3 4 5 Total

3 3 2 0 8

0 60 42 2 104

1 30 224 27 282

0 5 33 67 105

4 98 301 96 499

Level 1–2, and lower hospitalization rate for the study population. Table 2 provides the raw data of the agreement between nurses for final triage level for all patients. The overall interrater agreement between the nurses was moderate as demonstrated by a linear weighted kappa score of 0.55 (95% CI = 0.48 to 0.61). A second analysis of the data reported a quadratic weighted kappa score of 0.66 (95% CI = 0.42 to 0.80). There was a discrepancy in triage of more than one level in 10 patients (2%). A secondary evaluation of the triage levels suggested by the computerized tool based on the gathered information for each nurse was performed. The linear weighted kappa score for this interrater agreement was of 0.51 (95% CI = 0.44 to 0.57), while the quadratic weighted kappa score was of 0.53 (95% CI = 0.31 to 0.74). A secondary outcome was to evaluate factors associated with higher or lower agreement. No association was seen between agreement in triage and potential risk factors (Table 3). Also, none of the 39 chief complaints categories was associated with a higher or lower interrater agreement in triage. The use of a computerized system permitted us to evaluate situations when the triage level suggested by the PedCTAS, based on the clinical information gathered, was overridden by the triage nurses. The percentages of these ‘‘overrides’’ were 23.2% (116 ⁄ 499) for the regular triage nurses and 21.8% (109 ⁄ 499) for the research nurses. These patients were all included in the final analysis. Overrides were equally distributed among increase and decrease in triage level assignment (Tables 4 and 5). The interrater agreement between the nurses for overrides was poor (j = 0.32; 95% CI = 0.22 to 0.42).

2,804 (14.6) 1,446 (7.5)

(0.8) (19.6) (60.3) (19.2)

297 992 5,489 9,528 2,959

(1.5) (5.2) (28.5) (49.5) (15.4)

(87.4) (3.6) (1.0) (5.6)

14,988 1,710 392 1,974

(77.8) (8.9) (2.0) (10.2)

*Patient were hospitalized to receive treatment during the day and slept at home between treatments.

Table 3 Association between Independent Variables and Agreement in Triage between the Two Nurses Independent Variables Male Age less than 3 months Arrival by ambulance Referral Override by the triage nurse Triage nurse with less < 5 years of experience CI = confidence interval; OR = odds ratio.

OR (95% CI) 1.13 0.81 0.56 1.31 0.72 1.29

(0.77, (0.36, (0.18, (0.80, (0.46, (0.79,

1.66) 1.84) 1.79) 2.14) 1.12) 2.11)

ACAD EMERG MED • December 2008, Vol. 15, No. 12



www.aemj.org

Table 4 Correlation of the Triage Level Assigned by the Regular Triage Nurse versus the Triage Level Suggested by the Computerized Version of PedCTAS Triage Nurses’ Final Triage Level,* n

Triage Level Suggested by the Tool  1 2 3 4 5 Total

2

3

4

5

Total

0 4 0 0 0 4

1 3 82 5 7 98

1 2 50 212 36 301

0 0 2 9 85 96

2 9 134 226 128 499

PedCTAS = Pediatric Canadian Triage and Acuity Scale. *All patients with a final triage level of 1 were excluded from the study. There were two patients for whom the computerized system suggested a triage level of 1 but the nurse overrode it.  Based on the information provided by the triage nurse.

Table 5 Correlation of the Triage Level Assigned by the Research Nurse versus the Triage Level Suggested by the Computerized Version of PedCTAS Triage Level Suggested by the Tool* 2 3 4 5 Total

Research Nurses’ Final Triage Level, n 2

3

4

5

Total

7 1 0 0 8

8 84 7 5 104

1 56 200 25 282

0 0 6 99 105

16 141 213 129 499

PedCTAS = Pediatric Canadian Triage and Acuity Scale. *Based on the information provided by the research nurse.

DISCUSSION We report a moderate interrater agreement between nurses assigning triage level assisted by a computerized version of the PedCTAS when applied to children not needing immediate care in a pediatric ED. This is the first evaluation of the reliability of any version (computerized or not) of PedCTAS in a clinical setting. Our results represent slightly better agreement than previous evaluations of PedCTAS using case scenarios. The first evaluation of the PedCTAS reported kappa scores varying from 0.39 for 15 physicians to 0.51 for 29 nurses evaluating 55 case scenarios describing children presenting to an ED.14 In 2007, a second evaluation was performed in the same setting comparing the interrater agreement for nurses using the PedCTAS or its computerized version (Staturg triage tool).10 It reported a higher weighted kappa score for the Staturg triage tool (0.55 for Staturg vs. 0.51 for the paper version of PedCTAS). A major limitation of previous studies is fact that they evaluated case scenarios. Use of case scenarios has two potential opposite effects on agreement: first, it ensures that all evaluators have the same information (increase agreement), while, on the other hand, triage may have been blurred by

1265

the limitation of information (decrease agreement). Worster et al.19 suggested that the evaluation of live patients was associated with a higher agreement than the evaluation of paper case scenarios. Our study showed interrater agreement similar to previous evaluations of the adult CTAS performed in a clinical setting. In 2006, Dong et al.18 reported an interrater agreement that was moderate to good, depending on the type of analysis (linear weighted j = 0.52, 95% CI = 0.46 to 0.57; and quadratic weighted j = 0.66, 95% CI = 0.60 to 0.71) for nurses evaluating patients presenting to an adult ED using a computerized version of CTAS (eTRIAGE).18 Other triage tools have been evaluated in different settings; among them, was the Soterion Rapid Triage System, a five-level computerized triage tool.20 In this study by Maningas et al.,21 a research nurse performed a second triage blinded to the first evaluation on patients younger than 13 years. The study reported an impressive weighted kappa score of 0.90 (95% CI = 0.83 to 0.96) for the 117 patients evaluated by two nurses.21 This result was never duplicated. The Emergency Severity Index (ESI) is a five-level triage system based on the expected resource use for the patients.22 In 2005, Baumann and Strout23 reported the interrater reliability for the ESI (Version 3). They reported a weighted kappa score ranging from 0.84 to 1.00 for the evaluation of written case scenarios and 0.82 (95% CI = 0.66 to 0.96) for the evaluation of 20 patients in the ED by the triage nurse and a nurse investigator.23 The Australian Triage Scale was the first nationally accepted triage tools.15,24 While its validity has been assessed for case scenarios describing children, it has never been evaluated in a clinical setting. In 2004, Maldonado and Avner16 reported the interrater reliability of different health professionals performing triage of case scenarios describing children. Their conclusion was that the level of agreement was fair for their local triage system (kappa scores ranging from 0.26 to 0.39). Although a kappa score of 0.55 suggests acceptable agreement,25 the discrepancy in triage level for almost 30% of the study population is of concern. A small difference in the triage level (for example, a patient triaged urgent by a nurse and semiurgent by another) can have a high impact on the waiting time to be seen by a physician.11,12 Also, Gravel et al.26 reported that for children triaged semiurgent (Level 4), the rate of patients who left without being seen by a doctor was as high as 21%, compared to 2.3% for patients triaged as urgent (Level 3).26 This higher rate of patients leaving the ED before seeing a physician increases the risk of morbidity and bounce back to the ED.27 Currently, the level of agreement that would be clinically significant for triage is unknown. This level should be determined by a consensus of experts from multiple centers. Potential risk factors associated with lower interrater agreement were evaluated as a secondary outcome. None of the seven potential variables were significantly associated with agreement between the two nurses. Patients younger than 3 months and occurrence of override by the triage nurse had a trend toward a lower agreement, but this was not statistically significant. The absence of an association may be related to the small

1266

sample size; the study was not powered to evaluate these associations. An important secondary objective of the study was to measure the frequency with which the triage level suggested by the computerized version of PedCTAS was overridden by the nurses. The guidelines of the PedCTAS stipulate that ‘‘Triage personnel must use their experience and instincts to ‘up triage’ priority, even if the patient does not fit exactly with the facts or definitions on the triage scale’’ but strongly discourage down triage of patients.6 Override is difficult to measure when a paper-based triage system is used, as it is only possible to retrieve the final triage level attributed by the nurse without information on the triage level suggested by the guidelines. Use of the computerized triage tool enabled us to measure override, as both triages were documented (the one recommended by the computer based on the guideline and the final triage level assigned by the nurses). In 2007, Dong et al.28 reported that ‘‘override’’ occurred from 2% to 10% in a clinical study evaluating more than 1,100 adult patients. Most of these overrides were for lower acuity triage levels. We observed substantially higher rates of override (21% to 23%) in our study population. These overrides were equally distributed among higher and lower acuity triage levels. The exact reasons for the higher rates of overrides are difficult to ascertain. It is possible that the PedCTAS may not adequately assign level of urgency for a subset of children. If, however, this were the case, it would be expected that the overrides of the two groups of nurses would overlap. We observed that there was low agreement between nurses with regard to overrides (j = 0.31). A high proportion of undertriage was related to abnormal vital signs (e.g., very high heart rate) possibly because the patient was crying. Certainly, the reasons for these discrepancies need to be further evaluated. LIMITATIONS For ethical reasons, patients needing immediate care were not included in the present study. Therefore, there are no patient triaged to Level 1, and the proportion of patients triaged Level 2 was low. This limits the generalizability of our results for this population. It has, however, low impact on our conclusion, because these patients represent a small proportion of patients in the pediatric ED.29 Also, previous studies evaluating case scenarios showed a higher agreement for patients triaged Levels 1 and 2.10 The study was performed in a single university-affiliated children hospital. This setting has a high interest in triage; many studies have been performed there and the ED uses a unique computerized triage tool (Staturg from Statdev). The reported reliability may vary in other clinical settings and will require confirmation via multicenter studies. The study participants were recruited during weekdays from 7:00 AM until 11:00 PM. There is a possibility that interrater reliability is different for patients visiting the ED at night. While this may need further evaluation, in general the proportion of children visiting the ED from 11:00 PM to 7:00 AM is low.

Gravel et al.



RELIABILITY OF THE PEDCTAS

The evaluation of patients was done sequentially, as is a common practice when evaluating the reliability of triage tools.19,30 This had two effects on agreement. First, both nurses did not gather the same information, and this could decrease agreement. Second, the research nurse already knew that the patient did not need immediate care when evaluating the patient. This likely influenced the triage level. Another approach would have been to have one nurse who asked the questions and both nurses simultaneously assign triage level. This could, however, bias results toward higher agreement. The high proportion of override may have affected the interrater agreement because it seemed associated with a lower agreement. Some may view our study to be the evaluation of the triage process in our ED instead of the evaluation of the PedCTAS. However, the agreement for the triage levels suggested by the computer was also moderate (linear weighted kappa of 0.51 and quadratic weighted kappa score of 0.53). Finally, there is no clear definition of what would be the satisfactory level of agreement for triage system. This would need to be defined by a consensus of experts in triage. CONCLUSIONS Nurses performing triage assisted by Staturg, which is a computerized version of the PedCTAS, had a moderate interrater agreement when evaluating children not needing immediate care in a pediatric ED. A larger multicenter prospective cohort study would be the next logical step to further evaluate factors associated with lower reliability. This study revealed that nurses frequently override the triage level suggested by the PedCTAS in our setting. The reasons behind these overrides need further investigation. References 1. Beveridge R. CAEP issues. The Canadian Triage and Acuity Scale: a new and critical element in health care reform. Canadian Association of Emergency Physicians. J Emerg Med. 1998; 16:507–11. 2. Derlet R, Richards J, Kravitz R. Frequent overcrowding in U.S. emergency departments. Acad Emerg Med 2001; 8:151–5. 3. Taylor TB. Threats to the health care safety net. Acad Emerg Med. 2001; 8:1080–7. 4. Wilper AP, Woolhandler S, Lasser KE, et al. Waits to see an emergency department physician: U.S. trends and predictors, 1997–2004. Health Aff (Millwood) 2008; 2:w84–95. 5. Hoot NR, Aronsky D. Systematic review of emergency department crowding: causes, effects, and solutions. Ann Emerg Med. 2008; 52:126–36. 6. Canadian Association of Emergency Physicians. Implementation of the Canadian Paediatric Emergency Triage and Acuity Scale. Can J Emerg Med. 2001; 4(Suppl):1–32. 7. Jimenez JG, Murray MJ, Beveridge R, et al. Implementation of the Canadian Emergency Department

ACAD EMERG MED • December 2008, Vol. 15, No. 12

8.

9.

10.

11.

12.

13.

14.

15.

16.

17. 18.

19.



www.aemj.org

Triage and Acuity Scale (CTAS) in the principality of andorra: can triage parameters serve as emergency department quality indicators? CJEM. 2003; 5:315–22. Goransson K, Ehrenberg A, Marklund B, Ehnfors M. Accuracy and concordance of nurses in emergency department triage. Scand J Caring Sci. 2005; 19:432–8. Rehmani R, Norain A. Trends in emergency department utilization in a hospital in the Eastern region of Saudi Arabia. Saudi Med J. 2007; 28:236–40. Gravel J, Gouin S, Bailey B, Roy M, Bergeron S, Amre D. Reliability of a computerized version of the Pediatric Canadian Triage and Acuity Scale. Acad Emerg Med. 2007; 14:864–9. Lambe S, Washington DL, Fink A, et al. Waiting times in California’s emergency departments. Ann Emerg Med. 2003; 41:35–44. Yoon P, Steiner I, Reinhardt G. Analysis of factors influencing length of stay in the emergency department. CJEM. 2003; 5:155–61. Bergeron S, Gouin S, Bailey B, Patel H. Comparison of triage assessments among pediatric registered nurses and pediatric emergency physicians. Acad Emerg Med. 2002; 9:1397–401. Bergeron S, Gouin S, Bailey B, Amre DK, Patel H. Agreement among pediatric health care professionals with the Pediatric Canadian Triage and Acuity Scale guidelines. Pediatr Emerg Care. 2004; 20:514– 8. Crellin DJ, Johnston L. Poor agreement in application of the Australasian Triage Scale to paediatric emergency department presentations. Contemp Nurse. 2003; 15:48–60. Maldonado T, Avner JR. Triage of the pediatric patient in the emergency department: are we all in agreement? Pediatrics. 2004; 114:356–60. Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960; 20:27–46. Dong SL, Bullard MJ, Meurer DP, et al. Reliability of computerized emergency triage. Acad Emerg Med. 2006; 13:269–75. Worster A, Sardo A, Eva K, Fernandes CM, Upadhye S. Triage tool inter-rater reliability: a comparison of live versus paper case scenarios. J Emerg Nurs. 2007; 33:319–23.

1267

20. Maningas PA, Hime DA, Parker DE, McMurry TA. The Soterion Rapid Triage System: evaluation of inter-rater reliability and validity. J Emerg Med. 2006; 30:461–9. 21. Maningas PA, Hime DA, Parker DE. The use of the Soterion Rapid Triage System in children presenting to the emergency department. J Emerg Med. 2006; 31:353–9. 22. Gilboy N, Tanabe P, Travers DA, Rosenau AM, Eitel DR. Emergency Severity Index version 4: Implementation Handbook. Rockville, MD: AHRQ Publication No. 05-0046-2, 2005. 23. Baumann MR, Strout TD. Evaluation of the Emergency Severity Index (version 3) triage algorithm in pediatric patients. Acad Emerg Med. 2005; 12:219– 24. 24. Fitzgerald G. Emergency Department Triage. Queensland, Australia: University of Queensland, 1989. 25. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33:159–74. 26. Gravel J, Manzano S, Arsenault M. Safety of the modification of the triage level for children 6 to 36 months old with fever using the Paediatric Canadian Triage and Acuity Scale (PaedCTAS). Can J Emerg Med. 2008; 10:32–7. 27. Mohsin M, Forero R, Ieraci S, Bauman AE, Young L, Santiano N. A population follow-up study of patients who left an emergency department without being seen by a medical officer. Emerg Med J. 2007; 24:175–9. 28. Dong SL, Bullard MJ, Meurer DP, Blitz S, Holroyd BR, Rowe BH. The effect of training on nurse agreement using an electronic triage system. CJEM. 2007; 9:260–6. 29. Gouin S, Gravel J, Amre DK, Bergeron S. Evaluation of the Paediatric Canadian Triage and Acuity Scale in a pediatric ED. Am J Emerg Med. 2005; 23:243–7. 30. Yen K, Karpas A, Pinkerton HJ, Gorelick MH. Interexaminer reliability in physical examination of pediatric patients with abdominal pain. Arch Pediatr Adolesc Med. 2005; 159:373–6.