Evaluating Operator Performance for Patient ...

Proceedings of the Human Factors and Ergonomics Society 58th Annual Meeting - 2014

2388

Evaluating Operator Performance for Patient Telemetry Monitoring Stations Using Virtual Reality Mehdi Kohani, Jacob Berman, Danny Catacora, Byeol Kim, Monifa Vaughn-Cooke University of Maryland, College Park Patient telemetry monitoring stations in hospitals have received considerable attention due to the widely reported incidents of patient deaths linked to operator error. Monitoring stations are one of the most critical environments in a healthcare setting, where telemetry technician delayed response or failure to respond may contribute to adverse patient events. The technician’s task is to monitor patient vital signs and respond to irregular patient vitals by communicating this event to the patient care team. Poorly designed monitoring stations can potentially increase technician cognitive workload and stress, and decrease performance (time to respond, accuracy of response). This paper introduces a novel approach to evaluate the impact of monitoring station design on telemetry technician performance through the use of neurophysiological measurement (heart rate, heart rate variability, blood pressure, electroencephalography, core temperature, eye tracking) in a virtual reality (VR) environment. VR provides several benefits over conventional 2D simulations and physical mock-ups by providing an immersive environment where design alternatives can be generated with greater flexibility and speed and also without interfering with daily operations in a safety-critical environment. A replicate model of a telemetry monitoring station at MedStar Washington Hospital Center was simulated in VR. Features such as monitor layout, the number of patients simultaneously monitored, and the graphical user interface design, will be modified to evaluate technician performance for each design alternative. The outcomes of this research can be used to inform telemetry room design across several hospital settings and ultimately reduce adverse patient events due to poor telemetry technician performance.

INTRODUCTION

Copyright 2014 Human Factors and Ergonomics Society. DOI 10.1177/1541931214581497

Monitoring Technologies in Safety-Critical Systems Humans and complex technology are the two major elements of a safety-critical system. Examples of such systems can be found in aviation, energy, and health care. In critical domains, the operator needs to be constantly vigilant and take the correct action instantly, thus distractions and errors can potentially lead to loss of lives. The role of technology in safety-critical systems is to reduce errors, assist in problem recognition, provide alerts, eliminate routine actions, reduce unnecessary communication in complex situations, and ultimately improve safety and system performance (Grabowski & Sanborn, 2003). However, some studies have shown that increasing the complexity of technology can increase operator cognitive workload. (Bainbridge, 1983; Grabowski & Sanborn, 2003). Increased mental workload may cause frustration, stress, and eventually human error. Thus, a new pattern of errors has been emerged in safety-critical systems, motivated by poor technology usability. In order to establish a safe and effective interaction between the human operator and technology, human factors principles need to be taken into consideration for designing critical components of these systems. In the paper we will discuss the impact of a vital component of safety-critical systems (monitoring technology) for a particularly high-risk activity in health care, patient monitoring. Our research aims to advance knowledge of the impact of patient monitoring design configurations on the performance of hospital telemetry technicians through the use of system simulation and neurophysiological data collection of human response. We will also discuss lessons learned from other safety-critical domains (nuclear, aviation) to inform the design and simulation of patient monitoring technologies.

Control rooms in safety-critical systems function as the “brain” of the system, wherein critical system elements are monitored and technicians insure proper functioning. The nature of a monitor technician’s job and the data associated with the task varies in different settings, but they are all essentially carrying out the same task, which is monitoring system status and safety. Nuclear power plants and air traffic control towers have received significant attention and prior research into proper control room design through the application of human factors principles. A common concern found in these settings is the importance of the graphical user interface and the effectiveness of the corresponding alarm system. Operators in control rooms are bombarded by an overload of information and are required to respond to multiple critical tasks immediately. Therefore, operator capacity to retain and analyze information needs to be integrated in the design of control panels (De Carvalho, Gomes, & Borges, 2011). Use of Simulation in Safety-Critical Systems Field observations and simulation has been the most common approach in studying safety-critical systems, particularly control panels. There are many limitations to field observations, due to the sensitivity of the data collection process and the potential for causing experimental distractions for the operator while performing a critical task. Simulation provide a means to replicate a real-world scenario in the safety of a laboratory setting, where human lives are not at risk. In addition, simulation provides design modification flexibility and speed that is limited when working with physical layouts

Downloaded from pro.sagepub.com by guest on August 26, 2015


and hardware that may have to be redesigned for each design iteration. Using virtual reality (VR) simulations to assess human-machine interaction in designing control rooms has gained popularity among HFE researchers, due to several benefits of this technology. The key advantage of threedimensional VR environments over simple use of twodimensional computer screens, is the sense of immersion that the user feels (Bouchard, Côté, & Richard, 2007). Therefore, human factors evaluation of the design could be performed in the early stages before implementation in the real environment. It has been shown that simulating virtual control room desks for nuclear power plants is a useful tool to assess operator response time and ultimately conformity to HFE design principles during critical tasks (Aghina et al., 2012, 2012). In a notable study, (Luquetti dos Santos, dos Santos Grecco, Abreu Mol, & Rodrigues Carvalho, 2009) a VR desk for a nuclear power plant was developed and human factors issues were evaluated for information displays and panel layout using a survey. Patient Telemetry Monitoring in Hospitals Since most control rooms in safety-critical systems share similar elements (e.g. panel layout, information displays, data on system/human status), similar HFE principles and design approaches can be used across safety-critical domains, with extension to health care. Control rooms in healthcare facilities, such as patient monitoring stations used in intensive care units (ICU), cardiac telemetry stations, or emergency departments (ED), are reserved for monitoring of patients with the most critical monitoring needs. Patient beds in these facilities are equipped with telemetry monitors (often called bedside monitors) that display patient vital signs such as heart rate, temperature, blood pressure, ECG waveform, pulse oximetry, and breathing rate (Orphanidou et al., 2009). These devices can trigger alarms if signals show irregularities. Technicians monitor these signals and react using a predefined procedure (communication, reporting, etc.) when the alerts occur. Therefore, the key challenge for ICU or ED staff is to have the ability to immediately recognize abnormalities in patient vital signs. Missing signal alerts or not responding in a timely manner may cause further complication of an existing ailment or death (McGloin, Adam, & Singer, 1999). Prompted by reports of patient death (Emergency Care Research Institute (ECRI), 2008; Pennsylvania Patient Safety Advisory, 2009), the topic of human error in monitoring stations has received considerable attention. ECRI performed a comprehensive Failure Mode and Effects Analysis on 277 reports related to alarm response during medical telemetry monitoring between June 2004 and October 2006, which involved three incidents of patient death. Equipment errors, unavailability of telemetry staff, delayed detection of alarm condition and unrecognizing alarm condition were among the failure modes (ECRI, 2008). In another notable study, 194 incidents and serious events

2389

associated with remote cardiac monitoring were reported and analyzed by Pennsylvania Patient Safety Authority from June 2004 to December 2008, which involved 12 incidents of patient death (Pennsylvania Patient Safety Advisory, 2009). Power failures, communication issues (social factors) and patient misidentification were among the reasons for errors. Telemetry Technician Performance Assessment There are a limited number of studies that focused on redesigning the monitoring system to improve technician performance. In a simulation study of an actual emergency room telemetry station (Kobayashi et al., 2013), HFE performance assessment of technicians detecting cardiac arrhythmia on telemetry monitors, revealed limited accessibility, suboptimal usability, and general neglect of the telemetry system. Problems due to alarm audibility, visibility, and system input interface were found to be the primary causes of poor technician performance. Mitigation strategies for these causes were then taken into account to redesign the monitoring station and improve detection rates. Redesigning interface displays was the motivation for other another observation study, which aimed to provide information to the operator on the display monitor in a way that is compatible with their cognitive process (Koch et al., 2013). This study evaluated technician response time (situation awareness) and decision accuracy in HFE-informed designs. Compared with conventional monitors, the new interface decreased mental workload, which was confirmed by survey results. There are also studies on modifying the software interface of monitors to reduce the occurrence of false alarms (Aboukhalil, Nielsen, Saeed, Mark, & Clifford, 2008), which is the top leading cause of health technology hazard’s (ECRI, 2013). These findings suggest the need for further efforts to redesign telemetry monitoring stations, based on HFE principles to improve clinical support and mitigate the risk of missing signals or alarms. Neurophysiological Monitoring of Human Performance Another research shortcoming for operator performance evaluation in safety-critical environments is the reliance on subjective forms of control panel and monitoring technology design evaluation. In most cases, environments are mostly evaluated through expert evaluation of the control panel interface and the features of the alarm signals without measuring objective operator response (i.e., neurophysiological measures) to the design alternatives. When operators are integrated into the design evaluation process, subjective data collection (i.e., surveys) is typically the dominant form of user input. Although subjective input provides valuable information for the design process, more objective forms of data collection can be used to supplement subjective data and develop a more robust and reliable method to evaluate the impact of a design on operator performance.



Human performance can be evaluated through empirically collected neuropysiological response as indirect measures of operator stress and cognitive workload. We aim to integrate these measures in performance assessment while evaluating operators in different VR telemetry designs. Several studies have identified correlations between human performance and physiological and behavioral changes. Stress has been shown to have a negative impact on human performance (Porcelli et al., 2008), by changing many bodily systems such as the cardiovascular and neurological systems. This then affects measures such as heart rate, blood pressure, respiration rate, body temperature. For example, research has shown that when mental workload increases in tasks such as solving arithmetic problems, heart rate significantly increases (De Vries-Bouw et al., 2011). Other studies have found a link between an increase in mental workload and higher blood pressure (Allwood, Barcroft, Hayes, & Hirsjarvi, 1959). Research Objectives Our research will explore the impact of monitoring technology design (physical layout, interface design, alerts, etc.) on the performance of operators, specifically hospital telemetry technicians. This will be accomplished through simulation of an actual telemetry room (MedStar Washington Hospital Center) and patient alerting scenarios in VR with neurophysiological measures used to empirically assess human performance (human error, cognitive load and operator stress). Although simulation studies and observations in mockup monitoring station rooms has been conducted in this area (Görges, Westenskow, & Markewitz, 2012; Kobayashi et al., 2013), to the best of our knowledge, no study used VR environments to evaluate their design and technician performance. Due to the advantages of VR environments over 2D computer simulations and physical system mock-ups, we propose to use this technology in our research. The VR CAVE (cave automatic virtual environment) at the University of Maryland Hybrid-System Integration and Simulation Laboratory (Figure 1), will be used to provide an immersive 3D environment of the telemetry room. The outcomes of this research can be used to inform telemetry room design and ultimately improve human performance. The research results can also be extended to other safety-critical domains that have similar environmental and data monitoring characteristics.

METHODS The subsequent sections discuss the methods related to environment, design layout features, and alerting scenarios that will be applied to the telemetry room simulation. Existing Telemetry Station The Medstar Washington Hospital Center (WHC) was selected as an initial hospital setting to model the telemetry station, due to their large patient volume and ED telemetry monitoring capabilities (over 200 beds). The room consists of five telemetry stations with four overlay touchscreen monitors at each station. Each networked station computer is equipped with GE CIC Pro Central Station, which

2390

Figure 1: University of Maryland Virtual Reality CAVE provides high level detail patient information and vitals (i.e., heart rate, blood oxygen saturation, room location) and low level details (i.e., ECG) for each patient. Each monitor can display up to 16 patients, with 64 total per station. Visual and audio alerts are provided with the ability for the telemetry technician to personalize several station-specific features: patient vitals color categorization preferences; alerts sound volume (lowest 80%); and alert sensitivity for patients who are consistently abnormally high or low for a specific vital. Each telemetry station also has a telephone with a labeled direct line for each telemetry-equipped section of the hospital. We will duplicate the WHC specifications to provide a baseline for comparison of other design options intended to improve technician performance. Although retrospective data on WHC technician time-to-response, missed-signals, and near-misses is not available, the baseline VR design will be used to collect this data from actual telemetry technicians who have been trained using this configuration. Virtual Reality Simulation of a Telemetry Station The CAVE is a three-sided back-projected 3D environment (10ft square) with magnetic tracking and multiple forms of user input (joystick, glove) and output (surround sound and 3D glasses to visualize the environment). The telemetry room will be modeled using Unity, a 3D game engine used to design interactive environments. The simulated telemetry room, designed for one operator per experiment, will feature one of the five stations at WHC with four monitors, which will display patient vitals in a format similar to the GE CIC Pro Central Station software. For our experiment, participants will be seated in the CAVE facing the monitors. A physical table and chair will be placed in the CAVE with the monitors and surrounding environment virtually simulated. To replicate the computer input functions of the WHC telemetry room, touchscreen interfaces will be simulated. The CAVE’s tactile input functionality for the touchscreen interface will be produced by the glove (worn on the right or left hand). The glove has a magnetic tracking sensor on each finger, thus providing capabilities to locate the hand within the 3D space, touch the virtual screen, interact with patient data, and respond to alerts.



Patient Telemetry Station Design Features We will test and modify several design features of the telemetry station that will be narrowed down in preliminary design prototype testing with telemetry technicians (n=5) prior to collecting data in a larger technician population (n=30). 1. Graphical user interface: color coding, hierarchical menu options, intermittent visual signals, etc. 2. The number of patients displayed per monitor and per station: 64 patients is the WHC maximum per station (approx. 16 patients per monitor) 3. Layout of the monitors: use of peripheral vision, vertical and horizontal positioning, and monitor size. 4. Alarm sounds: differentiating sounds based on the alert type, frequency of alerts and impact on alarm fatigue. Our initial research and WHC technician interviews have shown that the user interface is one of the most critical design factors. Most control panels have bright contrasting colors that compete for the operator’s attention on the cluttered screens. In general, most systems also associate the color red as the state of alarm or failure. However, these flashing red alarms are sometimes not noticed because there are multiple contrasting color schemes that make it hard to distinguish alarm state. Recommendations suggest to improve the graphical user interface by using colors consistently in a dull-like screen to heavily contrast the colors used for the alarm systems (De Carvalho et al., 2006). By only confining the user interface to a few colors, users are more likely to notice an alarm trigger or change in the interface. Patient Simulation Scenarios To replicate the telemetry monitoring stations it was important to understand the scenarios that might arise to trigger an alarm and demand a response from the technician. We selected heart rate, respiration rate, blood pressure, temperature, and blood oxygen saturation to display on the monitors, which are the most common patient vitals used by technicians. Based on the selected vitals, we chose a set of normal and abnormal events to create different patient scenarios: stable, tachycardia, bradycardia and hypoxemia. To realistically display actual human data for each patient vital, stable patient data (at rest) was taken from samples of actual human subjects in the lab, while wearing the measurement tools for each vital. The stable patient scenario does not involve alarm activation, and will be the dominant status for all patients, with the exception of programmed alerting scenarios (tachycardia, bradycardia, hypoxemia). Similar to the stable scenario, the alerting scenario vitals data was derived from laboratory collected data, and then modified to provide the most realistic real-time record of data. Tachycardia is as an abnormally high heart rate. In typical adults it is classified as exceeding 100 beats per minute (bpm). The scenario for bradycardia is a low heart rate, typically classified as below 60 bpm in an adult. The last scenario is hypoxemia, which is low blood oxygen level characterized by a pulse oximeter reading below 90 percent. Each scenario begins with a stable patient. After a predetermined period of time, the vital (heart rate or blood oxygen level) begins to increase or decrease. After the vital

2391

has reached a critical level, response is required from the technician. This response includes clicking on the touchscreen interface to move from the low level detail multi-patient screen to reveal more high level detail for the patient of interest. All alarms (regardless of severity and true/false evaluation) require the technician to contact the patient care team in the area of the hospital listed on the interface. This action will be accomplished through the use of a simulated VR phone with hospital location push button tags. Once the participant has responded to the alarm, the vitals will return to normal levels (stable patient) after a predetermined time. Data Collection To assess design-induced cognitive workload and stress, participant (WHC telemetry technicians) neurophysiolgical responses during technology interaction will be collected in real-time. The presentation of non-stable patient data scenarios for each design and the ordering of individual designs, will be randomized for each participant. A combination of wired and wireless devices will be used. The Zephyr Bioharness is a biometric strap worn around the chest. The Bioharness communicates through Bluetooth and will be used to collect subject heart rate, heart rate variability, breathing rate, and core temperature. The CNAP Monitor is a wired system used to collect systolic, diastolic, mean blood pressure and pulse rate through the use of arm and finger cuffs. The B-Alert Wireless EEG and Cognitive State Metrics by AcqKnowledge is a 9 channel wireless head cap that measures electrical activity of the brain along the scalp. Specifically, B-Alert will be used to assess technician engagement, distraction, drowsiness and workload by monitoring different activation areas of the brain. Both BAlert and CNAP operate through a common BioPac interface. In addition, a customized eye tracking system design for the VR CAVE was integrated in 3D goggles by Mechdyne Corporation, and will be used to understand what aspects of design layout are attracting technician attention and eye gaze patterns during each simulated scenario. Pupil dilation, another factor associated with stress, will be evaluated using the eye tracking system. All experiments will be recorded through the use of several multi-angle pan-tilt-zoom network cameras and omindirectional microphones in the VR CAVE. Human performance will be quantified by measuring the response time from alarm activation to conclusion of response. Response will be in two forms: 1) touch screen clicking to verify receipt of the alerts and obtain additional patient information; 2) calling the appropriate section of the hospital. Data Analysis Multi-signal data will be integrated using a Matlab code to align frame rate and start time. Data obtained during a calibration period (resting state) will be used to calculate all delta values for neurophysiolgical measures. Statistical analyses will be used to determine significant relationships between human performance and neurophysiological predictors. In addition, the most significant predictors of human performance will be found, to guide future cognitive



load and stress evaluations of designs. A secondary goal will be the reduction of monitoring technologies (removal of nonsignificant predictors) included in the study. Multivariate and logistic regression will be used to evaluate the impact of design features on human performance.

DISCUSSION This study proposed two novel approaches to assess human performance in a hospital telemetry station. The use of VR technology to replicate a monitoring station, has the enormous benefit of providing the sense of immersion and design alternative flexibility, which has not previously been attempted to our knowledge in other research. Participants can hear the surrounding sounds and visualize the environment, the same way as the monitoring technicians. Moreover, design features can be implemented in the 3D model and can be tested before physically designing the feature. The other advantage of present study, is assessment of usability of design features, using neurophysiological response which provides a more robust form of human performance assessment to supplement surveys and expert elicitation. The research outcomes can extend to control room applications in other safety-critical domains by informing design: 1) the maximum number of signals that an individual operator can monitor simultaneously without degrading human performance; 2) the recognizable tone and frequency of alerts to identify abnormalities and reduce alarm fatigue; and 3) general design principles for control panel graphical user interfaces such as color coding, signal differentiation, screen layout, and hierarchical interface interaction.

CONCLUSION Telemetry monitoring stations are one of the most critical environments in a healthcare setting, where making errors or late responses may contribute to patient death. The objective of this study is to virtually simulate design alternatives and evaluate the impact on technician stres, and cognitive workload, thus informing human error risk mitigation strategies to improve the overall performance of monitoring operators. Future work will focus on selecting the combination of telemetry station experimental design alternatives and replicating these alternatives in VR. In addition, the number of neurophysiological measures will be refined. A long-term goal is to develop a comprehensive VR design and testing methodology for control panels in safetycritical environments that can be applied across domains.

ACKNOWLEDGEMENTS We would like to thank The University of Maryland James A. Clark School of Engineering for providing funding support for the VR CAVE and the telemetry room staff at MedStar Washington Hospital Center for providing invaluable knowledge on the daily telemetry station operation and technician needs.

2392

REFERENCES Aboukhalil, A., Nielsen, L., Saeed, M., Mark, R. G., & Clifford, G. D. (2008). Reducing false alarm rates for critical arrhythmias using the arterial blood pressure waveform. Journal of Biomedical Informatics, 41(3), 442–451. Aghina, M. A. C., Mól, A. C. A., Jorge, C. A. F., Freitas, V. G. G., Pereira, C. M. N. A., Lapa, C. M. F., & Cunha, G. G. (2012). Non-conventional interfaces for human-system interaction in nuclear plants’ virtual simulations. Progress in Nuclear Energy, 59, 33–43. Allwood, M. J., Barcroft, H., Hayes, J. P. L. A., & Hirsjarvi, E. A. (1959). The effect of mental arithmetic on the blood flow through normal, sympathectomized and hyperhidrotic hands. The Journal of Physiology, 148(1), 108–116. Bainbridge, L. (1983). Ironies of Automation. International Federation of Automatic Control, 19(6), 775–779. Bouchard, S., Côté, S., & Richard, D. C. S. (2007). Chapter 16 - Virtual reality applications for exposure. In D. C. S. Richard & D. Lauterbach (Eds.), Handbook of Exposure Therapies (pp. 347–388). Burlington: Academic Press. De Carvalho, P. V. R., Gomes, J. O., & Borges, R. S. (2011). Human centered design for nuclear power plant control room modernization. In CEUR Proceedings 4th Workshop HCP Human Centered Processes (pp. 25– 33). Genoa, Italy. De Vries-Bouw, M., Popma, A., Vermeiren, R., Doreleijers, T. A. H., Van De Ven, P. M., & Jansen, L. M. C. (2011). The predictive value of low heart rate and heart rate variability during stress for reoffending in delinquent male adolescents. Psychophysiology, 48(11), 1597–1604. ECRI institute. (2008). Alarm interventions during medical telemetry monitoring: A Failure Mode & Effects Analysis. ECRI institute. (2013). Top 10 health technology hazards for 2014. Health Devices, 42(11). Görges, M., Westenskow, D. R., & Markewitz, B. A. (2012). Evaluation of an integrated intensive care unit monitoring display by critical care fellow physicians. Journal of Clinical Monitoring and Computing, 26(6), 429– 436. Grabowski, M., & Sanborn, S. D. (2003). Human performance and embedded intelligent technology in safety-critical systems. International Journal of Human-Computer Studies, 58(6), 637–670. Kobayashi, L., Parchuri, R., Gardiner, F. G., Paolucci, G. A., Tomaselli, N. M., Al-Rasheed, R. S., & Jay, G. D. (2013). Use of in situ simulation and human factors engineering to assess and improve emergency department clinical systems for timely telemetry-based detection of lifethreatening arrhythmias. BMJ Quality & Safety, 22(1), 72–83. Koch, S. H., Weir, C., Westenskow, D., Gondan, M., Agutter, J., Haar, M., & Staggers, N. (2013). Evaluation of the effect of information integration in displays for ICU nurses on situation awareness and task completion time: A prospective randomized controlled study. International Journal of Medical Informatics, 82(8), 665–675. Luquetti dos Santos, I. J. A., dos Santos Grecco, C. H., Abreu Mol, A. C., & Rodrigues Carvalho, P. V. (2009). The use of questionnaire and virtual reality in the verification of the human factors issues in the design of nuclear control desk. International Journal of Industrial Ergonomics, 39(1), 159–166. McGloin, H., Adam, S. K., & Singer, M. (1999). Unexpected deaths and referrals to intensive care of patients on general wards. Are some cases potentially avoidable? Journal of the Royal College of Physicians of London, 33(3), 255–259. Orphanidou, C., Clifton, D., Khan, S., Smith, M., Feldmar, J., & Tarassenko, L. (2009). Telemetry-based vital sign monitoring for ambulatory hospital patients. Conference Proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Conference, 4650– 4653. Pennsylvania Patient Safety Advisory. (2009). Connecting Remote Cardiac Monitoring Issues With Care Areas, 6(3), 79–83. Porcelli, A. J., Cruz, D., Wenberg, K., Patterson, M. D., Biswal, B. B., & Rypma, B. (2008). The effects of acute stress on human prefrontal working memory systems. Physiology & Behavior, 95(3), 282–289.
