membership (team members are often rotating, with team members of varying skill levels, with some in training), the need for some team members to teach or be.
1486
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003
OBSERVATIONALMEASURES OF TEAM PROCESS AND PERFORMANCE IN HEALTHCARE
F. Jacob Seagull University of Maryland Baltimore, Maryland
Stephanie Guerlain University of Virginia Charlottesville, Virginia
This symposium presents four research approaches that demonstrate a set of diverse tools and techniques for the study of team process and performance. Observational analysis is a cornerstone of human factors research, and analysis at the team level requires approaches tailored to the special considerations of group processes. In exploring these issues, we first present the advantages of video data as a medium in an ergonomic and task analysis of brief, risky beneficial procedures, and comparisons to traditional observational methodologies are made. Next, both qualitative and quantitative methods are used for describing notable events and performing content analysis with a software tool (“RATE”) that facilitates the coding and analysis of video records of surgical teams. Third, we explore a technique that uses the communication patterns of trauma teams as a measure of team processes, structure and function. Lastly, we present a validated behavioral marking system (“ANTS”) for individual anesthetists within the context of a team that includes the dimensions of team work, task management, situation awareness and decision making. These four techniques examine a range of valuable analytic tools demonstrated through empirical research.
INTRODUCTION Team performance analysis is relevant to today’s complex workplaces but has not been extensively conducted in healthcare environments. The papers presented in this symposium report teamwork studies in the domains of anesthesia, routine and emergency surgery, and trauma resuscitation. These domains are difficult to study due to the variability in team membership (team members are often rotating, with team members of varying skill levels, with some in training), the need for some team members to teach or be taught while doing their primary job, and the diverse nature of patient conditions. Common approaches were for the investigators to study several examples of a particular type of problem (e.g., a particular procedure), to correlate technical with social skills, and to look at certain hnds of communication events (such as teaching, leadership, and exchanging information). All studies had human observers judging either technical or social slulls or both, and all studies either used or developed a standardized template for noting events of interest. Some investigators use software and/or video to assist in the standardization process. Team work can be rated globally, with an overall score, events, errors, or communications can be “counted” or “categorized” or specific exchanges can be analyzed in further detail to uncover events of interest, underlying themes, distractions, barriers, or training opportunities. We next summarize the four studies, and present some integrating conclusions at the end.
Mackenzie et al.: Observational Analysis of Video Records of Team Performance Mackenzie, et al., demonstrate that video can be a powerful addition to observational studies, as it allows fine-grain analysis due in part to the opportunity for repeated review of video records. This type of review can facilitate the identification system-factors affecting performance. Video acquisition and analysis of expert teams performing multiple simultaneous tasks in a dynamic, naturalistic domain provides a framework for extracting observational measurement. Team processes and team performance can be both qualitatively and quantitatively measured. Using a task analysis template derived from expert opinion, they extracted data from multiple performances of single complex tasks undertaken at two levels of urgency. Such an approach to video analysis has broad generalizability to examination of expert team performance in multiple complex domains. Data relevant to measurement of Performance can be obtained from the video record when such data may be difficult to collect, understand or interpret from observation alone.
Turrentine, Guerlain et al.: Studying Communication Turrentine, Guerlain and colleagues studied team communication patterns from both live observation and post-hoc re-review of 48 surgery cases at an academic
PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 47th ANNUAL MEETING—2003
medical center. In this study, communication was measured quantitatively by dividing the procedure into phases and then scoring individual team members on desired communication patterns using a grading scale from 1 to 5 . Using this measure, the intervention group (who used a checklist that stressed the use of a preoperative briefing) had significantly higher scores in several categories, such as introducing and empowering the team. In addition to this summative analysis, the authors conducted a more detailed analysis of the conversations recorded during the surgery. A subset of team member quotes were qualitatively analyzed and divided into four phases: setting the course, gathering data, exchanging information, and tribulations. This qualitative data underscored certain patterns of communication, and was used to develop baseline measures of communication in the OR for follow-on studies in team communication training and targeted technical training. Furthermore, the natural discourse recorded during surgery provided fodder for knowledge acquisition of critical cues used by surgeons during cases.
Xiao et al.: Team Communication Patterns as Measures of Team Processes: Xiao and his colleagues analyzed the communication patterns during resuscitation of trauma patients. Trauma teams often have fluid membership due to highly fluctuating workload and multi-disciplinary nature. Additionally, in many settings trauma teams consist of members who are in training. In their paper, team verbal communication was used to evaluate the impact of task urgency and shared team experience. To measure team communication, the communication frequencies among team members were used. The results provide initial support for the utility of communication analysis for the study of team performance and team leadership. Tools for assessment of team processes are a key to effective research in team performance, especially in complex, time-pressured tasks. The team communication patterns depicted the adaptive nature of team structures, especially when the teams were confronted with potentially competing goals, such as onthe-j ob training and treatment of trauma patients.
Flin et al.: Rating Anaesthetists’ Non-Technical Skills-The ANTS system Flin, Fletcher et al., present details of a behavioral marker system (Anesthetists’Non-Technical Skills (ANTS)) for rating individual anesthesiologists’ cognitive and social skills on the basis of observed
1487
behavior in a team context. The ANTS system can be used to rate behaviors from observations made in theatre, in the simulator or from video recordings. The four skill categories are task management, team work,, situation awareness and decision making. The importance of rating individual social skills as well as overall team skills was emphasized, as the anesthesiologists work with many different team members. The elements of the teamwork category are coordinating activities with team members, exchanging information, using authority and assertiveness, assessing capabilities, and supporting others. Findings from an experimental study where consultant anesthesiologists rated the behavior of anesthesiologistsfrom videotapes of team scenarios showed that inter-rater reliability was highest for task management and team working and lowest for situation awareness.
CONCLUSIONS The papers presented in this symposium provide a diverse sample of measurement techniques for team processes and performance in healthcare. While diverse, there were also commonalities across the collection. All used video at some point as a way to supplement the objectiveness of the measures. While not limited to video use, these observational techniques benefited from the advantages that video affords, such as the use of multiple raters, and repeated viewing for analysis at high level of detail. Assistive software tools (RATE) or formalized, validated measurement scales (such as ANTS) were also presented. These tools and techniques may be available to researchers for further validation or for application to novel domains. The presented papers also stressed the investigation of team processes as compared to outcome measures alone. In the healthcare setting, outcome measures are often confounded by inherent variability or external influences that limit the ability to control experimental variables. Examination of processes, thus, can be a fruitful strategy in searching for measurable impact of experimental manipulations on medical practices. Finally, the strength of observational techniques is not limited to the use of quantifiable, objective measures. Contributors to this symposium make specific reference to the value of observing “interesting,” uncommon events and drawing qualitative conclusions from those relatively unique and less-quantifiable episodes. Validated, effective measures for objective assessment of medical and team performance remain an important goal as there is a growing need for investigation of team performance and processes within healthcare settings.
