motion, economy of diathermy, and generates an over- all score for each repetition that equally weights score components for both hands. All subjects were ...
Determining Standards for Eaparoscopic Proficiency Using Virtual Reality WILLIAM C. BRUNNER, M.D.,* JAMES R. KORNDORFFER, JR., M.D.,* RAFAEL SIERRA, M.D.,* J. BRUCE DLNNE, PH.D.,* C . LILLIAN YAU, PH.D.,t RALPH L. CORSETTI, M.D.,* DOUGLAS P. SLAKEY, M.D.,* MICHAEL C. TOWNSEND, M.D.,* DANIEL J. SCOTT, M.D.*
From the Department of * Surgery and the fDepartment of Biostatistics, Tulane University Health Sciences Center, New Orleans, Louisiana Laparoscopic training using virtual reality has proven effective, but rates of skill acquisition vary widely. We hypothesize that training fo predetertnined expert levels may more efficiently establish proficiency. Oui' purpose was to determine expert levels for performance-based training. Four surgeons established as laparoscopic experts performed 11 repetitions of 12 tasks. One surgeon (EXP-1) had extensive Minimally Invasive Surgical Trainer-Virtual Reality (MIST VR) exposure and formal laparoscopic fellowship training. Trimmed mean scores for each were determined as expert levels. A composite score (EXF-C) was defined as the average of all four expert levels. Thirty-seven surger^/ residents without prior MIST VR exposure and two research residents with extensive MIST VR exposure completed three repetitions of each task to determine baseline performance. Scores for EXF-1 and EXP-C were plotted against the best score of each participant. On average, the EXI'-C level was reached or exceeded by 7 of the 37 (19%) residents. In contrast, the EXP-1 level was reached or exceeded by 1 of 37 (3%) residents and both research residents on all tasks. These data suggest the EXP-C level may be too lenient, whereas the HXP-1 level is more challenging and should result in adequate skill acquisition. Such standards should be further developed and integrated into surgical education.
D
URING THE PAST DECADE, laparoscopic surgery has undergone an explosive growth in tenns of clinical applications. For mary procedures, an open approach is now rarely indicated, and a laparoscopic approach has become the standard of care. However, laparoscopy continues to demonstrate a long learning curve due to technical factors, including the use of long instruments that provide decreased tactile feedback, a two-dimensional imaging system that alters depth perception, and fixed access points in the abdominal wall that create a fulcrum effect and limit range of motion.'^^ Many surgical residents may not receive sufficient training to perform advanced laparoscopic procedures upon graduation.''^ In an effort to augment training, laparoscopic skills laboratories have populari/.ed the use of surgical simulation. Several studies have demonstrated that skills developed on simulators transfer to the operating room Presented at the Annual Scientific Meeting and Poslgraduale Course Program. Southeastern Surgical Congress. Atlanta. GA. January 31-February 3. 2004. Address correspondence and reprint reiiiiesls to Daniel J. Scott, M.D., Assistant Professor of Surgery. Director. Tulane Center for Minimally Invasive Surgery. Departmeni of Surgery. SL-22. 1430 Tulanc Avenue. New Orleans, LA 701 12. 29
and improve performance during actual operations.^ ^ Skills laboratories are now playing an increasing role in surgical education because of numerous constraints."*- " The use of operating room time is very costly and often a limited resource.'- Case numbers are variable, and relying solely on operative experience may be inadequate. Resident work-hour restrictions further limit the amount of clinical experience available. Meanwhile, a heightened public awareness concerning patient safety has resulted in an outcry for improvements in training.'"^ Numerous nationwide initiatives have been launched to define and objectively measure competency, including programs by the Accreditation Council for Graduate Medical Education (ACGME) and the American College of Surgeons, but very few standards have been established for technical skills.'"^ Now more than ever, innovative, efficient, and effective training methods are needed, and surgical simulation holds great promise. The Minimally Invasive Surgical Trainer-Virtual Reality, or MIST VR (Mentice. Inc.. San Diego. CA) is a system that has been extensively validated, but there is no consensus as to an optima! curriculum design. Using a predetermined training duration or number of repetitions has been successful, but a "one size fits all" protocol does not account for the wide vari-
30
THE AMERICAN SURGEON
ability in individual rates of skill acquisition.'^- '•'^^''' Determining plateaus in performance may better identify training endpoints but is impractical for real-time analysis.'^ E.stablishing expert performance leveis for goal-oriented practice may enhance efficiency and help ensure proficiency but have only been published for I of the 12 cutrently available MIST VR tasks.'^ The purpose of this study was to develop a comprehensive set of standards for performance-based training of surgery residents. Materials und Methods MIST VR is a computer-based simulator that consists of a Pentium-cia.ss personal computer with a video monitor and a laparoscopic interface (Immersion Medical. Gaithersburg. MD). The interface includes a frame with two pistol-grip instrument handles and an electrocautery foot pedal to create a virtual laparoscopic environment. Twelve basic laparoscopic task.s are available, six in Core Skills I and six in Core Skills 2. as previously described.''' The computer automatically collects data concerning time, errors, economy of motion, economy of diathermy, and generates an overall score for each repetition that equally weights score components for both hands. All subjects were trained on one of three identicai MIST VR stations in the Tulane Center for Minimally Invasive Surgery Simulation and Training Laboratory (Fig. 1). All subjects were given a brief introduction to the system including avoidance of errors and task design, and the computer-generated tutorials were demonstrated for each task. The level of difficulty was set to the "Easy" default settings for all testing. Two research assistants were available for assistance and questions, although no active instruction was given.
FIG. I. Three MIST VR stations in ihe Tulane Center for Minimally Invasive Stirgery Simulation and Training Laboratory.
January 2005
Vol. 71
Board-certified surgeons (n = 4) with extensive laparoscopic experience (>100 choiecystectomies) volunteered as experts for the purpose of establishing performance standards. One surgeon had extensive prior exposure to the MIST VR system as well as formal fellowship training in laparoscopy. Three surgeons had extensive clinical experience in both basic and advanced iaparoscopy including herniorraphy. Nissen fundoplication. splenectomy, coiectomy, and living donor nephrectomy, but no prior exposure to the MIST VR system. Each expert surgeon was given a warm-up period of up to I hour to become accustomed to the system and then completed 11 consecutive repetitions of each ofthe 12 tasks. The mean overall score and standard deviation (SD) for each surgeon was determined. Outlying .scores, defined as >2 SD above the mean, were trimmed and the mean was recalculated. The trimmed means were used as expert levels, and an expert composite score (EXP-C) was defined as the average of the four expert levels. Surgery residents (n = 37, R1-R5) were voluntarily enrolled under an IRB-approved protocol, and none had prior MIST VR exposure. Each resident completed three consecutive repetitions of all 12 tasks as a baseline a.ssessment. The best overall .score achieved during baseline testing for each subject was compared with the expert levels. Surgery re.search residents (n = 2. R2. R4) volunteered as pilot training subjects. Although formal training duration was not recorded, both research residents had extensive hands-on experience during system in.stallation, subsequent curriculum development, and ongoing training supervision. Both research residents had more than 20 hours of MIST VR exposure and had ma.stered all 12 tasks. Each research resident completed 3 consecutive repetitions of all 12 ta.sks for comparison with the expert levels. The best overail score achieved by each resident and research resident was plotted for all tasks to allow graphical comparisons with expert levels. Data was compiled using Microsoft Excel (Microsoft, Inc., Redmond. WA). Analysis was performed using Sigma Stat and graphically depicted using Sigma Plot (SPSS. Inc., Chicago, IL). Results Eour expert surgeons, all male, all right-hand dominant, completed 11 repetitions of each MIST VR task in two 2'hour sessions with I hour of warm-up. Trimmed mean and standard deviation for each task for each surgeon and all four surgeons together were determined as listed in Table 1. Trimming removed less than 1 score per surgeon per task, or 3% of overall values. The laparoscopic fellowship-trained expert attained the best trimmed mean score among the 4 experts for each of the 12 tasks, with results shown
No. 1
LAPAROSCOPIC PROFICIENCY USING VIRTUAL REALITY
Brunner et al.
31
TABLE I. Expert Surgeon Performance*
ri^ in
O
Task
EXP-1
EXP-2
EXP-3
EXP-4
EXP-C
Acquire place Tran.sfer place Traversal Withdraw in.scrt Diathermy Maniptilaie diathi;rmy
13.8 ±2.9 16.9 ± 1.5 15.4 ±2.8 12.3 ±1.2 24.2 ± 1.9 36.6 ± 5.2
15.8 ±4.8 25.5 ± 6.7 26.2 ± 10.6 19.6 ±2.4 25.8 ±2.8 58.8 ± 16.8
23.1 ±9.1 17.3 ± 2.2 33.7 ± 13.3 15.0 ±2.0 27.0 ± 3.7 41.2 ±5.3
15.8 ± 3.4 26.5 ± 4.3 31.3 + 5.8 20.9 ± 5.9 31.7 ±6.5 53.6 ±7.5
14.8 ±3.5 21.5 ±3.7 26.7 ±8.1 17.0 ±2.9 27.2 ±3.7 47.6 ±8.7
SC Stretch 12.0 ± 1.7 9.9 ± 1.6 SC Clip 8.5 ± 1.8 II.1 ±3.8 SC Stretch clip 12.0 ± t.6 18.4 ±5.2 SD Stretch lO.O ± 0.9 10.3 ± 1.2 SD Diathermy 29.0 ±1.1 30.8 ± 1.6 SD Stretch diathe-rmy 37.4 ± 4.0 40.9 ± 3.8 * Values at« mean ± SD. t imiued means shown for Experts 1-4, EXP-C SC, stretch clip: SD. stretci diaihermy.
8
graphically as EXP-1 in Figs. 2 and 3. The performance levei combining lhe trimmed mean scores of all four surgeons is graphically depicted as EXP-C in Figs. 2 and 3. Thirty-seven surgery residents (R1-R5), 11 women and 26 men. 2 left-hand dominant, completed 3 consecutive repetitions of each task during two 1-hour sessions. Two research residents (RR), both tnale, I left-hand dominant, also completed three consecutive repetitions in less than 1 hour. The best overall scores for each subject, grouped by residency training level are shown in Figs. 2 and 3. Two or more (2-12, mean 6.42) resident trainees were able to reach the FXP-C standard within the first 3 repetitions for all 12 tasks, as shown in Table 2. The training level of residents surpassing the FXP-C pertbmiance standard varied widely from R1-R5, with senior residents (R4-R5, n = 9, 24% of trainees) representing 33 per cent to 100 per cent of the residents reaching proficiency for any given task. At least one resident trainee (1-7, mean 1.7) reached the EXP-1 standard for 9 of the 12 tasks. Senior residents alone reached the FXP-I standard in six tasks, and represented 28 per cent to 67 per cent of the residents reaching proficiency in three tasks. No resident trainees were able to reach the FXP-1 standaid at baseline testing for 3 of the 12 tasks (Traversal. Withdraw Insert, and Manipulate Diathermy). Both research residents reached the EXP-C and EXP-1 levels for all 12 tasks. Discussion The use of expert levels for MIST VR training has only been reported previously in one study by Seymour.^ Seymour established an expert level based on the average score achieved by four surgeons during 10 repetitions of a single task. Manipulate Diathermy, using the "Difficult" default setting. In a pro.spective randomized trial, residents who practiced the Manipu-
12.0± 1.5 14.5 ±2.2 12.1 ± 1.7 11.3 ± 2.9 I2.I ±2.1 10.8 ±2.6 17.2 + 3.7 18.7 ±2.9 16.6 ±3.3 12.1 ±2.6 11.6± 1.9 1 1.0 ± 1.7 34.1 ±2.1 36.9 ± 2.0 32.7 ± i.7 46.3 ± 5.4 48.9 ± 3.8 43.4 ±4.2 defined as the average from all expert data.
late Diathermy task until the expert level was reached {3-8 hours) committed significantly fewer errors and achieved better performance during an actual Iaparoscopic cholecystectotny. Seymour's study thus validated expert levels as effective training endpoints. Using such standards .should better ensure proficiency compared to traditional curricula that require trainees to practice for a predetermined duration or number of repetitions. Upon completion of time or repetition oriented protocols, trainees may have reached different ieveis of performance: some may be proficient and others may not, depending on the rate of learning for each individual. In contra.st, perlomiancebased endpoints allow training to be tailored to each individual's needs. Subjects who require extensive ttaining receive it, whereas those with greater innate ability or faster rate of skill acqtiisition complete training tnore rapidly. The end result is that all trainees tnaster a given skill set to a specified standard, thus ensuring proficiency in that domain. Additionally, using such standards tnaxitnizes efficiency, as valuable time is not wasted overtraining individuals who rapidly acquire a skill. Aside from the single expert level (on the difficult setting) established in Seymour's study, no standards have been established for the MIST VR system. We routinely train medical students and residents in our laboratory using the easy default settings for all 12 currently available MIST VR tasks. Our goal was to generate a comptehensive set of standards for training purposes. To this end, we included Core Skills 2 exercises, which seem complimentary to Core Skills 1 but have not been widely studied. Previous work suggests that significant improvement in operative performance may be achieved usitig the easy default settings for early training.'^ We chose easy default settings as suitable for our novice trainees, and our current standards demand near error-free petformance to reach
32
THE AMERICAN SURGEON
January 2005
Acquire Place
Vol. 71
Transfer Place •
EXP 1 l i e ej
0 •
40
* 0
I
1 t
.
t
1
•
t
*
1
0
t
•
a
RJ
R3
tl
Resident Year
Resident Year
Traversal
Withdraw insert
BS
—
EXP-1 (IS < EXP-C (18
RR
EXP-C (IT 0)
i J_ 1
R2
R3
ai
RS
Rl
R2
R3
nt
fiS
Resideni Year
Resident Year
Diathermy
Manipuiate Diathermy EXP-1 (24 2) EXP-C (2T Jl
BO
EXP-113BB) eXP-C (47 Bl
i
SO
t ff
"
•
8 "
30
1
i
: 1
I
- -i—
30 '
10 Rl
R2
"S
Ft*
RS
RR
Resident Year
RI
(tJ
R4
Resident Year
FIG. 2. Resideni Core Skills I baseline performance compared to EXP-C and EXP-1 standards.
proficiency consistently. During system introduction for each trainee, we stress error-a voi dance first, as iiiiprovements in speed, efficiency, and overall score will follow. Although the rationale for using performance standards is clear, iittle work has been done to identify an accurate method for determining suitable performance standards. We therefore explored several aspects of this process. Using Seymour's model, we chose to generate performance data from known experts. Using I 1 repetitions of 12 tasks for 4 experts resulted in a tremendous database for analysis, with 528 overall scores. Despite using a homogenous group of experts {with similar
clinical skills), we noted a distinct difference in the surgeon (Expert 1) who had extensive previous experience with the MIST VR system. We know that VR translates to OR (MIST VR training improves operative performance) but we now have a new question. Does OR translate to VR? Judging by the performance of Experts 2. 3, and 4 (Table 1). who had no prior MIST VR exposure, the answer might be "no." Although Core Skills ! has been extensively validated, the MIST VR system does not have complete face validity, defmed as the extent to which the simulation resembles the real task, or fidelity. The sy.stem lacks tactile (haptic) feedback, does not use actual lap-
No. 1
LAPAROSCOPIC PROFICIENCY USING VIRTUAL REALITY
Brunner et at.
33
SC Clip
SC stretch
EXP.I (B t] EXP.C I1I.1)
EXP.1 l« SJ EKP-C (10 »J
30
35
20
1 I
1
•
•
I
t
1
10
1
1
I
5
•
Resident Year
Resident Year
SC stretch Clip
SD Stretch EXP-1 [10 0) EXP-C (11 OJ
EXP.I 112 01 EXP-C (1B8)
£
H3
20 •
R*
Resident Year
Resident Year
SD Diathermy
SD Stretch Diathermy EXp.1(3/4] EXP-C ( « • )
•
I »
Resident Year
FiG. 3.
'
*
Resident Year
Resident Core Skills 2 baseline performance compared lo EXP-C and EXP-1 standards.
aroscopic imaging, and relies solely on color-coded. 3D graphics for depth perception.'-'^ Although these shortcomings seem not to hinder the value of MIST VR as a training tool, they may interfere with assessment. Although all experts tested were known to possess sufficient skills during actual operations, when faced with the artificial environment created by the virtual simulator, they were penalized for deviating from the single correct pathway dictated by the software, when in reality, their pathway may have been an appropriate alternative. In other words, except for Expert I who had previous MIST VR exposure. Experts
2, 3, and 4 had difficulty with the game interface, as reflected in their scores. When comparing the number of residents who reached the EXP-1 and EXP-C levels, the more stringent standard seems appropriate. Our goal is to rigorously train these individuals and to avoid undertraining. Although 46 per cent of subjects achieving proficiency were junior residents using the EXP-C level (Table 2). they represented only 30 per cent of residents reaching the EXP-1 performance level. When the more robust Core Skills 1 skill set was examined overall, the rate of junior residents achieving
THE AMERICAN SURGEON
34
January 2005
Voi. 71
TABLE 2. Number of Residents Achieving Proficiency During Baseline Testing*
1
U
cS
u
Rl
R2
R3
R4
R5
RR
Task
(u = 17)
(n = 5)
(n - 6)
(n - 4)
(n - 5)
Acquire place Transfer place Traversal WithdrLiw insert Diathermy Manipulate diathermy
0(0) 0(0) 0(1) 0(3) 0(0) 0(0)
0(0) 0(1) 0(0) 0(0) 1(1) 0(0)
0(1) 0(0) 0(0) 0(3) 0(1) 0(0)
1(1) 0(0) 0(2) 0(1) 1(2) 0(0)
0(0) 2(4) 0(1) 0(3) 1(3) 0(0)
tn - 2) 2(2) 2(2) 2(2) 2(2) 2(2) 2(2)
SC Stretch SC Clip SC Stretch clip SD Stretch SD Diaihermy SD Sticlch diathemiy
0(0) 3(6) 0(3) 0(2) 1 (1) 0(4)
0(0) 0(0) 0(1) 0(1) 0(0) 0(1)
0(0) 0(1) 0(1) 0(1) 0(0) 0(1)
0(0) 0(0) 0(1) 0(1) 1(2) 0(2)
1(3) 3(4) 0(3) 1(3) 0(1) 1 (4)
2(2) 2(2) 2(2) 2(2) 2(2) 2(2)
* Baseline testing defined as the best score from three repetitions, values are for EXP-l (EXP-C) levels. SC. stretch clip: SD. stretch diathermy. 3. Recommended Performance Levels for MIST VR Core Skills 1 and 2
TABLH
Core Skills 1
Core Skills 2
Acquire Place 13.8 Transfer Place 16.9 Traversal 15.4 Withdraw Insert 12.3 Diathermy 24.2 Manipulate Diathermy 36.6
SC Stretch 9.9 SC Clip 8.5 SC Stretch Clip 12.0 SD Stretch IOO SD Diaihermy 29.0 SD Stretch Diathermy 37.4
SC, streteh clip; SD, stretch diathermy. proficiency drops from 39 per cent lo 17 per cent. Overall, the pa.ss rate for residents using EXP-C approached 19 percent (7 of 37). which dropped to 3 per cent (1 of 37) using the EXP-l criteria. Senior residents represented 71 per cent of residents who reached the EXP-1 ievel for any given task. According lo these data, our recommendation for performance levels i.s based on the EXP-1 levels (Table 3). Frotn the pilot data generated by our research residents, the recommended performance levels seem attainable, as both subjects achieved proficiency on all 12 tasks (Table 2). In conclusion, this study provides the first set of comprehensive standards to aid programs using the MIST VR system tor training. Using the recommended levels should enhance training efficiency and better ensure proficiency. Additional studies are needed to fully validate these levels as achievable for ali trainees. Further investigation will also be needed before these standards can be used for assessment purposes. Acknowledgment Equipment (one MIST VR station) was provided in part through an unrestricted educational grant from Ethicon Endo-Surgery. Inc.
REFERENCES 1. Jordan JA. Gallagher AG. McGuigan .1. McCiure N. Virtual reality training leads to faster adaptalioii to the novel psychomotor restrictions encountered by laparoscopic surgeotis. Surg Etidosc 2001:15:1080-1. 2. Jones DB. Brewer JD. Soper NJ. The inllucnce of threedimensional video systems on laparoscopic task performance. Surg Laparosc Endosc 1996:6:191-7. 3. Gallagher AG. McCiure N. McGuigan J. et al. An ergonomie analysis of the "fulcrum eflecl" in endoscopic skill acquisition. Endoscopy 1998;30:617-2O. 4. Rattner DW, Apelgren KN. Eubanks WS. The need for training opportunities in advanced laparoscopic surgery: The residents' perspective. Surg Endosc 2()01: l.'5:1066-70, 5. Park A. Wii/ke DB. Donnelly MB. Ongoing deficits in resident training for minimally invasive surgery. J Gastrointest Surg 2002:6:501-9. 6. Chung R. Pham Q. Wojtasik L, et al. The laparoscopic experience of surgical graduates in the United States. Surg Endosc 2003:17:1792-5. 7. Scott DJ. Bergen PC. Rege RV. et al. Laparoscopic training on hench models: belter and more cost effective than operating room experience? J Am Coll Surg 2000:191:272-83. 8. Hamilton HC, Scott DJ, Kapoor A, et al. Improving operative performance using a laparoscopic hernia simulator. Am J Surg 2OOhl«2:725-X. 9. Seymour NE. Gallagher AG. Roman SA. et al. Virtual reality training improves operating room perlormancc: resuUs of a randomi/ed. double-hlinded sludy. Ann Surg 2002:236:458-63. 10. Bergamaschi R. Earewell to see one. do one. teach one? Surg Endosc 2001:15:637. 11. Callcry MP. Expansion beyond compression. Surg Endosc 2003:17:677-8. 12. Bridges M. Diamond DL. Thefinancialimpact of teaching surgieal residents in the operating room. Am J Surg 1999:177: 28-32. 13. Kohn LT. Corrigan JM, Donaldson M. To Err Is Human: Building a Safer Health System. Washington, DC: Institute of Medicine. 1999
No. 1
U\PAROSCOPIC PROFICIENCY USING VIRTUAL REALITY
14. Sabo RR. The challenge of emerging surgical technology: the college can help. Bull Am Coll Surg 2OO2;87:8-I4. 15. Hamilton EC. Scott DJ. Fleming JB. et al. Compari.son of videotrainer and virtual reality on acquisition of laparoscopic skills. Surg Endosc 2002;i6:406-l I. 16. Gallagher AG. Satava RM. Virtual reality as a metric for the assessment of iaparoscopic p^.ychomotor skills. Learning curves and reliahility measure.s. Surg Endosc 2002; 16:! 746-52. 17. Ali MR, Mowery Y. Kaplan B. DeMaHa EJ. Training ihe
Brunner et at.
35
novice in taparo.scopy. More challenge is better. Surg Endosc 20()2;16:l732-6. 18. Brunner WC. Korndortfer JR Jr, Sierra R. et al. Laparoscopic virtual reality training: are 30 repetitions enough? J Surg Research 2004:122:150-6. 19. Scott DJ. Jones DB. Virtual reality training and teaching tools. In: Soper NJ. Swanstrom LL. Eubanks WS, eds. Mastery of Endoscopic and Laparoscopic Surgery. 2nd ed. Philadelphia: Lippincott. Williams & Wilkins. 2004 (in press)
