Evidence of virtual patients as a facilitative learning ...

21 downloads 3002 Views 431KB Size Report
Nov 12, 2014 - original VPs were mostly computer based clinical cases (Haag et al. 2007, Zary et al. 2006). ... from a dedicated main server. The anesthesia ...
Adv in Health Sci Educ DOI 10.1007/s10459-014-9570-0

Evidence of virtual patients as a facilitative learning tool on an anesthesia course Joseph Y. C. Leung • Lester A. H. Critchley Alex L. K. Yung • Shekhar M. Kumta



Received: 4 April 2014 / Accepted: 12 November 2014  Springer Science+Business Media Dordrecht 2014

Abstract Virtual patients are computerised representations of realistic clinical cases. They were developed to teach clinical reasoning skills through delivery of multiple standardized patient cases. The anesthesia course at The Chinese University of Hong Kong developed two novel types of virtual patients, formative assessment cases studies and storyline, to teach its final year medical students on a 2 week rotational course. Acute pain management cases were used to test if these two types of virtual patient could enhance student learning. A 2 9 2 cross over study was performed in academic year 2010–2011 on 130 students divided into four groups of 32–34. Performance was evaluated by acute pain management items set within three examinations; an end of module 60-item multiple choice paper, a short answer modified essay paper and the end of year final surgery modified essay paper. The pain management case studies were found to enhanced student performance in all three examinations, whilst the storyline virtual patient had no demonstrable effect. Student-teaching evaluation questionnaires showed that the case studies were favored more than the storyline virtual patient. Login times showed that students on average logged onto the case studies for 6 h, whereas only half the students logged on and used the storyline virtual patient. Formative assessment case studies were well liked by the students and reinforced learning of clinical algorithms through repetition and feedback, whereas the educational role of the more narrative and less interactive storyline virtual patient was less clear.

J. Y. C. Leung  A. L. K. Yung  S. M. Kumta Teaching and Learning Resources Centre, The Chinese University of Hong Kong, Shatin, Hong Kong J. Y. C. Leung (&) Office of Educational Services, Prince of Wales Hospital, Shatin, New Territories, Hong Kong e-mail: [email protected] L. A. H. Critchley Department of Anaesthesia and Intensive Care, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong

123

J. Y. C. Leung et al.

Keywords Anesthesia  Pain management  Medical education  E-learning  Simulation  Virtual patient

Introduction Advances in modern technology have improved medical science greatly with numerous breakthroughs in the past century. However, this new knowledge demands new delivery plans from medical teachers in training medical staffs to take advantage of the full potential that modern technology gives them. Better utilized resources and improvement of the quality of future doctors can be achieved if we are able to incorporate new technology into our medical education (Berwick and Finkelstein 2010). With the above concept in mind, modern technology has facilitated the replacement of traditional methods of classroom based teaching and work place apprenticeships with more sophisticated teaching methods, such as simulation (Issenberg et al. 2005; Cook et al. 2011). The evolution of computer technology, the internet and E-learning has resulted in the development one specific type of simulation called virtual patient (VP), which is best defined as computerized representations of realistic clinical cases (Ellaway et al. 2008). However, the effectiveness of VPs in medical education has yet to be established fully. The first publish description of VPs dates back to 1973 (Harless et al. 1973) and the original VPs were mostly computer based clinical cases (Haag et al. 2007, Zary et al. 2006). However, their applications are not just confined to medicine, but also to a wide range of other health care disciplines, such as dentistry, nursing and pharmacy (Chaikoolvatana and Goodyer 2003; Garrett and Callear 2001; Littlefield et al. 2003). VPs were developed originally to teach clinical reasoning through delivery of clinical cases and data. The advantages of using VPs in medical education include the ability to present an unlimited array of standardized patient cases and provide a sense of reality without needing real patients (Cook and Triola 2009). The evolution of the internet and mobile communication technology has changed the way we teach, but also the potential uses of VPs on courses (Cook et al. 2010). The concept of students accessing VPs at any time in any place has emerged. Thus, VPs allow clinical learning to take place outside the classroom, hospital or clinic. Furthermore, the classical teaching role of VPs as vehicles to present standard clinical cases and promote clinical reasoning has broadened to deliver teaching on more specialized procedural areas of undergraduate medicine, such as anesthesia and acute pain management (APM) (Critchley et al. 2009; Leung et al. 2011a). The purpose of the present study was to determine whether VPs used on a short final year medicine 2-week anesthesia course would improve learning. Use of VPs in undergraduate anesthesia The Department of Anaesthesia and Intensive Care of The Chinese University of Hong Kong (CUHK) has used VPs to supplement class room and clinical teaching on its undergraduate final year medicine course since 2006 (Critchley et al. 2009). They use a VP platform called Formative Assessment Case Studies (FACS) which was written in house by the CUHK Teaching and Learning Resources Centre (TLRC). The VP platform is housed and maintained by the centre (Kumta et al. 2003). FACS is a sequential branching VP platform that uses question nodes and feedback (Table 1). It is accessed via the internet

123

Effectiveness of virtual patients Table 1 Instructional design of FACS and storyline VP Instructional design

FACS

SL-VP

Path type

Branching with single main forward path

Linear navigated only forwards

User modality

Single user

Single user

Media and resources

Text, static graphics and images

Text, static graphics and images

Narrative use and patient focus

Told from attending doctor’s perspective

Told from patient’s perspective

Interactivity use

Multiple choice decision steps

User summarizes key points in free text boxes

Feedback use

Provided for both correct and incorrect decisions

Model answer provided

Framework showing the typology of (a) the FACS cases and (b) the Storyline VP (Huwendiek et al. 2009)

from a dedicated main server. The anesthesia course uses six FACS-VP cases to supplement teaching of preoperative Anesthetic care. Individual cases are centred on a single main theme which included: (1) preoperative assessment of a healthy patient, (2) a patient scheduled for spinal anesthesia, (3) laboratory investigations in an elderly patient, and patients with (4) diabetes, (5) chronic lung disease, and (6) heart disease. Each case consisted of 32–49 webpages with 9–13 decision steps with feedback pages for incorrect decisions. These FACS cases introduce the student to the more important perioperative medicine issues relevant to working as a surgical house officer. There is also a link to a supplementary materials website that contained notes on 19 different topics related to the content of these FACS cases. Providing this resource prevented overloading of the FACS cases with supporting and often repeated information. Students spent on average 13–20 min working through each FACS case, but time spent varied from 0 to 60 min with quartiles (i.e. 25th and 75th) from 5 to over 30 min, and cases were often revisited several times (Critchley et al. 2009). The undergraduate anesthesia syllabus is described more fully elsewhere (Critchley et al. 2009). In 2008, the Department of Anaesthesia and Intensive Care introduced a second and different style longitudinal VP called storyline (SL-VP) (Critchley et al. 2008). Longitudinal VPs describe the progress of an illness, or course of treatment, over time and had previously been developed and used on the Edinburgh University Medical School undergraduate medicine course (unpublished work). The anesthesia SL-VP used narrative to describe the story of a patient admitted to hospital for routine surgery which takes place over several days. The software design of this style of VP was a linear series of consecutive webpages divided into distinct chapters (Table 1). Each chapter consisted of 21–30 webpages with 21–47 multimedia applications mainly pictures and tables. Interactivity was limited to question pages that required the student to summarize key points from the preceding webpages as free text answers which they subsequently compared to a model answer and could be self-scored. The SL-VP via text narrative and multimedia applications provided the student with the longitudinal experience of a typical patient receiving anesthetic care starting with admission to hospital and preoperative assessment, then the various stages anesthetic care during surgery, and ending with post-operative recovery room care (Leung et al. 2011a). The case scenario chosen was a middle aged lady admitted for hysterectomy who was diabetic and hypertensive.

123

J. Y. C. Leung et al.

When first introduced onto the course students would receive a 1–2 h computer laboratory session working through the SL-VPs followed by a focused group interview (Leung et al. 2010). This was latter reduced to a shorter 30 min introductory tutorial that showed the students how to access and use the FACS system. They accessed the system from the library or at home. The SL-VP is also supported by the FACS platform at the TLRC and accessed via the internet. A conceptual basis for the VP interventions Previously, Kern et al. had described six conceptual steps to curriculum development which are as follows: (1) problem identification, (2) needs assessment of the learner, (3) education goals, (4) instructional strategies, (5) implementation and (6) evaluation and feedback (Bordage 2009; Kern et al. 1998). To address the first step (1) problem identification, difficulties had arisen on the CUHK anesthesia course with attaching its medical students to sufficient pre-operative anesthesia ward rounds and exposing them to appropriate clinical cases. (2) The needs assessment of the learner was that CUHK students would soon be surgical house officer’s and it was desirable that they had some background training in perioperative medicine and understood of the role of anesthesia. (3) Thus, the educational goals were to provide (a) a basic knowledge of what happens to a patient when they have an anesthetic and (b) understanding of the key anesthetic issues when preparing patients for surgery. (4) The instructional strategy had in previous years been lectures supported by printed handouts and clinical attachments, but this approach was failing because of increased student numbers, changes in clinical practice (i.e. move towards pre-operative assessment clinics and day of surgery admissions) and limited resources (i.e. availability of tutors for afternoon preoperative clinical teaching rounds). Thus introducing VPs to supplement anesthesia teaching proved to be successful because by introducing students to a range of virtual cases the lack of clinical opportunities to see relevant cases was partly overcome (Leung et al. 2010). (5) Implementation of VPs on the anesthesia course has already been described in the previous section. (6) Although the feedback from student course evaluation forms was positive (Critchley et al. 2009; Leung et al. 2010), there was limited evaluation evidence that showed whether introducing VPs into the anesthesia course did in fact improve students learning. Hence, this was the key issue that needed to be addressed. Assessing the impact of VP’s on learning and knowledge retention is an area that has been lacking in current VP research (Botezatu et al. 2010). Most published research related to VPs has been about their applications onto a course or about student evaluation of a course that uses VPs. A review done by Cook (2006) stated that out of the 232 papers that was published on the topic of ‘‘simulation in medical education’’ in 2005, only 13 were related to the assessment of simulation as a teaching tool (Cook 2006). A more recent systemic review by the same group identified only 30 randomised trials out of 2,705 publications on internet learning (Cook et al. 2011). Acute pain management (APM) teaching at CUHK The Department of Anaesthesia and Intensive Care at CUHK provides an Acute Pain Service (APS) in its main clinical teaching center, the Prince of Wales Hospital. The

123

Effectiveness of virtual patients

purpose of the service is to supervise the running of (a) intravenous patient controlled analgesia (IV PCA) morphine pumps, (b) local anesthesia infusion pumps such as continuous epidurals and (c) transition to oral pain medications which are used on the surgical wards to treat more severe post-operative pain (Hung et al. 2002). Both nursing and junior medical staff working on the surgical wards should be familiar with these hi-tech pumps and their protocols as malfunction can lead to morphine overdose, cardiovascular collapse and potentially death. Since the 1990s teaching of APM had been limited at CUHK to tutorials and attachment to morning APS ward rounds. However, following the success of the anesthesia FACS-VP cases there had been calls from the students for additional FACS cases on APM. The conceptual basis for an APM teaching initiative was as follows: 1.

2. 3. 4. 5.

6.

The identified problem was that, as future surgical house officers, the students would attend patients where the APM pumps were being used and these pumps may not be functioning properly or complications from their use may arise. The needs assessment of the learner was to be familiar with the how the APM pumps worked and their protocols on graduation. The education goal was that every student knew how the pumps were set up, used and monitored on graduation. The instruction strategy had been tutorials and clinical attachments to ward rounds. New VP teaching materials were to be added. Implementation of the new VP teaching materials involved authoring three new FACSVPs that covered (a) setting up an IV PCA morphine pump, (b) managing an IV PCA pump on the surgical ward, and (c) managing a continuous epidural infusion on the surgical ward (Table 2). Similar content was added to the SL-VP as a new chapter on post-operative pain management. These VPs were completed in the late summer of 2009 and introduced after the start of academic year 2009/2010 (Leung et al. 2011a). The two VP interventions provided an opportunity to evaluate the impact of VPs on students’ learning. The students were already receiving teaching from the original six anesthesia FACS-VP cases and SL-VP, thus the APM-VP cases were not new teaching interventions. However, by selectively exposing different cohorts of students to this new teaching content using a 2 9 2 cross over design the aim was to study the effects of exposure to the two types of VP, interactive and longitudinal, on student learning. Thus, the APM-VPs were used as research tools to study how introduction of VP learning materials affected learning within the broader anesthesia syllabus. By using purpose written examination questions the influence of these test items on learning could be isolated and thus evaluated. These were delivered by two end of module and one final year examinations. Mapping of these questions to learning objectives is shown (Table 2).

Methods Teaching intervention During the academic years of 2009/2010 and 2010/2011, which ran from July to May of the following year, final year students on a 5-year medical course at the CUHK were assigned to one of 16 groups of 7–10 students by the Medical Faculty. The students rotated through four separate medical or surgical modules (i.e. two of each), each 10-weeks in

123

J. Y. C. Leung et al. Table 2 Key learning objectives and matched examination questions VP exercise and metrics

Key learning objectives

Mapped examination questions

FACS case A 14 decision steps 79 webpages

Choice of IV PCA for post op pain relief Types of IV PCA pumps used in the hospital Programming functions of the IV PCA pump IV PCA prescriptions and record sheets Setting up IV PCA in the recovery room

2-MEQs, 2-MCQs 8-MCQsa MEQ

FACS case B 16 decision steps 76 webpages

The role of the APS nurse Education of the patient on IV PCA use Bedside assessment of pain management IV PCA pump settings and their adjustments Common side effects and morphine overdose Weaning IV PCA and starting oral analgesics

2-MEQs, 1-MCQ 1-MEQ 3-MEQs, 8-MCQsa 3-MEQs, 8-MCQsa 2-MCQsa

FACS case C 12 decision steps 66 webpages

Pros and cons of epidural analgesia Continuous infusion epidural pumps Epidural prescriptions and record sheets Education of the patient on IV PCA use Signs of a high epidural block and treatment Management of epidural catheter dislodgement Acute cord compression and urgent MRI scan

2-MEQs 2-MEQs, 1-MCQ 1-MEQ 2-MEQs, 2-MCQsa 1-MEQ

SL-VP chapter 21 webpages 2 quizzes 35 pictures

How IV PCA is assessed Patient education on use of IV PCA Pump settings and what they mean Alternative oral analgesia regimens Comparison to epidural infusions

3-MEQs, 8-MCQsa 2-MEQs, 1-MCQ 3-MEQsa, 8-MCQsa 7-MCQsa 2-MEQs, 2-MCQs

Taken from Tables 3 and 4 for the three APM-FACS cases and the new chapter on APM in the SL-VP APS acute pain service; IV PCA intravenous patient controlled analgesia a

Some MCQs and MEQs covered more than one mapped learning objective

duration. Each group of students spent two-weeks of this time attached to an anesthesia module (i.e. there were 4 groups scheduled per 10-week module), where they were assigned to have access to one of the four combinations of APM-VP learning materials as part of 2 9 2 crossover design study: (A) No FACS and No SL-VP; (B) Both FACS and SL-VP; (C) No FACS, but with SL-VP; and (D) FACS, but No SL-VP. An outline of the study protocol is provided (Fig. 1). Ethical considerations The students were not informed of this intervention and formal ethical approval was waivered by the Dean of Education of the CUHK. The examinations scores relating to APM did not affect passing the anesthesia module which was based primarily on the MCQ result and other course work. Evaluation of the APM-VP effect Three written examinations that contained components designed to test the students learnt knowledge regarding APM, or act as high quality controls (i.e. general anesthesia course topics taught by both tutorials and VPs), were used to assess this one specific area of learning. These three examinations included:

123

Effectiveness of virtual patients

Fig. 1 Outline of study protocol. Exposure to the four virtual patient (VP) intervention groups A–D, where A is no exposure, B is both VPs, C is storyline only and D is FACS only. Academic calendar is shown with distribution of anesthesia courses within each 10-week surgery module. Vertical arrows indicate when course examinations were sat. AM, acute medicine course which occupied 2 weeks

1.

2.

A 60 item multiple choice question (MCQ) paper which contained five to nine APM (test) items (Table 3). These MCQ items had been developed over several years by annual review of their performance statistics (i.e. item difficulty and discrimination index) using a MCQ database and management software called IDEAL-HK, which had been written by the Office of Educational Services of the CUHK (Leung 2011c). The item difficulty represented the proportion of students that choose the correct answer. For APM MCQ items used in year 2010/2011 the mean difficulty was 74 ± 18 % and range 46–95 % (n = 19 MCQ items). The discrimination index represented the proportion in the top 27 % minus the bottom 27 %, with respect to total MCQ mark, that chose the correct answer for a particular item. For year 2010/2011 the index for APM MCQ items was ?18 ± 15 % and range -22 to 40 %. Three different examination papers were used and these were rotated between the sixteen student groups. MCQ paper 2 had the most APM items and was purposely used the most (i.e. 7-times). Student performance was scored as percentage of correct answers over the total number of items. Scores for both the APM (test) items and the remaining general anesthesia (control) were calculated for each student. [In academic year 2009/2010 a slightly different analysis was performed. Data from only one of three MCQ papers was used (i.e. paper 2 used 8-times) and nine APM (test) items and eleven general anesthesia (control) items were utilized (Leung et al. 2010)]. An accompanying short 20-min modified essay question (MEQ) paper was administered with the end of module MCQ paper. The paper contained two sections that were based around a perioperative management issue such as diabetes or chronic airways disease. The first section involved a patient history with preoperative assessment questions that acted as controls and the other section had two to three questions on post-operative pain management (or APM) that acted as test items. Four different test papers were written that were rotated between the sixteen student groups (Table 4).

123

J. Y. C. Leung et al. Table 3 Acute pain management topics tested by MCQ papers Paper 1

Paper 2

Paper 3

Assessment of APS nurse



1



Choice of appropriate pain management



1

1

IV PCA pump settings and management

2

4

2

Assessment of acute post-operative pain



1

1

Management of continuous epidurals





1

Complications and side effects

1

1



Physiology of pain management

1

1



4

9

5

Subject matter of MCQ items

Total number of MCQ test items in paper APM-VP groupings A. None

2

2



B. Both VPs

1

2

1

C. Only FACS

1

2

1

D. Only SL-VP



1

3

4

7

5

Times paper was used

Number of questions or test items on each topic is shown. Each MCQ item had 5 possible answers (A–E). APS Acute Pain Service, IV PCA intravenous patient controlled (morphine) analgesia. Below number of times each MCQ paper was administered to each of the four VP intervention groups A–D

3.

A final year surgery end of year examination MEQ paper consisting of six questions, one of which was set by the anaesthesia course and followed the same format as the end of module MEQ (Table 4). Seven items were used as controls to test knowledge on perioperative management issues and three as test items for APM learning.

The MCQ and MEQ examination papers [i.e. examinations (1) and (2)] were administered on the last day of each 2 week anesthesia attachment or as part of the final year surgery examination [i.e. examination (3)] held in the May. The MEQ examination papers [i.e. examination (2) and (3)] were remarked by one of the authors (JYL) and given a score based on the keywords and concepts mentioned within the written answers. Course evaluation questionnaire A student–teacher course questionnaire was administered at the end of each two-week anesthesia course. Part of this evaluation involved grading (i.e. Likert scale 1–6) the FACS and the SL-VP, and in particular the methods used on the course to teach APM, which included a lecture, ward rounds with the APS team and access to the two VP websites. Preliminary study and refinements During the academic year of 2009/2010 an initial study was performed using the 2 9 2 crossover design for APM-VP exposure (Leung et al. 2011a; Leung et al. 2011b). The fourstudent groups in the first surgical module received (A) no exposure [July to September], the second module (B) both VPs [September to November], the third module (C) SL-VP

123

Effectiveness of virtual patients Table 4 Outline of modified essay questions MEQ

Case

Co-morbidities

General anesthesia section

Acute pain management section

End of module examination 1

Male 55-years Bowel surgery

Non-insulin dependent diabetic

Peri-operative blood glucose control Pre-operative investigations

Role of the pain nurse in setting up IV PCA Interpretation of clinical data used to assess IV PCA and pump settings

2

Male 65-years Gastrectomy

Smoker Chronic bronchitis

Optimisation of chest condition Evaluation for postoperative intensive care admission

Indications for using a continuous thoracic epidural infusion for pain relief Assessment of infusion— low BP and pulse

3

Female 65-years Hip replacement

Arthritis Hypertensive Recent ACS

Peri-operative management of hypertension and ACS Peri-operative management of patient on steroid medications

Assessment of IV PCA by pain nurse Interpretation of clinical data used to assess IV PCA and pump settings

4

Female 45-years Radical hysterectomy and lymph node dissection

On thyroxine Loud systolic heart murmur

Pre-operative investigations for thyroid disease Pre-operative investigations for a heart murmur

Educating patient about epidural analgesia Actions if epidural catheter displaced and pain management needs continuation

Previously fit and healthy

Airway assessment Adjuncts to intubation Management of massive bleeding intraoperatively Premedications

Choice of method of pain management Outline how IV PCA is set up Bed side evaluation by pain nurse

Final surgery exam Male 35-years Amputation of left leg for malignant tumour

ACS acute coronary syndrome, IV PCA intravenous patient controlled analgesia

only [December to February] and the fourth module (D) FACS-VP only [February to April]. Writing of the new APM-VPs was not completed until after the start of the first surgical module, and this dictated the order of exposure to the APM-VPs within the year. The preliminary results from 2009/2010 showed that FACS-VP was effective, but the study was limited because the exposure to APM-VPs had not been randomized and this may have biased the results in favor of FACS-VP. Intervention (D) students who took the surgery module last sat the final year examination within a few weeks compared to no exposure (A) students who sat the final examination almost 9 months later. Thus, the study was repeated the following academic year of 2010/2011, but with (1) better randomization of APM-VP exposure implemented with the four APM-VP teaching interventions A–D distributed evenly amongst the four anesthesia groups within each surgical module cycle. (2) Results from all three MCQ papers were analyzed and items revised when necessary. (3) The four end of module 20-min MEQ papers (Table 2) were revised after academic year 2009/2010 to eliminate sections that were found not to provide discriminative answers.

123

J. Y. C. Leung et al.

Student login times Each student logged into the FACS website using a unique student identification number which enabled the TLRC to record the duration and number of times each student used the website. Statistical analysis Data are presented as numerical values or percentages which are compared using Mann– Whitney test. For the 2 9 2 crossover comparisons of student examination results a multivariate analysis of variance (MANOVA) was performed which enabled the assessment of the interactions between the two types of VP. MANOVA determined the effects on APM examination scores of (1) FACS, (2) SL-VP and (3) both VPs working together without the need to use interventional groupings A–D. The software programme SPSS, version 15.0 (SPSS Inc., Chicago, IL, US) was used. P \ 0.05 was considered as significant.

Results Results from year 2010/2011 Data from the previous academic year 2009/2010 have been presented elsewhere and are mentioned to enhance understanding of the background to this study (Leung 2011c). During the year 2010/2011 a total of 130 final medical students divided into 16 groups attended the two-week anesthesia course and completed all three assessment examinations. There were 32–34 students in each APM-VP intervention group A–D (Table 5). End of module MCQ examination There was no effect on MCQ scores from the control items when different exposures to APM-VPs (i.e. intervention groups A–D) were used (Table 5). However, MANOVA detected significant effects on MCQ scores from the APM items when students were exposed to FACS and SL-VP (p \ 0.05). Only FACS had a true effect on MCQ scores (Table 5). [End of module MCQ examination post hoc results FACS (p = 0.005; F = 8.3, df = 1), SL-VP (p = 0.06; F = 3.6, df = 1) and combined (p = 0.18; F = 1.9, df = 1)]. End of module MEQ examination There was no effect on end of module MEQ scores from the control items when different exposures to APM-VPs were used (Table 5). MANOVA detected a highly significant effect on MEQ scores for APM items for students exposed to FACS, but not SL-VP (Table 5). [Post hoc results FACS (p = 0.0001; = 19.3, df = 1), SL-VP (p = 0.5; F = 0.5, df = 1) and combined (p = 0.1; F = 2.7., df = 1)]. Final MEQ examination Similarly, there was no effect on end of year MEQ scores from the control items when different exposures to APM-VPs were used (Table 5). MANOVA showed a significant effect on MEQ scores for APM items for students exposed to FACS, but not SL-VP

123

n = 32

n = 32

D: (FACS)

B: (both)

0.77

Combined

71 (65–77) %

0.18*

0.045*

0.004*

74 (68–87) %

85 (79–87) %

69 (62–75) %

45 (41–49) %

0.39

0.15

0.92

42 (38–46) %

43 (39–48) %

41 (37–45) %

0.10

0.47

0.0001

57 (51–63) %

64 (58–70) %

49 (43–55) %

46 (40–52) %

Test

0.79

0.56

0.49

49 (46–53) %

48 (44–51) %

49 (46–53) %

49 (46–53) %

Control

MEQ-Finals

0.46

0.23

0.01

55 (51–59) %

54 (50–59) %

51 (46–55) %

46 (42–51) %

Test

* Post hoc between groups analysis of MCQ test item data: FACS p = 0.005, SL-VP p = 0.06

Percentage scores [mean (95 % confidence limits)] for the control (general anesthesia) and test (APM) items for both the MCQ and MEQ (modified easy question) in the end of module and final surgery examinations. Data for the four APM-VP treatments groups (A–D) shown. Included results from statistical analysis using MANOVA showing p values for the three interventional effects, FACS, SL-VP and them combined

0.94

0.29

SL-VP

68 (65–71) %

66 (63–69) %

67 (64–70) %

66 (63–69) %

Control

Control

Test

MEQ-module

MCQ-module

FACS

MANOVA (p values)

n = 34

n = 32

A: (none)

Students

C: (SL-VP)

Intervention

Table 5 Examination results for the 4-interventions

Effectiveness of virtual patients

123

J. Y. C. Leung et al. Table 6 Login times for 4-interventions APM-VP group

SL-VP site

Anesthesia FACS

APM-FACS

Total login time

No APM

24 (0–179)

198 (144–332)

No access

264 (152–456)

Both VPs

7 (0–144)

230 (173–304)

134 (72–193)

377 (364–450)

SL-VP only

47 (7–139)

262 (159–399)

No access

300 (239–502)

FACS only

34 (0–163)

236 (164–306)

154 (107–212)

422 (274–530)

Median (quartiles) durations shown in minutes

(Table 5). [Post hoc results FACS (p = 0.01, F = 6.9, df = 1), SL-VP (p = 0.2; F = 1.4, df = 10) and combined (p = 0.5; F = 0.6, df = 1)]. Login times Login data for the 130 students were compiled from the TLRC records of year 2010/2011. Time logged on times to the FACS website ranged (median (quartiles)) from 312 (190–430) min for all 9 FACS cases and 34 (0–148) min for the SL-VP (p \ 0.0001) (Table 6). Just over 50 % of students logged onto and used the SL-VP site whereas almost every student used the FACS site with the lower 10th percentile of students logged on for 98 min (Fig. 2). When made available students logged onto (1) the setting up of IV PCA FACS for 53 (34–74) min, (2) using IV PCA on the ward FACS for 37 (27–59) min and (3) using epidural analgesia FACS for 41 (20–61) min. A similar time break down for use of the APM SL-VP could not be compiled because only the login time for using the whole site was recorded. Whereas each FACS was a separate software item on the system, the SL-VP was a complete package and the login clock did not differentiate between chapters. Student–teacher questionnaire Nearly all the 130 students (n = 126–129) students completed the student–teacher evaluation form. Evaluation scores for each group shown (i.e. n = 16 groups). The overall

Fig. 2 Percentile plot comparing login times

123

Effectiveness of virtual patients

evaluation score of the anesthesia course (Qu: Overall, I am satisfied with the course) was [median (quartiles)] 5.00 (4.88–5.09) out of 6.00. This evaluation score compared favourably with that from other courses across the CUHK for this item 4.65 (4.22–4.97) (mean from over 3,700 classes), and ranked within the upper 25 % of courses. The class evaluation of FACS (Qu: The use of FACS to teach preoperative assessment is valuable) was 5.19 (4.96–5.70) and SL-VP (Qu: The virtual patient scenario is valuable) was 4.50 (4.00–5.00). In respect to VPs as a tool to teach acute pain management, the class evaluation classroom teaching (Qu: I learnt most about acute pain management from the lecture/ tutorial) was 4.92 (4.74–5.04), pain ward round (Qu: I learnt most … pain ward round) was 4.63 (4.50–4.90), SL-VP (Qu: I learnt most … the virtual patient) was 3.94 (3.79–4.85) and FACS (Qu: I learnt most … the FACS website) was 5.04 (4.35–5.25). The difference in evaluation score between FACS and SL-VP was 0.50 (0.23–1.20). There were no demonstrable differences when exposure to SL-VP and FACS were used as a factor.

Discussion The findings from this study showed that exposure to APM-FACS further improved examination performance and thus learning of a subject already being taught on the anesthesia course, namely APM. Both immediate (i.e. end of module MCQ and MEQ) and delayed (i.e. final year MEQ) effects were shown (Table 5). However, a similar influence on examination performance was not evident following exposure to an additional chapter on APM in the SL-VP (Table 5). The present results reconfirm our previous findings from year 2009/2010 that exposure to FACS cases improved student examination performance (Leung et al. 2010; Leung 2011c). Analysis of student login times to the FACS server revealed that the SL-VP, and presumably its APM content, was only being used to its full potential (i.e. \47 (7–139) min) by approximately half those student with access, whilst the APM-FACS (i.e. \154 (107–212) min) was being used by nearly all students on the course with access and for longer durations of time (Table 6; Fig. 2). To complete the SL-VP one is expected to take 2–3 h. A corresponding diluted effect (i.e. 50 % reduction) on enhanced examination performance was not apparent in the analysis (Table 5), thus it can be concluded that exposure to the SL-VP did not enhance learning of APM. It is also noteworthy that exposure to both VPs reduced examination scores relative to FACS alone in both end of module examinations which is difficult to explain in terms of MCQ papers sat, which were similar, (Table 2) or login times, also similar (Table 6). Whether this result was due to pure statistical variation or to a subtle variation in timing of when the anesthesia course was taken during the 10-week surgery module is hard to say. The order of exposure within the module was consistently A, B, C, D for administrative reasons. Data from student course evaluation questionnaires showed that SL-VP (score 3.94) was less popular than APM-FACS (score 5.04) and this was reflected by shorter student login times for SL-VP (Table 6). In previous years login times for the SL-VP had been longer, usually 2–3 h compared to 30 min for the present study, and this change can be attributed to dropping the compulsory computer session held at the TLRC (Leung et al. 2010). As use of the VP website was not a compulsory, login times were not routinely followed up so this drop in use was found retrospectively. Hence, it is noteworthy that both the preoperative assessment and APM-FACS cases were all well used (i.e. total login time averaged 6 h) indicating that CUHK medical students were selective in their use of web based learning

123

J. Y. C. Leung et al.

materials and FACS was one of the preferred methods of learning on the anesthesia course, unlike SL-VP. In design FACS is a series of learning gates that cannot be passed through without having gained sufficient knowledge to answer the decision steps correctly. The APM FACS cases were constructed to provide relevant knowledge as feedback which affected learning by reinforcing the users knowledge and cognitive understanding of APM. Thus, the improvement in examination performance after using FACS can be explained by this process of knowledge reinforcement. A good working knowledge of how pain pumps were set up, assessed and managed by the APS team was needed to answer the MCQ items and MEQ cases correctly. The MCQ items used for assessment were mostly type II with a vignette that required a management decision (Fig. 3). It can be seen from the example provided that a wide range of knowledge about IV PCA was required to answer the MCQ item correctly. The MEQ cases were also constructed to elicit a similar range of knowledge. Thus, both the MCQ items and MEQ cases were well equipped to map the learning objectives of APM part of the anesthesia course (Table 2) and therefore validated the main findings of the study. From our experience of using FACS cases on the anesthesia course it would be an ideal learning tool for teaching other types of procedural health care algorithms such as advanced cardiac life support in cardiopulmonary resuscitation (CPR) where treatments are based different clinical findings. The branching design of VP platforms that use decision steps and feedback, such as the FACS system, would be ideal to teach and reinforce the often complex networks presenting treatment options used in acute medicine. So far FACS has not been applied to these areas of medical education. Although the FACS platform may be unique to the CUHK and therefore difficult to access, there are other VP platforms of similar design that are more widely available, such as Open Labyrinth and VP Sim. It is difficult to judge the educational role and benefits of using the SL-VP within the CUHK anesthesia course. It does not appear to promote learning that results in better examination scores and only half of CUHK students used the website to full advantage by

Fig. 3 Example of MCQ test item

Acute pain management MCQ item: An otherwise healthy 48-year old woman following a sigmoid colectomy is prescribed IV PCA morphine for pain relief. She is alert and taking sips of water. She is reviewed on day 1 by the Acute Pain Service. They find that: Pain score 6/10 (rest) and 9/10 (activity) 40 deliveries and 100 attempts in 24 h 24 h morphine consumption is 60 mg RR 15 /min, SaO2 95% on oxygen 2 L/min Qu: How should this patient’s pain be managed? A. Continue her IV PCA without changing the settings B. Decrease the settings so less morphine is delivered C. Teach the patient to use the PCA pump correctly D. Check the patency of intravenous delivery system E. Start a non-steroidal anti-inflammatory drug

123

Effectiveness of virtual patients

logging on for sufficient time (i.e. [2–3 h). Designed as a longitudinal rather than an interactive VP it was not intended to promote clinical decision making skills, but rather give a global picture of process of anesthesia. Furthermore, it was meant to fill the gaps in the anesthesia undergraduate syllabus that the busy and ad-hoc nature of clinical attachments to the operating theatres often failed to cover. By reading through the SL-VP these gaps were hopefully made apparent to the student who by reading further about the missed topics would learn their relevance to anesthesia practice, albeit through a virtual patient experience. However, the assessment methods used in the present study did not evaluate this aspect of learning from VPs. A particular problem with undergraduate medical education in Hong Kong is that many CUHK students are English as second language learners and long descriptive narrative in English maybe for many an unpopular feature of learning. Thus, the narrative part of the SL-VP could be made shorter and its pages more attractive to Cantonese speaking students by using more multimedia applications. During clinical sessions (i.e. operating theatre attachments and APS ward rounds) students were not given much opportunity to practice APM and mainly observed. As the FACS cases could be repeatedly used this did provide some opportunity for deliberated practice. However, there was no variation in the cases available and this is something that could be addressed in the future, either by introducing different APM cases or using the randomizer function in the FACS system. The study had some limitations: (1) Student numbers were fixed to 32–34 per intervention group and this may have limited the power of the study. However, the outcomes of the study were similar to that found in the previous year (Leung et al. 2010). (2) Study interventions and evaluation had to fit course design and teaching schedules. Thus, aspects of the study such randomization of the timing of exposure to the different teaching interventions and which examination papers were administered were not ideal (Table 3). Finally (3) only one subject was evaluated APM, although it was used as marker of enhanced learning following the use of the two VPs.

Conclusion The FACS cases were well liked and used by final year medical students on the anesthesia course. The cases enhanced learning of clinical topics such as APM and the management of IV PCA pumps. In particular, FACS enhanced the learning of complex treatment protocols and potentially could be applied to other health care courses to teach clinical management algorithms. The SL-VP was less effective and its role in medical education is far less clear. Acknowledgments The authors wish to thank Judy Wong for her work on the early stages of development of the VPs. This project was supported by a Chinese University of Hong Kong Teaching Development Grant for 2005–2008 (Project ID: 4170251). Ethical Standard The formal ethical approval for this study was waivered by the Dean of Education of The Chinese University of Hong Kong. Conflict of interest The authors report no competing interests.

123

J. Y. C. Leung et al.

References Berwick, D. M., & Finkelstein, J. A. (2010). Preparing medical students for the continual improvement of health and health care: Abraham Flexner and the new ‘‘public interest’’. Acad Med, 85, S56–S65. Bordage, G. (2009). Conceptual frameworks to illuminate and magnify. Medical Education, 43, 312–319. Botezatu, M., Hult, H., Tessma, M., & Fors, U. (2010). Virtual patient simulation for learning and assessment: Superior results in comparison with regular course exams. Medical Teacher, 32, 845–850. Chaikoolvatana, A., & Goodyer, L. (2003). Evaluation of a multimedia case–history simulation program for pharmacy students. American Journal of Pharmaceutical Education, 67, 16. doi:10.5688/ aj670116. Cook, D. A. (2006). Where are we with web-based learning in medical education? Medical Teacher, 28, 594–598. Cook, D. A., Hatala, R., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., et al. (2011). Technologyenhanced simulation for health professions education: A systematic review and meta-analysis. JAMA, 306, 978–988. Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2010). Instructional design variations in internet-based learning for health professions education: A systematic review and meta-analysis. Academic Medicine, 85, 909–922. Cook, D. A., & Triola, M. M. (2009). Virtual patients: A critical literature review and proposed next steps. Medical Education, 43, 303–311. Critchley, L. A., Kumta, S. M., Ware, J., & Wong, J. W. (2009). Web-based formative assessment case studies: Role in a final year medicine two-week anaesthesia course. Anaesthesia and Intensive Care, 37, 637–645. Critchley, L. A., Wong, J., & Leung, J. Y. (2008). Virtual patients and undergraduate anaesthesia teaching. Medical Education, 42, 1120–1121. Ellaway, R., Poulton, T., Fors, U., McGee, J. B., & Albright, S. (2008). Building a virtual patient commons. Medical Teacher, 30, 170–174. Garrett, B. M., & Callear, D. (2001). The value of intelligent multimedia simulation for teaching clinical decision-making skills. Nurse Education Today, 21, 382–390. Haag, M., Singer, R., Bauch, M., Heid, J., Hess, F., & Leven, F. J. (2007). Challenges and perspectives of computer-assisted instruction in medical education: Lessons learned from seven years of experience with the CAMPUS system. Methods of Information in Medicine, 46, 67–69. Harless, W. G., Drennon, G. G., Marxer, J. J., Root, J. A., Wilson, L. L., & Miller, G. E. (1973). CASE: A natural language computer model. Computers in Biology and Medicine, 3, 227–246. Hung, C. T., Lau, L. L., Chan, C. K., Chow, B., Chui, P. T., Ho, B., et al. (2002). Acute pain services in Hong Kong: Facilities, volume, and quality. Hong Kong Medical Journal, 8, 196–201. Huwendiek, S., De Leng, B. A., Zary, N., Fischer, M. R., Ruiz, J. G., & Ellaway, R. (2009). Towards a typology of virtual patients. Medical Teacher, 31, 743–748. Issenberg, S. B., McGaghie, W. C., Petrusa, E. R., Lee, G. D., & Scalese, R. J. (2005). Features and uses of high-fidelity medical simulations that lead to effective learning: A BEME systematic review. Medical Teacher, 27, 10–28. Kern, D. E., Thomas, P. A., Howard, D. M., & Bass, E. B. (1998). Curriculum development in medical education: A six-step approach. Baltimore: John Hopkins University Press. Kumta, S. M., Tsang, P., Leung, K. S., & Cheng, J. (2003). Fostering critical thinking skills through a webbased tutorial programme for final year medical students: A randomized controlled study. Journal of Educational Multimedia and Hypermedia, 12, 267–273. Leung, J. Y. (2011c) Comparison of two types of virtual patient when teaching acute pain management to final year medical students. Dissertation. The Chinese University of Hong Kong. Leung, J. Y., Critchley, L. A., Yung, A. L., & Kumta, S. M. (2010). A comparison of two types of virtual patient used to teach perioperative care to final year medical students: Interactive case studies versus story line scenario. Medical Education, 44(Suppl. 4), 1. Leung, J. Y., Critchley, L. A., Yung, A. L., & Kumta, S. M. (2011a). Introduction of virtual patients onto a final year anesthesia course: Hong Kong experience. Advances in Medical Education and Practice, 2, 71–83. Leung, J. Y., Critchley, L. A., Yung, A. L., & Kumta, S. M. (2011b). Identifying the value of interactive e-learning through responses from final year medical students. Medical Education, 45(Suppl. 3), 1–15.

123

Effectiveness of virtual patients Littlefield, J. H., Demps, E. L., Keiser, K., Chatterjee, L., Yuan, C. H., & Hargreaves, K. M. (2003). A multimedia patient simulation for teaching and assessing endodontic diagnosis. Journal of Dental Education, 67, 669–677. OpenLabyrinth v3.2. (n.d.). http://openlabyrinth.ca/VPsim/. Retrieved 2 Sept 2014. VP-Sim. Lab for Educational Technology. (n.d.). http://vpsim.pitt.edu/shell/. Retrieved 2 Sept 2014. Zary, N., Johnson, G., Boberg, J., & Fors, U. G. (2006). Development, implementation and pilot evaluation of a Web-based Virtual Patient Case Simulation environment—Web-SP. BMC Medical Education, 6, 10. doi:10.1186/s14726920610.

123