ventional web site (WWW) with similar course material but without special learning tools. Context: ... Conclusions: Based on our results, web-based learning seems to be more effective when students .... and could participate in discussion at their own pace. Course B ..... to create the courses to be able to present a detailed ...
International Journal of Medical Informatics (2006) 75, 156—162
Learning outcomes in medical informatics: Comparison of a WebCT course with ordinary web site learning material Kalle Romanov a,∗, Anne Nevgi b a
University of Helsinki, Research and Development Unit for Medical Education, P.O. Box 63, FIN 00014, Helsinki, Finland b University of Helsinki, Department of Education, P.O. Box 9, FIN 00014, Helsinki, Finland Received 13 June 2005 ; received in revised form 14 June 2005; accepted 14 June 2005 KEYWORDS Education, distance; Education, medical/* methods; Computer communication networks; Internet; Medical informatics
Summary Objective: The purpose of this study is to compare whether students’ learning outcomes would be better in a designed learning environment (WebCT) than in a conventional web site (WWW) with similar course material but without special learning tools. Context: Third-year medical students in an introductory course on medical informatics at the University of Helsinki, Finland. Methods: Students were randomly assigned to a WebCT group (n = 39) and a WWW group (n = 46). The students in the WebCT group utilized the course material in general discussion groups, special discussions about lectures, quizzes and students’ own notes. The WWW group had access only to the course material. The learning outcome was assessed by administering an on-line examination and the learning experience of the students was assessed by an on-line quiz. Results: The course grade was significantly higher in the WebCT group as compared to the WWW group. This finding was more prominent among females. The students of the WebCT group also experienced significantly more improvement in collaboration with the use of computers than the students in the WWW group. Conclusions: Based on our results, web-based learning seems to be more effective when students are provided with specially designed learning tools. © 2005 Elsevier Ireland Ltd. All rights reserved.
1. Introduction
∗
Corresponding author. Tel.: +358 9 191 25052; fax: +358 9 191 25139. E-mail address: kalle.romanov@helsinki.fi (K. Romanov).
Applications of computer-assisted instruction (CAI) are increasingly utilized in medical schools, especially with multimedia (CD-ROM) software. These applications can be used as a method to deliver
1386-5056/$ — see front matter © 2005 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2005.06.004
Learning outcomes in medical informatics educational material, which can also provide the students with automatic feed-back of their learning achievements. Web-based instruction is a further evolution of CAI. The networked technology (Internet, Intranet) has made it possible to create web-based materials, and a student can study these materials flexibly in terms of time, space and pace. Compared with the CD-ROM, the networked technology also enhances learning by offering improved communication possibilities, such as the asynchronous and synchronous discussion forums and electronic mail [1,2]. Whereas there are many different solutions for distributing learning materials through the network, the easiest and simplest design is to deliver to the students the course or lecture material in the form of text files and visual presentations (e.g. PowerPoint) using the world wide web (WWW) technology. A more advanced form is to deliver the course material in a hypertext form on the website of the course [3]. A marked increase in the number of computer programs for CAI in the medical sciences has occurred simultaneously with the increased use of computers [4]. However, in recent reviews of the use of information and communication technology (ICT) in instruction, CAI was not found to be superior to traditional methods in terms of gains in learning or learners’ satisfaction [5,6]. On the other hand, in recent decades the constructive learning theory and especially the socio-constructive theory of learning, have emphasized that the learner should be seen as active, intentional and as a participant in small-group activities, an individual who reflects on his/her own assumptions and thought processes and thereby thus constructs knowledge and creates new concepts with other students [7,8]. According to the literature, it is not recommended to only offer students lecture notes and learning materials on a course website, but instead to construct a more interactive virtual learning environment using the facilities of computermediated communication (CMC) in order to help students to become self-directed and active learners [8]. The advantage of collaborative learning has been supported by research, and learning in collaborative settings has led to a deeper-level learning and an improvement in critical thinking skills [9—11]. This has also led in the medical curriculum to the construction of interactive wwwenvironments that enabled the students to experience group interaction and collaborative activities. In France, as a part of the French Virtual Medical University project, the Clinical reasoning learning (CRL) has been developed in order to offer students an interactive, computer-supported
157 collaborative learning environment to improve their clinical reasoning and problem-solving skills (11). These applications are even more attractive, since the potential for web-based education that incorporates collaborative learning could be regarded as a promise of better learning outcomes [9,12]. Few studies compare the learning outcome in a course using only a conventional web site on a course in a www-based learning environment. However, many studies confirm that dialogue, communication and especially asynchronous discussion affect students’ learning outcomes positively [12]. Jung et al. (2002) conducted a study comparing the effect of three types of interaction on learning achievement: satisfaction, participation, and attitude toward learning in a web-based instruction (WBI) environment. Undergraduate students were randomly divided into three groups: (i) academic interaction group, (ii) collaborative interaction group, and (iii) social interaction group. Students in all groups could participate into electronic in an discussion. However, in the academic group, the students were not given encouragement and the instruction was content-centred. In the collaborative group, students were organised to study and share ideas in a web-based learning environment, and in the social interaction group, they were encouraged by an instructor. The results revealed that the social interaction group outperformed the other groups, which indicated that the type of interaction in learning lead to the different learning outcomes [13]. Khalifa and Lam [3] studied the effect of the design of web-based learning environment on the learning outcome and learning process. They distinguished two different types of web-based learning environments according to the design of website: distributed passive learning (DPL), and distributed interactive learning (DIL). In the DPL environment, the web was mainly used to deliver the learning material in the form of text files and presentations slides (e.g. PowerPoint); in the DIL environment, the learning material was organized in hypertext design to enable the learners to explore the learning material freely at their own pace and to interact with the instructor and other learners. Students were divided into two groups. Both groups had access to both sites alternatively to prepare for different lectures. The results revealed that students in both groups always performed better when preparing for a lecture using the DIL environment, thus indicating that the hypertext-based learning environment supported their learning process and lead to a better learning outcome. The interaction with peers and the instructor was not examined but the authors
158 expected that the DIL environments also enabling learners to interact with peers and the instructor will be even more effective and lead to a better learning outcome [3]. Students’ experiences have been compared in two different web-based courses by one of the authors [14]. Course A was organised as a free and open learning model, in which the students could begin their studies at any time and could participate in discussion at their own pace. Course B was organized according to a strict schedule in which participation in discussion was compulsory. These two groups had very different student learning outcomes. Students in Course A performed poorly in their studies as compared to the students in Course B. Whereas the students in Course B could discuss with their peers to get support for their studies, the students in Course A complained of loneliness and isolation. Hudson studied the effect of interaction in computer-aidedlearning (CAL) of third-year medical students. The study had a pre/post-test control group design. The students were randomly assigned to three groups of different learning styles and as well as to one control group. The different learning styles were: (i) a didactic style where texts and tutorials were offered in a structured way, (ii) a problem-solving style, where the material was presented in the context of the case and the computer asked appropriate questions in a series of multiple-choice tests, and (iii) a free text style, which allowed the user to respond to open-ended questions by typing in natural language responses. The students in the free text group improved statistically significantly in their ability to apply and retain knowledge statistically significantly compared to the control group, but the former demonstrated no gain when compared to the didactic group. However, the results support the conclusion that the more interactive computer-learning environment improves learning outcomes [15]. The web-based learning environments affect the learning outcome differently according to the pedagogy adopted (traditional instruction versus constructive), the design of course website (linear versus hypertext), and the interaction possibilities provided by the learning platforms (interaction only with the learning material versus interaction with peers and the instructor). Consequently, when utilizing the Internet for educational purposes, a teacher has at least two different designs to choose from: (1) a conventional web site for individual learning without student collaboration, and (2) course material in a designed learning environment with special learning and communication tools. The introductory course, ‘‘Topics in Medical Informatics’’ was intended to give third-year med-
K. Romanov, A. Nevgi ical students an overview of the computer applications in medicine and health care, in the interpretation of data, in decision-making systems, and in subsequent actions. The course was also intended to strengthen their computer dexterity, Medline searching skills and Internet adroitness. One objective of our study was to determine the kind of web-based course in medical informatics that would provide students with the best opportunities for achieving positive learning outcomes. The web-based course environment was created for students using WebCT in two different forms: (i) course material offered through conventional wwwtechnology and by engaging in an e-mail discussion with an instructor for a non-interactive web-based learning, and (ii) course material offered using the WebCT with learning tools for self-directed learning as well as interactive forums with an instructor and peers for a shared learning process. The principal aim of the present study was to compare whether the learning outcome measured by a typical course test would be better in the WebCT with special learning tools and interaction possibilities than in the non-interactive webbased learning environment using the same learning materials. Secondly, we aimed to evaluate if the attitudes and learning experiences of using information technology would be more positive in the WebCT group, in which the students could utilize IT-based learning tools.
2. Materials and methods Ninety-three third-year medical students were enrolled in a medical informatics course at the University of Helsinki in the spring term of 2002. The students were randomly assigned to one of two study groups of equal size, a WebCT group of interactive learning and a WWW group of noninteractive learning. Initially, there were eight who did not complete the course, so a total of 85 students completed the course, 23 (27%) males and 62 (73%) females (Fig. 1). Of the students in the WebCT groups, 31% were males (N = 12), whereas in the WWW group, 24% were males (N = 11). The introductory course, ‘‘Topics in Medical Informatics’’ lasted for 3 weeks, which is equivalent to a half-credit course. The course management tools of the WebCT (version 3.5) enabled the teacher to organize the students for discussion both into small groups intended solely for the group members and into larger groups open to all participants. The teacher managed the learning modules (lectures) by using the content module-tool, in
Learning outcomes in medical informatics
Fig. 1 Schematic design of the study.
which a self-administered quiz can also be easily added for student use. The course content included one introductory small group session (4 h, in groups of 10 students) and web-based learning material containing five modules. The students had around-the-clock access to the multimedia classrooms on campus. A majority of students also had a network connection from their homes to study the course material. A short 15—30 min training of WebCT was given in the small group session. Additionally, the students in WebCT course were provided with practical examples of using learning tools in WebCT with a special emphasis on using group discussion. Futrhermore, they were encouraged twice during the first course week by the teacher to actively participate in the discussion by: (i) commenting or questioning spontaneously about the modules (‘lectures’), or by (ii) responding to open questions (‘‘sparks’’) in the modules addressed to the students by the course teacher (example: ‘‘what ethical issues should be considered when using artificial intelligence in clinical work? give some examples’’). At the beginning of the course, all students entered the course area and completed a questionnaire, labelled ‘‘Test of Information and Communication Technology Skills’’, which measured their computing skills and knowledge of information searching. Both groups had access to the course material in a WebCT environment, but only the students of the WebCT group could use the learning tools of the virtual learning environment. The WebCT group access to the study materials, discussion forums, message system to other students and the teacher as well as their own notes and four self-tests. As mentioned above, the WWW group students had access to the course materials via the WebCT, but the material was actually available in open WWW environment containing five modules (identical to the material in WebCT group, but written in plain HTML using simple navigation tools (back, forth, etc.)). In the WWW group, students could send e-mail questions
159 and comments to the course teacher for purposes of one-to-one communication. Immediately after the course, the students completed two questionnaires, which measured: (1) their experience in using information technology in their studies (‘‘Impact of ICT’’), and (2) the experience of their own learning style, commitment and autonomy (‘‘Learning Experience’’). The ‘‘Impact of ICT’’ consisted of eleven items, and the ‘‘Learning Experience’’ consisted of eight items on a five-point Likert scale. The course performance of students was evaluated by a WebCT test (‘‘Course Test’’) consisting of 38 items with a maximum grade of 77. The items of the Course Test assessed the mastering of the key concepts in the course material by asking questions in various forms of multiple choices, matching and short answers. One example of the items is the following: ‘‘Combine Medline’s ‘Subheading’ with the right concept(s) of the following: (a) title, (b) PubMed, (c) MeSH, (d) Citation Manager’’. The initial ICT Skill Test, the Course Test and two questionnaires for using the ICT were identical in both the WebCT and WWW groups. All questionnaire data were collected as WebCT quizzes, and the material was downloaded to a PC to be analyzed with SPSS software. The differences in course grades between the WebCT group and the WWW group, between high independent and low independent, between high commitment and low commitment, high activity and low activity, and between sexes, were analysed by independent samples and a one-way t-test. The differences between the WebCT group and WWW group with regard to the impact of ICT, and learning experiences, were analysed by one-tailed Mann-Whitney U-test with the Bonferroni adjusted significance level ˛* = 0.10/19 = 0.0052, which corresponds to a 10% significance level without a correction for multiple comparisons.
3. Results 3.1. Background variables The WebCT group and WWW group scored almost equally in the initial test of ICT skill. Only one significant difference was found, ‘‘the use of statistical software’’, which was more frequently used by students in the WWW group (25% versus 8%, p < 0.05). The WebCT and WWW groups showed no significant differences in the students’ self-assessed commitment (high/low), in their activity with interaction (high/low), or in their autonomy (high/low) during the course (data not shown).
160
K. Romanov, A. Nevgi
3.2. Learning outcome The grade of the Course Test was the dependent variable of the present study. The students’ course grades varied between 14 and 69 with the mean being 49.8, and the median 49. Students in the WebCT group performed better than those in the WWW group (53.7; S.D. 11.4 versus 49.7; S.D. = 10.1) (the one-way t-test, p < 0.03) (Table 1). There was a non-significant difference in the course grade between men and women; among women, differences emerged between the grade in the WebCT and the WWW groups, but among men, these findings were non-significant. No significant differences were observed when the mean of the course grade was compared using dichotomous motivational variables, a student’s self-assessed commitment, or an activity involving interaction and autonomy (Table 1).
Table 1 Course grade by group and background variables
Course grade WWW (S.D.) WebCT (S.D.) Independence High (S.D.) Low (S.D.) Commitment High (S.D.) Low (S.D.) Activity High (S.D.) Low (S.D.) Sex Male (S.D.) Female (S.D.) a b
3.3. The experience in using information technology in learning At the end of the course, all students responded to the questions of scales ‘‘Impact of ICT’’ and Table 2
p-value
49.7 (10.1) 53.7 (11.4)
46 39
