To the best of our knowledge, there has been no attempt to apply the .... EAs, we first embarked on creating an online tutorial using Adobe. Flash, to teach IL to ...
Using Affective Embodied Agents in Information Literacy Education Yan Ru Guo, Dion Hoe-Lian Goh, Brendan Luyt Wee Kim Wee School of Communication and Information Nanyang Technological University 31 Nanyang Link, Singapore 637718 {w120030, ashlgoh, brendan}@ntu.edu.sg
ABSTRACT This study aims to evaluate the impact of affective embodied agents (EAs) on students’ learning performance in an online tutorial that teaches academic information seeking skills. A hundred and twenty tertiary students from two major universities participated in the between-subjects experiment. The results suggested that the use of affective EAs significantly increased students’ learning motivation and enjoyment, compared to neutral-EAs or text-only conditions. However, there were no significant differences in knowledge retention between the three groups. This study paves the way for a better understanding of embedding affective EAs in online information literacy (IL) education. Furthermore, the improvement in students’ learning motivation and enjoyment can serve as a basis for future research in this context.
Categories and Subject Descriptors J.0 [Computer Applications]: General. K.3.1 [Computers and Education]: Computer Uses in Education - Computer-assisted instruction (CAI)
General Terms Measurement, Performance, Design, Reliability, Experimentation, Human Factors, Theory, Verification.
Keywords Affect, affective agents, embodied agents, emotions, enjoyment, information literacy, information seeking, Information Search Process (ISP), knowledge retention, motivation
1. INTRODUCTION In the information society, it has become commonplace to accept that IL is a basic skill that people should equip themselves with. With the proliferation and accessibility of information made possible by the Internet, there is an increasing need for IL in order to effectively find, use, and evaluate information to meet specific needs. The fast speed of communication and easy access to information brings about the issue of information accuracy and reliability, where people need to carefully assess the varying quality of content before putting it into use [1]. Therefore, the ability to seek, acquire, navigate and evaluate information is
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Conference’14, Month 9, 2014, London, UK. Copyright 2014 IEEE978-1-4799-5569-5/14/$31.00.
critical to knowledge workers, students and the general public alike, as such skills can help people navigate vast amounts of information, and effectively make use of content that is relevant. At the individual level, IL has been progressively recognized for its value not only in academia, but also in everyday life, as such skills are indispensable for obtaining new information and constructing new knowledge. At the societal level, IL has been recognized as central to the practice of democracy and citizenship, and to the mission of developing lifelong learners [2]. There is today a growing consensus on the need for IL and a sense of urgency about its implementation [3]. IL education has become the shared responsibility of all educators and information providers. The Association of College and Research Libraries identified six tasks that an information literate person is able to do, namely, to determine the extent of information needed, to access the needed information effectively and efficiently, to evaluate information and its sources critically, to incorporate selected information into one’s knowledge base, to use information effectively to accomplish a specific purpose, and to understand the economic, legal, and social issues surrounding the use of information, and access and use information ethically and legally. Among the six tasks, the present study will focus on the aspect of accessing the needed information effectively and efficiently, in other words, information seeking. Information seeking refers to the purposive and intentional effort to acquire information, in order to satisfy some need, and it is a basic activity in which everyone participates on regular basis [4]. The need for IL education is especially important for university students, as they encounter situations where they need to search for information from multiple sources for various knowledgeintensive tasks more frequently. Librarians, among all professionals, have always been the ones that teach students and the general public about IL skills. However, it has been argued that people who grow up with digital technologies are unwilling to initiate interaction with librarians when searching for academic resources. They prefer to use email, social media and search engines instead, despite the uncertain quality and reliability of such information [5]. The above problem does not pertain only to IL education, but to education in general. The differences in the expectations between students and their teachers had already existed in 1967, when [6] pointed out that the generation gap is a fundamental gap between “a generation bred on the book and a generation bred on the tube and related forms of electronic communication” (p. 21). This observation is truer today than it was in 1967 as the young generation of today grows up with digital technologies, which have become their primary means of living and communicating. Their pervasive use of digital technologies has led them to expect
all forms of communication, including learning, to involve immersive experiences that engage all their senses. This puts an increasing challenge on educators about how to educate young people in today’s classrooms. At the same time, it also presents new opportunities to engage the youth. It has been observed that in conventional learning environments, expert tutors pay as much attention and spend as much time to help students achieve affective and emotional goals when tutoring, as they do to help them achieve of cognitive and informational ones [7]. Because of the important role that emotions play in learning, researchers have, over the last few years, attempted to factor in the learners’ affective states when designing educational systems [8, 9]. This echoes recent research in affective computing, where scholars have called for the design of systems and devices that can recognize, interpret, process, and stimulate human affect, as well as express affects [10]. One of the common ways to achieve this is the use of EAs in interface design. In the computing literature, the term “agent” refers to an autonomous computer program that can “act” on its own [11]. An EA thus refers to a life-like agent, i.e., one with a physical face and body [12]. In the last decade, the notion of embodiment exerted a great influence on computer science as researchers found that the “disembodied” approach of logic and abstract symbols in computer systems was unable to achieve desirable outcomes. In view of this, the subsequent shift towards embodiment, including embodiment of agents, has brought great success to computer science [13]. EAs are appraised to have transformed our experience of interacting with computers by explicitly referencing human interactions [14]. It is also believed that EAs’ ability to detect and express affective states is crucial for improving their believability, eliciting emotions in the users, as well as contributing to more entertaining interactions [15]. Accordingly, an affective EA is defined as one that is capable of eliciting certain emotional experiences from users through multiple modalities such as speech, facial expressions and body gestures [13]. The embodied agent REA that was designed by [13] is one example of an affective EA. Studies have found that using EAs can make the interactions between computers and humans more natural [16]. The manipulation of an EA’s affective states can significantly influence learners’ learning motivation and self-efficacy [17]. Another function of affect associated with failed expectations is to direct attention to the proceeding or accompanying events as important lessons to be learnt [18]. Therefore, the timely intervention from instructors could greatly enhance the learning experience and effectiveness. In particular, the use of affective EAs in a pedagogical role such as an instructor, mentor, assistant, and companion, has been found to not only increase students’ learning motivation and perceived self-efficacy as mentioned above, but also help students overcome negative emotions such as boredom or frustration during learning process [19]. However, despite the initial spurt of research on affective EAs in education, a number of research gaps remain. They motivate the present study, and are presented as follows. First, although much research has been conducted on the use of EAs in online educational systems to teach lower order thinking skills, facts, concepts, and procedures, there has been little on higher order thinking skills, such as how to apply, analyze, or evaluate knowledge [20]. Among the limited initiatives to design
educational systems that teach higher order thinking skills, many were not brought to completion, with some projects getting jettisoned, and others prototyped, tested, but finally terminated. IL education involves higher order thinking skills; it teaches important procedural knowledge that synthesizes complex level of thinking and knowledge [21]. In addition, with the proliferation of search engines, social media, and other communication tools, methodologically rigorous research to understand exactly what is involved when searching for information has become more important, yet more confusing than ever. It is now recognized that the information seeking process is not the well-defined, rulesdriven process it has been portrayed to be, but is in fact, a lot messier [22]. Next, most research on EAs has focused on their design and technical implementation, as well as theoretical postulations about their benefits. Little empirical evidence to support their positive impact on students’ learning performance has been reported so far [23, 24]. Additionally, research findings have not conclusively shown that the use of affective EAs can improve learning performance [25], with some reporting positive results and others, negative. For example, [26] reported that students’ attitudinal knowledge can be enhanced when agents have affective facial expressions. On the other hand, [27] reported that the presence of affective EAs did not make any difference in students’ learning outcomes or perceived motivation. In addition, methodologically, there is a lack of research to evaluate affective EAs against neutral ones [9]. Based on the research gaps identified above, the objectives of this study are two-fold: 1. 2.
To design an online tutorial to teach IL skills to university students, by incorporating affective EAs; and To evaluate the impact of the affective EAs on the learning performances of students.
The paper proceeds as follows. First, we will review extant literature on the use of affective EAs in the online educational context. Next, a three-phase research method will be presented. The first phase involved the design of an information seeking tutorial. In the second phase, two sets of EAs, each consisting a teacher EA and a student EA, were created and integrated into variant tutorials: affective-EAs, neutral-EAs (no affective expressions), in addition to the control condition with no EAs (text-only). In third phase, we evaluated the effectiveness of using EAs in online learning, where participants were recruited to view one of the tutorials and complete a survey questionnaire. The paper concludes with a discussion of the results.
2. LITERATURE REVIEW 2.1 Information Literacy Education The notion of IL has been around for many decades. However, only in the 1990s has it caught public attention when American Library Association released the influential “Presidential Committee on Information Literacy” report. This called for more attention to IL [21]. The association suggested that people not only need a knowledge base, but also need skills for exploring it, connecting it to other knowledge bases, and making practical use of it. In other words, people need IL skills. The emphasis and standards of IL have been growing over the decades. For example, an early definition given by [28] referred to IL as “the ability to use techniques and skills for the wide range of information tools as well as primary sources in molding information-solutions” (p. 2). In 2000, the American Library Association extended the
standards to emphasize that an information literate person is able to exert greater control over his/her own learning, which echoes the call to life-long learning [36].
the learners’ emotional and cognitive states are better understood [8]. One of the seminal works on the role of emotions in information seeking is the ISP model.
While most IL definitions apply to the general public, there are additional requirements specifically for students, for whom the quality of the information they find determines the quality of the scholarship they produce. For example, it is expected that information-literate students not only be proficient in reading, thinking, learning, and communicating skills, but also to be motivated and responsible [29]. Due to the widespread use of the Internet among the younger generation, [30] expanded information literacy to include electronic searching and information retrieval skills as well. In addition, students, especially university students, frequently face the task of writing academic papers, and therefore, the ability to search for reliable sources of information in various academic databases and journals, to narrow down the search focus and select an appropriate topic based on available information, to summarize different ideas and synthesize to a paper within limited time, are of great importance to their academic performance. It also involves extending their own knowledge base to create new perspectives.
Kuhlthau’s ISP model takes into consideration the information seeker’s affective states. It predicts that information seekers could experience positive affective states such as confidence and assurance, but also negative affective states such as anxiety and frustration. For these reasons, the ISP model deserves special attention and is highlighted here. According to [18], negative emotions could arise when the results deviated from our expectations. This is the case with information seeking process. Inspired by the personal construct theory, [35] regarded information seeking as a constructive, vigorous process, which involves not only thoughts and actions, but also feelings.
Similarly, [31] likened the collapse of conventional schooling as the fall of Berlin Wall, and cautioned that traditional linear schooling is progressively being replaced by new digital learning environments. This new environment poses new challenges and requires students to learn new skills. They need “the ability to pick out reliable sources from an overwhelming heap of misinformation, to find relevant material amid an array of options, to navigate the shifting ethics of creative commons and intellectual property rights and to present conclusions in a manner that engages modern audiences” [32]. Although most young people today, the so-called digital natives, are adept at sending emails, downloading music or watching movies online, the majority have little idea on how to effectively seek credible sources of information [1]. In a report named Researchers of Tomorrow, which is UK’s largest study on the research behaviors of Generation Y (born between 1982 and 1994) to date, revealed that they are insufficiently trained to take full advantage of the latest opportunities in the digital information environment, such as the rich digital library collections [34]. From both observation and literature, although libraries regularly organize workshops, student turnouts have been very low, perhaps because students want personally-customized, just-in-time assistance [21]. All these developments have created new demands on librarians, but concurrently, new opportunities and new directions are available for librarians to deliver IL instruction. Among many models in use for IL instruction, the ISP stands out as a usercentered approach as it begins with statements of the informational need articulated by information seekers [22]. This model is also one of the few theoretical models that has been empirically verified by numerous studies [21]. The next section will review the original study on ISP model by [35], as well as follow-up studies by others.
2.2 Information Search Process Model The role of emotions and affect has become increasingly important in the research of information seeking behavior [36]. As learning partly involves searching for information, and making sense of the new material encountered, it is believed that the understanding of the learning process will be greatly enhanced if
The ISP model claimed that in the early stages of searching, negative feelings are common, especially when the user has little knowledge of what was available or when the search problem was not clear [35]. However, as the search progresses, and the awareness of the process increases, there is a corresponding improvement in the level of satisfaction and confidence. At the end of the search process, the seeker will feel a sense of relief or satisfaction when the required information is found, or disappointment and anxiety when it was not. The ISP model also asserted that the search process is iterative, and information seekers can repeat previous steps to ensure that the materials retrieved are relevant as their requirements become clearer. A number of studies have shown that the level of anxiety and discomfort will increase up when people seek information, especially for novice information seekers. In fact, [35] regarded anxiety as an integral part of the information seeking process, and such anxiety is often associated with uncertainty and confusion. However, Kuhlthau’s main studies were carried out in the 1990s and early 2000s, when most information seeking happened in offline environments. With the proliferation of online search engines and digital databases, there is a need to investigate the information seeking behaviors in online environment as well. Kuhlthau’s work has inspired others to take on the task. For example, [37] investigated the cognitive, physical and affective searching behaviors of young children on a specific search task when using the Yahoo! search engine. The children experienced positive feelings such as enjoyment and confidence, as well as negative feelings such as confusion and frustration. Further, the study found that the children’s searching process was nonlinear, and they frequently shifted between different searching techniques, looped various hyperlinks, and backtracked within the websites. In another study by [22], participants were required to write a research paper. The post-test suggested that problems are associated not only with the cognitive tasks to search for information, but also the emotional aspects of searching. The study found that students experienced a rise in anxiety level during the information seeking process, but a reduction when the searching was concluded. In summary, the ISP model has been empirically verified in various studies over the course of thirty years, in both online and offline settings, in both library context and other informational setting, and can serve as a holistic model for librarians to understand and teach information seeking skills [22]. For the purpose of this study, the ISP model sheds light on a possible means of easing negative emotions that are associated with information searching. To the best of our knowledge, there has been no attempt to apply the ISP model in an online IL
educational system context. This may be an important way forward in the teaching of IL as online education holds special attractions to young students.
2.3 Affective Embodied Agents Studies have found that people tend to interact with computers in the same way as they do with people [38]. EAs, which are “animated or static entities that are displayed on a computer screen and attempt to interact with users in some way” [9], could make the interaction between humans and computers more interesting and engaging. Facial expressions, speech, and deictic gestures such as pointing with arms or hands and nodding of the head are common ways through which EAs communicate with users. Extending the definition of affective computing [10], we define affective EAs to be those that can detect and express emotion. Explorations into EAs have become a burgeoning research area. By providing visual clues of their operation, well-designed EAs could enrich one’s learning experience, and make it easier to attract people’s attention [13]. Interfaces that used EAs have been praised as the ultimate interface [13]. Reeves and Nass also contributed to the fundamental understanding of EAs in computermediated communication [38]. They contended that our interaction with computers could evoke a sense of intersubjectivity, encouraging us to respond to computers in fundamentally social ways, just like in human-to-human communication [38]. Their argument can also be applied to learners’ interactions with EAs, as learners could interact with EAs as in a natural communication context. A growing number of research papers have been written on the use of affective EAs to promote interactive learning [12, 24, 39]. In one study, a Wakamaru robot was used as an affective EA to provide feedback to users in a speed-reading test [19]. The experiment employed a 2 (no agent vs. agent) × 2 (no praise vs. praise) × 3 (no comparison vs. positive comparison vs. negative comparison) factorial design, with 192 participants. The results showed that participants who received positive feedback not only played significantly more rounds of the test, but also reported significantly higher ratings of intrinsic motivation (an indication of their interests and persistence), than participants who received no positive feedback. In addition, participants who were given the test with the affective EA reported significantly higher levels of motivation than those who were given the test without the EA. As seen from the affective EA design, even a character as rudimentary as a Wakamaru robot succeeded in generating a significant effect on motivation. This is an indication of the potential for incorporating affective EAs as motivational and persuasive tools. Yet other studies have been conducted to investigate the impact of providing other affective expressions, such as politeness and empathy, on students’ motivation and performance. These have provided more evidence that students can benefit from pedagogical characters that express emotions. For example, [40] investigated the effect of politeness in pedagogical agents to enhance communication and students’ learning performance. They created a polite EA in a factory modeling and simulation system to teach students product inventory and management. Seventeen students were divided into two groups with different politeness strategies from the agent: direct and polite. The polite tutor encouraged students by emphasizing on success, and suggested collaboration with students when the student failed. The experimental results suggested that politeness in the EA can lower
students’ perception of difficulty of the learning material and make the presence of a tutor less intimidating. This finding indicates the importance of politeness in designing an EA to facilitate learning experience. More importantly, it has been found that the manipulation of an EA’s affective states in educational systems can significantly influence learning motivation and self-efficacy [17]. In particular, the use of affective EAs in a pedagogical role such as an instructor, mentor, assistant, and companion, has been found to not only increase students’ learning motivation and perceived selfefficacy as mentioned above, but also help students overcome negative emotions such as boredom or frustration during learning process [19, 41]. In [41], an affective-support EA was designed to actively support users to recover from negative emotional states. Behavioral results suggested the use of affective EAs can ease users’ negative emotional states, compared with text-only conditions. The ability of affective EAs to reduce negative emotions is especially important in the context of this study, which investigates the use of affective EAs to ease potential negative emotions arising from information seeking. As shown above, there are two major gaps in current literature on EAs in the online educational context. First, extant literature focuses mainly on the use of affective EAs to teach factual knowledge in the areas of mathematics, history, and science. There is little research done to investigate the use of affective EAs to teach procedural knowledge, which is more difficult to teach than factual knowledge [20]. In our work, we contend that the information seeking process is considered important procedural knowledge that should be taught to students. Second, a majority of the literature has focused on the design and implementation of EAs. There is a lack of empirical evaluation of their effectiveness, especially on the way EAs influence learners’ performance. Therefore, this study addresses the two gaps by designing and developing an online tutorial that teaches the information seeking process, and evaluates students’ learning performance. This is done by comparing affective-EAs with neutral-EAs and text-only condition.
3. METHODOLOGY In order to shed light on the mixed findings on the effectiveness of EAs, we first embarked on creating an online tutorial using Adobe Flash, to teach IL to university students, and to gather user data to fulfil the objectives of this study. Although online learning systems that teach mathematics, history, or computer literacy have been available for some time [12, 23], the notion of affective EAs in online learning is relatively unknown to IL education. In fact, the notion of affective EAs is almost alien IL education research. In addition, [42] argued that although studying existing systems can lead to understanding of the system structure, it is essential to build them, in order to theorize about potential features beyond what has already been achieved. Therefore, the study embarked on creating a new online tutorial that teaches information seeking skills.
3.1 Designing an Information Seeking Tutorial First, we created a 15-minute 2D online tutorial featuring a novice female student learning how to search for academic information for assignments. Given the differences in individual reading and learning speeds, forward and backward buttons are provided in the tutorial for users to control the progress of the tutorial. There are two EAs in the tutorial (see Figure 1), one representing a young female student who is new to academic information seeking, and
another representing an experienced female teacher. This mirrors the approach taken by [43]. A female agent was chosen to represent the teacher as female agents are perceived to exert more influence on students, especially in academic environments [44]. The female teacher in this tutorial instructs learners on how to cope with an academic search situation as a novice. She therefore serves as a “coping model” [39]. The agent’s appearance is the most important design feature as it dictates the learner’s perception of the agent as a virtual social model [9]. In addition, deictic gestures like pointing and nodding are important to convey emotions for the affective agents [12]. Hence the agents were designed in such a way that they have pleasant facial expressions (e.g., smiling, calming, encouraging), and appropriate deictic gestures (e.g., pointing to the website for information).
ISP model, this is the stage in which information seekers might be intimidated by the huge amount of retrieved information, or confused by some contradicting opinions. They might be doubtful about what to do next. Therefore, in the next step (see Figure 3), the teacher tried to give the student confidence by offering some encouraging words, such as “don’t worry”, “don’t panic, it is normal to feel frustrated”, and so on.
Figure 2. Screenshot of Tutorial I (Affective-EAs) Figure 1. Affective teacher and student EA Next, the content employed the ISP model as the structure for the tutorial. Based on the ISP Model, the tutorial was divided into six stages, namely, Task Initiation, Topic Selection, Prefocus Exploration, Focus Formulation, Information Collection, and Search Closure. Each stage is associated with different searching strategies, emotions on the part of the student, as well as instructions from the teacher. At the start, the student was presented with the assignment of writing a literature review on a topic selected from a list. The student became aware of a lack of knowledge, and recognized a need for information. In the literature, this awareness is frequently accompanied by feelings of uncertainty and apprehension [35]. The teacher acted as a learning guide. Next, the student needed to further explore information on the selected topic, and this stage is marked by feelings of confusion, uncertainty, and doubt. The teacher displayed empathic emotions and employed empathic words to encourage the student. Additionally, the teacher also introduced some useful skills to help the student retrieve more relevant information, such as using Boolean operators, and performing backward and forward chaining. Furthermore, the stage of information collection involves the gathering of information related to the topic. As the student now had a clear sense of direction, her confidence level increased accordingly. As the assignment neared completion, the student felt relieved and a sense of satisfaction. The task in Search Closure is to bring an end to the search, and to prepare oneself to write or otherwise use the findings [35]. Given the overview of the tutorial above, we will take the stage of Prefocus Exploration as an example to illustrate how the two EAs were used (see Figure 2). At this stage, the student needed to find a focus for the assignment. To do this, the student entered “knowledge sharing through storytelling” into the Google Scholar search bar, intending to retrieve a list of articles. According to the
Figure 3. Screenshot of Tutorial I (Affective-EAs)
3.2 Experimental Design and Materials The experiment used a between-subjects design, and participants were randomly assigned to one of three conditions: affective-EAs, neutral-EAs, and text-only. This approach is consistent with the work of other studies, e.g., [24, 27]. In the affective-EAs condition (see Figure 2 and Figure 3), the two EAs, one student and one teacher not only maintained a visual presence throughout the tutorial, but also displayed affective expressions, such as frustration and confusion from the student agent, and encouragement and support from the teacher agent. For the neutral-EAs condition (see Figure 4), the two EAs also maintained a visual presence throughout the tutorial; however, they displayed no affective expressions. For these two conditions, conversations and instructions were presented in speech bubbles. For the textonly condition, the two EAs were absent, with only textual
instructions, presented in square dialog boxes.
are important in academic writing because”, and “How many different stages are there in a typical academic information search process?” Subjective Feedback. This section gathers participants’ subjective feedback of the tutorial. Three questions were asked on what they liked and did not like about the online tutorial, and suggestions on how to improve it. Demographic Data. Information on participants’ demographic data such as age, gender, computer experience, perceived computer knowledge, ways of computer usage, and frequency of computer usage was collected.
3.3 Participants and Procedure Figure 4. Neutral teacher and student EAs The posttest survey questionnaire was adapted from existing literature to suit this study’s purpose. It comprised five sections. The first two sections focused on students’ learning motivations and enjoyment. All question items that were used to measure these two constructs were formulated based on extant literature [45, 46]. The questions were measured on a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree). Furthermore, the third section aimed to test the amount of knowledge that students attained from the tutorial. In the penultimate section, participants were asked for their opinions of the tutorial, and for suggestions to improve it. Demographic information was collected in the final section. Each section is further explained below.
Motivation. Motivation is an important component to learning, and it can be a catalyst in learning in order to achieve one’s goals [45]. Designed by [45], the Instructional Material Motivational Survey was used to measure the motivational support provided by instructional programs, which is very suitable in this study. It includes four subconstructs, which includes: attention (e.g., “There is something interesting that got my attention”), relevance (e.g., “The content is relevant to my study”), confidence (e.g., “The tutorial was too difficult”), and satisfaction (e.g., “I really enjoyed learning in the tutorial”). Enjoyment. Enjoyment is the active participation in an experience, and it has been found to be a strong indicator of technology use intention [46]. It can be operationalized at three levels: affective enjoyment, which is a willingness to invest emotionally in the experience (e.g., “I feel emotionally attached to the tutorial”), cognitive enjoyment, which is a willingness to develop skills and solve problems (e.g., “I think this tutorial is a good way of learning information seeking skills”), and behavioral enjoyment, which is a willingness to participate on a kinaesthetic level (e.g., “I was able to easily use the application to accomplish my designated tasks”) [47]. Knowledge Retention. Although it has been pointed out that the instructional effectiveness of online education is often based on its novelty, rather than increased knowledge retention by students, knowledge retention is still important to measure, because it is the most direct and immediate result from the intervention [12]. Therefore, the third section ascertained the amount of knowledge retained by students after watching the tutorial, in the form of multiple choice questions and sentence completion questions. Some example questions include: “Which of the following are the typical information sources for academic information?”, “Citations
First, a pilot study was carried out with 22 university students in order to gather feedback for the online tutorial and to improve the questionnaire. Based on their comments, the sequence of the questions was adjusted, and some ambiguous questions were made clearer. Next, mass emails and flyers were distributed in two major universities in Singapore to recruit participants. One hundred and twenty students (both undergraduate and graduate) participated in the study. Controls were put in place to ensure that the sample was more representative of the population, for example, students from various disciplines from two major universities, including both undergraduates and graduates were recruited. The demographics of the sample are shown in Table 1. Table 1. Demographics of Sample (n=120) Gender Male Female Age Less than 20 21-25 26-30 Educational background Natural Science Arts, humanities, and social science Engineering
n
%
37 83
30.83 69.17
38 79 3
31.67 65.83 2.50
15 92 13
12.50 76.67 10.83
The experiment was conducted in a controlled laboratory setting. Participation in this study was voluntary and anonymous, and no information that could be used to identify individual participant was collected. After signing an online consent form, participants were directed to the study’s website. Upon entering the website, participants were provided with instructions, and told that they were to evaluate an online tutorial. At the completion of the tutorial, participants were asked to complete an online questionnaire (described earlier). The whole study lasted approximately 25 minutes. As a token of appreciation, each participant was given $5 upon completion of the questionnaire.
4. Results Table 2 shows the means and standard deviations of each construct from the questionnaire. Next, results from one-way analyses of variance (ANOVAs) indicated that there were significant differences with respect to Attention [F(2,117)=5.97, p
