Engineering Online: Assessing Innovative ... - Wiley Online Library

10 downloads 39652 Views 116KB Size Report
Jul 1, 2002 - JOSEPH C. PALAIS ... assessment of online graduate engineering courses. This case ... Fiber optics is the focus of the online course described.
Engineering Online: Assessing Innovative Education II. THE CLASS SETUP

SUSAN HAAG College of Engineering and Applied Sciences Arizona State University

JOSEPH C. PALAIS Department of Electrical Engineering Arizona State University

ABSTRACT This paper presents research on student performance outcomes, satisfaction, and retention rates in a fiber optics course at Arizona State University. Although the course typically has been taught by conventional methods for over 20 years, the instructor offered alternative delivery methods, such as the Web, during the Fall 2000 semester. This course was also unique in that it was comprised of diverse groups such as undergraduates and graduates and on- and off-campus students. Although online students were significantly more satisfied with the course, performance outcomes and retention rates were favorable overall. It is likely that course options and convenience aided in student outcomes and course retention.

I. INTRODUCTION In June 1998, the Arizona Board of Regents approved the Tri-University Master of Engineering program. This program combines courses taken from Arizona State University (ASU), Northern Arizona University, and The University of Arizona. A major thrust of the program is the development, delivery, and assessment of online graduate engineering courses. This case study examines processes and best practices in order to continually improve the online Masters of Engineering program during the formative years. The purpose of the study was to improve future online teaching by: 1) eliciting student attitudinal data; 2) assessing the extent to which students migrated to the Web delivery; 3) examining course grades and completion rates; and 4) investigating why students choose certain delivery modes over others. Prior anecdotal data regarding student preferences for convenient course delivery gave impetus to this research. Recent widespread interest in online instruction has been well documented [1, 2]. Institutions have made significant attempts to implement Web-enhanced or Web-delivered courses. Although advocates of this movement support all aspects of distance education, engineering assessment and evaluation literature of program and course success is limited or inconclusive. It is crucial that we assess distance delivery, curricula, and retention, as well as learner performance outcomes, interaction, and satisfaction in order to continually improve the quality of online education. July 2002

Since the inception of the Master of Engineering program, many courses have been developed and offered. This paper discusses the assessment of one of them, a course on fiber optics. The student population was comprised of upper-division undergraduates, graduates, and both on-campus and off- campus learners. Six of the off-campus learners were working full time and received ASU classes remotely via instructional television and over the Web. A courier service transmitted curricular materials and assessment instruments between the remote locations and ASU. Additionally, five full time industry employees were enrolled in this course through National Technological University (NTU). The off-campus students could choose between either video or Web delivery of all class materials. Fiber optics is the focus of the online course described. The course has been traditionally taught since 1979; however, the instructor transformed it for online delivery for the Fall 2000 semester. As a result, alternative delivery modes to meet student needs were made available. The lectures were recorded the preceding summer in a university TV studio. The lectures were recorded simultaneously for video and online delivery. When students arrived on campus, they were given the following options to view course curriculum and materials: TV in the class room, check out videos from the campus video loan library, video live at a company site, or Web delivery. The media used to deliver the Web-based course involved a combination of Blackboard and Tegrity WebLearner. Blackboard enables instructors to provide students with course materials and activities (i.e., announcements, assessment, assignments, course information, course documents, discussion board, external links, and the syllabus). The instructor provided guidance to students through the use of the “Announcements” feature in Blackboard. Students could link to the recorded lectures and laboratory directions, view the course schedule, and experiment with simulations and animations. Homework assignments were handed in at scheduled times. Course exams were also administered at scheduled times. The on-campus students took exams together in a classroom and the distance-learning students took “proctored” exams at company sites. Online learners were required to keep up with all assigned work. The instructor’s prior experience with asynchronous courses strongly indicated that students need structure in terms of assessment and assignments. Without a schedule, students lag behind.

III. METHODS AND DATA ANALYSIS The objective of this study was to assess student perceptions regarding the course options, opportunity for interaction, and aspects of delivery and course content. This particular course was interesting Journal of Engineering Education 285

for study as it had a large enrollment (making results more generally applicable), and it had been taught by conventional methods for over 20 years (so that comparisons with previous offerings could be made). Although a survey was developed for all Master of Engineering students (assessing courses at all three Arizona universities), the instrument was customized for the fiber optics course [3]. This effort was devoted specifically to capture the unique local perceptions. In that way, it was context sensitive. The formative assessment was designed to provide information for the purpose of improving the virtual learning environment. The survey was developed to quantitatively measure student perceptions of the course experience with a series of statements using a five-point Likert scale: 1  Strongly Disagree, 2  Disagree, 3  Neutral, 4  Agree, and 5  Strongly Agree (See Appendix). Survey content validity was established by having the pilot questions reviewed by a team of experts that included engineering faculty, faculty who previously taught online courses, and instructional technology and assessment professionals. Additionally, the survey was piloted during the 1999 Spring and 1999 Fall semesters prior to the Fall 2000 fiber optics administration. Although that survey provided attitudinal measurement, openended questions yielding qualitative data shed further light on areas of assessment interest (such as undergraduate and graduate differences). To assess course efficacy, we used conventional measures of success at the university level such as course grades and course completion. The latter was measured by whether or not the student took the final exam. A. Participants Survey population was identified as all students enrolled in the fiber optics course during the Fall 2000 semester. The population lacked homogeneity in terms of age, academic grade level, ASU student status, and location. Overall, there was a 100 percent response rate as a result of the survey administration process. B. Data Analysis Data were collected and imported into SPSS (Statistical Package for the Social Sciences) and Microsoft Excel for analysis. Students were coded to reveal differing variables (i.e., gender and ethnicity; and undergraduate and graduate status). Nonparametric tests [4] were conducted for special groups to examine differences between groups (i.e., undergraduate and graduate students) within the class due to small sample sizes and because the data were in ordinal form.

T-tests were conducted for course grade point average comparisons. Finally, qualitative data were coded to examine for recurring themes, patterns, and discrepancies between groups.

IV. EMPIRICAL RESULTS OF STUDY There were three primary empirical results of the study. First, despite the fact that the students enrolled in what they believed to be a traditional classroom, 40 percent migrated to the Web delivery. Significant differences existed between students who chose to view the course online and those who did not. More specifically, “Web watchers” (those who migrated to the Web for the recorded lectures and associated class materials) were more satisfied than “non-Web watchers” (those who used video recordings of the lectures to view materials) with technical aspects of the course, student communication, and schedule accommodation. Web watchers were more likely to take another course on the Web. Additionally, there were significant differences between graduate and undergraduate students. Finally, the course retention rate was 95 percent, which is considered to be exceptional for an online course. A. Finding 1 The largest percentage of students selected to receive the course materials online. Although migration to the Web became a popular choice, 60 percent of the students opted to receive course materials by traditional means (e.g., various forms of TV and simply reading the text book). Table 1 below shows the number and percentage of students who chose the various delivery options. The “TV in classroom” and “View video live at company” options require some explanation. The class lectures were pre-recorded simultaneously on videotape and for online viewing. On occasion the instructor did visit the classroom to augment class material. In this case, both classroom and video live students could have direct interaction with him. 1) Web: Forty percent of students experienced the course online. Approximately one third of the undergraduates and one half of the graduates opted for the online version. Qualitative data revealed that student choice depended upon “convenience,” which was a recurring theme among all selection types. Eighteen of the 22 who chose the Web responded with further comments providing reasons for their choice. Web watchers used descriptive words like “flexible” and “easier” and stated that online delivery accommodated their daily schedule. The following excerpts shed further light on this choice: “I work full time and have two kids, so it is more convenient.” Another

Table 1. Selection options. 286

Journal of Engineering Education

July 2002

student added,”I can watch lectures after work without the long drive to school to check out video tapes.” A third male stated that he could watch lectures “whenever he wanted without video tape late fees.” Another reason for choosing the Web concerned review of materials. One female graduate student stated that it was “easy to go back and review lectures again later on.” Another concurred, “It is convenient with no time conflict; I can jump into any sections instantly.” 2) Classroom: Thirty-one percent of the students chose to view video recordings of the lectures in a campus classroom for convenience, comfort, and focus. Respondents frequently stated that they lived near campus, were on campus anyway, or had a class prior to fiber optics in the same building. Students chose to view the course in the classroom because they were more “comfortable” or “familiar” with that environment. Several stated that they chose the classroom environment because it “focused my attention” and “kept me on schedule.” 3) Video: Students also selected the video loan option for convenience. One video loan student stated that it was more convenient to check out videotapes at school because he had “kids at home and therefore could devote more time and attention on campus.” Others cited the convenience of reviewing lectures “via videotapes.” 4) Textbook: Four students simply chose to read the book and did not watch the lecture. Learners who chose to only read the book said it was “convenient” or “easier” for them. One student responded with the following excerpt, “web lectures were the same as the book.” In actuality, material not appearing in the textbook was often added to the lecture. Regardless of the delivery method, student choice depended upon convenience and lifestyle. We compared differences between students who opted for the online version and those who did not. Our first hypothesis is that online students would do as well as those receiving course materials not on the Web (i.e., TV in classroom, check out videotapes, view video live at company, and read the text). This hypothesis has been well supported by the data. Table 2 contains mean scores and mean differences in scores for the whole sample. The table also contains a p-value for each mean difference. The p-value indicates the likelihood of obtaining a difference as large as that observed if it occurred simply from randomness in the data. A low p-value implies that we would probably not observe such a large difference from purely random data and that the difference must

be the result of some systematic effect. Typically we label any difference with a p-value of 0.05 or less as meaningful, that is, statistically significant. 1) Internet delivery and schedule accommodation: The online Master of Engineering program consists of broad objectives such as “the program should accommodate students anywhere and anytime.” For example, courses should be available, accessible, and convenient to students. Therefore, we attempted to assess the fiber optics course with these measures. The fiber optics course options (e.g., alternative methods of delivery) were convenient for learners. Although all students agreed (mean 3.74) that the Internet delivery of this course was technically satisfactory, the Web watchers’ responses were significantly stronger (p-value 0.006). Similarly, while all students felt that the course accommodated their schedule, the Web watchers’ responses were significantly stronger than those who chose traditional methods. Web watchers were more likely to take another course on the Web (see Table 2). 2) Student Communication: Although all student responses revealed that student communication was satisfactory, disaggregated data highlighted the fact that Web watchers responses were significantly stronger than non-Web watchers’ responses (p-value 0.039). 3) Grade Differences: We examined achievement (grade differences) between students who viewed the course online and those who did not. No significant difference was revealed between Web watchers (3.05) and non-Web watchers (3.00), which was consistent with the “no significant difference phenomenon” supported with a plethora of prior research [5, 6, 7, 8]. We further examined the data to assess a difference between Web watchers and non-Web watchers by class level. No significant differences in grades existed between undergraduate Web watchers and non-Web watchers. Similarly, there was no significant difference in grades between graduate Web and non-Web watchers. Therefore, Web watchers did as well as those who chose traditional means to receive course materials. B. Finding 2 Disparities existed between undergraduate and graduate students. Significant differences were revealed in the following areas: course grade, course experience, student-student interaction, and student-instructor interaction. 1) Course grade: Overall, graduates earned significantly higher course grades than undergraduates regardless of how they received

Table 2. Satisfaction differences between web watch and non-web watch. July 2002

Journal of Engineering Education 287

the course materials and curriculum. This result is consistent with grade differences between undergraduate and graduate students in the conventional delivery of this course over the past six years. See Table 3 for course grade details. 2) Course experience: Graduates were significantly more satisfied with the course as a learning experience than undergraduates (p-value of 0.025). Both undergraduates and graduates were consistent in their responses regarding many of the course components. On average, both groups agreed or strongly agreed that the course accommodated their schedule, the course material was accessible, the instructor’s responses were timely, and the course material reflected current engineering practices. Additionally, all students believed the course material provided appropriate fundamental knowledge to understand current developments in their field. 3) Student-student interaction: Undergraduates were satisfied with the amount of student-student interaction (3.84); however, they were significantly more used to student-student interaction (4.03) than graduates (3.46). Students provided more detail regarding interactivity in the qualitative sections of the survey. They offered the following comments about student-student interaction, “Student-student interaction is not important to me” and “I don’t interact with other students unless it is assigned.” 4) Student-instructor interaction: Although there were no significant differences, some students seemed more concerned about the student-instructor interaction than student-student interaction in this course. Although concerns in this area were minimal, the qualitative data revealed that a few students found the distance format less conducive to interaction. This phenomenon is reflected in the following excerpts: “It is harder to communicate with the instructor because of the format.” Because the “instructor is not in the lecture room, it prevents student-instructor interactions.”

C. Finding 3 The course completion rate was high. As stated previously, we used a conventional university measure of success, course completion. Course completion was determined by whether or not the student took the final exam. Consistent with enrollment in prior semesters, there were 58 students enrolled in the beginning of the semester. One student withdrew after earning a low score on a test early in the semester. One student moved and one student failed to take the final exam. Therefore, 55 students completed the course. The course retention/completion rate was 95 percent, which is considered exceptional for an online or traditional course. In addition it is consistent with educational experiments where the instructor makes an effort to accommodate students within the framework of the experiment. The course provided plenty of options and was convenient, flexible, and user friendly.

V. CONCLUSION This study examined an initial effort to offer alternative course delivery options to students in a fiber optics course. The experiment revealed that diverse delivery methods were convenient for learners and that student performance remained the same (6 year grade comparison). Additionally, the high retention rate of 95 percent highlighted the fact that offering multiple delivery methods can become a strength of a course. As a result of multiple data sources, the study revealed three major findings. First, a large percentage of the class migrated to Web delivery. Data supported our first hypothesis that stated that online students would do as well as those receiving course materials not on the Web (i.e., TV in classroom, check out videotapes, view video live at company, and read the text). In other words, there were no significant

Table 3. Course for undergraduate and graduate students: 1995 through 2000. 288

Journal of Engineering Education

July 2002

differences in course grades between the two groups regardless of how the data were examined. However, some differences existed between students who chose to view the course on the Web and those who did not. Web watchers were more satisfied with the course overall and with course convenience and communication. Furthermore, they were more likely to take another course on the Web. Second, there was a significant difference between graduate and undergraduate performance as reflected in course grades; however, this grade difference phenomenon has persisted over the past six years in the fiber optics course. Finally, the course retention rate for all groups (i.e., undergraduates, graduates, ASU and non-ASU students, Web watchers and non-Web watchers) was exceptionally high. It is likely that course diversity and convenience aided in course retention. Student-student and student-instructor interaction are topics of concern for all institutions planning courses and programs online and are important areas for future research. Our study revealed that students were more concerned about interaction with the instructor than with other students. Although this course did not incorporate formal, planned student-instructor interaction, interaction is a critical area for additional research in innovative online courses. We are currently designing a study that examines how interaction affects student performance (course grades), course completion rates, and student attitudes. Our future study will make synchronous chat and online office hours available throughout the semester. Students should be required to attend and post in the chat

July 2002

sessions and this should become a part of their semester grade. We will then be able to compare this study with the future research employing effective interaction components.

REFERENCES [1] Wallace, D.R., and P. Mutooni. 1997. A comparative evaluation of World Wide Web-based and classroom teaching. Journal of Engineering Education. 86(3): 211–219. [2] Whittington, C.D., and Niall Sclater. 1998. Building and testing a virtual university. Computers Education. 30(1, 2): 41–47. [3] Haag, S., and J.C. Palais. 2000. Engineering web-course survey fall 2000. (Web). Arizona State University. , accessed February 2002. [4] Sprinthall, R.C. 1990. Basic Statistical Analysis. Englewood Cliffs, New Jersey: Prentice Hall. [5] Clark, R.E. 1983. Reconsidering research on learning from media. Review of Educational Research. 53(4): 445–459. [6] Clark, R.E. 1994. Media will never influence learning. Educational Technology Research and Development. 42(2): 21–29. [7] Kozma, R.B. 1991. Learning with media. Review of Educational Research. 6(2): 179–211. [8] Kozma, R.B. 1994. Will media influence learning? Reframing the debate. Educational Technology Research and Development. 42(2): 7–19.

Journal of Engineering Education 289

APPENDIX: SURVEY QUESTIONS

AUTHOR BIOGRAPHIES Susan Haag is the Director of Assessment and Evaluation for the College of Engineering and Applied Sciences (CEAS) at Arizona State University. She received her Masters in Educational Psychology emphasizing Instructional Technology and Ph.D. in Policy Studies from Arizona State University. She is currently assisting the college in their preparation for the 2003 ABET site visit. She teaches advanced research methods in Educational Psychology and applies both qualitative and quantitative assessment methods to new initiatives and curricular changes in the CEAS. Additionally, she is a member of the National Science Foundation’s Foundation Coalition. Her academic research focus is on educational reform and evaluation, technology infusion, online education, institutional policy, and Neuro-psychology. Address: College of Engineering and Applied Sciences, Engineering Deans Office, Arizona State University, Tempe, 290

Journal of Engineering Education

AZ, 85287-5006, fax: 480-965-5993, e-mail: susan.haag@asu. edu. Joseph C. Palais is a Professor in the Department of Electrical Engineering at Arizona State University. He received the B.S.E.E. degree from the University of Arizona and the M.S.E. and Ph.D. degrees in Electrical Engineering from the University of Michigan. He is the Associate Chair of the Department and is the author of the textbook, Fiber Optic Communications, Prentice-Hall, Inc., 1998. He has been active in continuing education, receiving the IEEE Educational Activities Board Meritorious Achievement Award in Continuing Education in 1993. Dr. Palais is a Fellow of the IEEE, elected for contributions to university and continuing education, primarily in the area of fiber optic communications. Address: Department of Electrical Engineering, Arizona State University, Tempe, AZ, 85287-5706; fax: 480-965-3837, e-mail: [email protected]. July 2002