Title: Arial, 14 pt

3 downloads 0 Views 294KB Size Report
N/A due to “pass/fail” assessment. Hypothesis B. Students who disengaged with the formal assessment prior to the first test are more likely to have lower grades ...
Reconsidering Retention in MOOCs: the Relevance of Formal Assessment and Pedagogy Oleksandra Skrypnyk University of Southern Australia, Australia Pieter de Vries Delft University of Technology, Netherlands Thieme Hennis Delft University of Technology, Netherlands

Abstract: The motivation to enrol in a MOOC is more diverse than the motivation to enrol for a conventional course. This diversity requires reconsidering what is meant by enrolment, participation, retention and achievement. This paper suggests that retention be measured as relative to student engagement with assessment. With this purpose, three short studies were conducted to gain insights into disengagement from assessment, based on the data gathered in the first five DelftX MOOCs. The empirical part of the paper demonstrates that, if measured in relation to formal assessment, retention rates vary dramatically from course to course. In the analysed cases, fewer learners disengaged from the formal assessment in the course with highest degree of student autonomy, high learning support and scaffolds. We have also observed that the students who start with formal assessment and receive significantly lower grades, are likely to disengage from further assessment.

Introduction Massive Open Online Courses (MOOCs) evolved from the technological affordances that allowed scaling traditional online education. As a result, MOOCs inherited the language and concerns associated with its predecessors, formal online courses. However, there are certain idiosyncrasies that challenge immediate translation of existing evidence from the field of online education into the emergent area of scaled online learning. Due to their open registration policies where any student can join at any time, MOOCs have a higher degree of asynchronicity than other courses taught at distance. Asynchronicity is associated with distance education where students are at times ‘out of sync’ with each other engaging with the content at their convenience, rather than at commonly scheduled times (Mullaney, 2014). MOOCs challenge the boundaries of the traditional cohorts of learners even further by allowing students to also be ‘out of sync’ in relation to the time they can start or stop taking the course. This heightened asynchronicity is exacerbated by the diversity of motivations for enrolment in a MOOC. In a conventional course, students enrol to receive credit and formal recognition of mastery, providing the student cohort with a shared goal. In contrast, students undertaking a MOOC are driven by a variety of goals, from sampling course content, to being interested in a subject, and are not bound to an interest in credentialing. Primary motivation for enrolment and participation can simply be associated with the intellectual challenge and opportunity to socialize with peers around a topic of interest (Eynon, 2014). Despite these differences, in making sense out of MOOCs, researchers and practitioners tend to apply the language that is well defined by the formal educational contexts. De Boer et al. (DeBoer, Ho, Stump, & Breslow, 2013) argue that the concepts of enrolment, participation, curriculum and achievement are not entirely useful to describe scaled online learning, and need to be re-operationalized and re-conceptualized. In line with this position, this paper addresses the refinement of the concept of retention, widely applied to evaluating MOOCs, and controversial within the MOOC academic discourse. The paper discusses the patterns of engagement with formal assessment as associated with the learners who demonstrate commitment to taking the course, but differ in their decisions to sustain that commitment. The discussion is supported by exploratory analysis of the course level data from the first five MOOCs delivered by the Delft University of Technology. Three short studies reported in the paper followed various methodological paths. First, by visualizing descriptive course level information we were able to identify the patterns of disengagement with course’s formal assessment activities, in particular similarities and differences observed across five MOOCs. Second, we further explored the trend observed across all courses - a seeming importance of the first assessment task. Simple

inferential analyses were conducted to identify the relationship between the performance on the first assessment and further engagement in assessment activities, as well as the relationship between the first exam and the pursuit of the certificate of completion. Finally, by referring to evaluation analysis of course’s pedagogical design, conducted as a part of institutional internal evaluation, we have inquired into the differences between the contexts within which disengagement with assessment occurred.

Framing the Retention Debate MOOC retention rates are claimed to be low, under 7.5% of the enrolled participants, which is widely discussed in the academic work (Jordan, 2014). Such interest is not surprising: student retention is an oft-referred measure that determines the societal value ascribed to higher education (Mullaney, 2014). Furthermore, retention is a topical issue within the context of MOOCs predecessor, since online education low retention rates have been identified as a barrier to its expansion (Allen & Seaman, 2013). That being said, MOOC research has been divided around the concept of retention. Some attempt to identify factors associated with drop-out - a so-called ‘failure to complete the course’ (Halawa, Greene & Mitchell in Cress & Delgado-Kloos, 2014), while other challenge the premise that completion defines the success and learning in the course, especially when the registration is free (Cress & Delgado-Kloos, 2014). For example, Liyanagunawardena (Liyanagunawardena, 2014) questions whether a traditional definition of dropout is at all applicable in MOOCs. Conventionally, dropout is a student who commits to participating by paying enrolment and tuition fees in a traditional course, but does not persevere. That does not translate directly into MOOC context, where registration and enrolment are free, and do not require a binding commitment. This paper supports the viewpoint that learners’ success is not defined by the certification in the course, which is generally equalled to completion in MOOCs. However, we argue that retention still may provide valuable insights used to improve MOOC design and align expectations of the MOOC instructors and learners. More specifically, the focus on the students who demonstrate similar commitment but don’t sustain it in the same way, may provide insights into who was more vulnerable to disengagement, as well as time when additional support or re-design may be required to meet the needs of these learners.

Action Phases and Commitment to Learn The ability to self-regulate one’s learning has been repeatedly named as a critical skill for participation in distance education where, along with the convenience of engaging with the content at their own pace, the students are required to be more active in managing their own learning (Moore & Kearsley, 2011). Unsurprisingly, effective self-directed learning skills have been reported as the must for effective MOOC learners (Kop, Fournier, & Mak, 2011; Milligan & Littlejohn, 2014; Milligan, Margaryan, & Littlejohn, 2013). While, thousands of MOOC participants report their intention to finish the course through pre-course questionnaires (De Vries et al, 2015, in press), fewer complete, or even start the course. Holding a strong goal intention but not achieving the goal may be explained by people’s failure to self-regulate during goal striving (Gollwitzer & Sheeran, 2006). Gollwitzer (Gollwitzer, 1990) points out the distinction between goal setting and goal striving in his so-called “Rubicon model”. It includes four distinct phases. The first two - pre-decisional and post-decisional phases both precede taking action. These first two phases are characterized by wishing and deliberating, i.e. evaluating the wish in comparison to other competing wishes. Upon forming the goal intention, individuals move to the next phase of planning, within which they are attempting to commit to a certain course of action, but are not yet engaged in the action itself. The remaining two phases, i.e. goal initiation and implementation phases, are associated with the action itself. According to Gollwitzer, the shift from deliberating and planning to the actual implementation depends on the strength of person’s commitment to implementing the goal (ibid.). With the reference to the Rubicon model, and by considering the little effort invested into the actual enrolment in a MOOC, one can assume that some learners who enrolled in the MOOC may as well be in the pre-actional phase where registration for the course just signifies intention. As the course begins, students who interact with course’s content, as well as those who complete course activities would indicate higher commitment levels. Through our exploratory work with the course data, we observed that in the case of the DelftX MOOCs, 95% of all students who received certificates of completion in five MOOCs engaged with the formal assessment process, i.e. course assignments, from the very first beginning of the course. This observation is in a line with a study by Coffrin, Corrin, de Barba & Kennedy (Coffrin, Corrin, de Barba, & Kennedy, 2014) who suggest that early marks in MOOCs are a good predictor of the grade at the end of the course. It is quite logical that students, who received formal certification, engaged in the course assignments. Research reported in this study starts with the premise that all MOOC participants who engaged in MOOC assessment

tasks early on demonstrate the same level of commitment to learning in that course. In other words, this study approaches students’ engagement with early assessment activities as an indicator of commitment beyond that of intending to learn to the actual implementation of this goal. However, not all students demonstrating the same levels of commitment to learn early on sustain this level of commitment, and some disengage. Series of analysis reported in this paper investigates this argument, addressing patterns of engagement and disengagement with formal assessment, the levels of disengagement and contexts within which such disengagement occurred.

Data Description Analyses reported in this paper were conducted using the course level data from five MOOCs delivered by Delft University of Technology (TUD) in 2013-2014. The TUD has been an active member of the Open Movement since 2007, and joined the edX consortium in 2013 to strengthen the presence in the field of open and online education. TUD’s first five MOOCs represent a variety of applied hard and soft science courses at bachelor’s level, on topics of solar energy, infrastructure systems, water treatment, aero engineering and credit management (Table 1). Table 1: First generation of DelftX MOOCs. MOOCs

Period

#Enrolled

# Certified

Level

#1 ET3034TU Solar Energy (SolarX)

16.09 – 06.12.2013

57.091

2.730 (4,8%)

Bcs

#2 CTB3365 Introduction to Water Treatment (WaterX)

16.09 – 25.11.2013

29.088

545 (1,9%)

Bcs

#3 1110X Introduction to Aeronautical Engineering (AeroX)

03.03 – 19.5.2014

15.820

578 (3,7%)

Bcs

#4 TW3421 Credit Risk Management (CreditX)

18.04 – 30.6.2014

20.925

709 (3,4%)

Bcs

#5 NGI101x Next Generation Infrastructures. Part 1 (NGIx)

23.04 – 08.07.2014

16.091

517 (3,2%)

Bcs

139.015

5.079 (3,7%)

total

Analysis Study 1. The first study reported in this paper was framed along the descriptive research question: What are the patterns of students’ disengagement with formal assessment in DelftX MOOCs? To provide a visualization of patterns of engagement with formal assessment, we have plotted the number of students engaging in regular homework assessments and mid-course/final exams along the time sequence when they were offered to students (Figure 1). It could be seen that in some courses, e.g. AeroX, around 19% of course participants who attempted the first homework further engaged in further assessment activities leading to the certification; while in other courses, such as CreditX, almost 76% of the students who attempted the first formal assessment activity finished the course with formal certificate. In other words, in the analysed cases, the rates of retention among the students who intend to receive formal certificate and demonstrate their intention by engaging in formal assessment vary from 19% to 76%, depending on the course.

Figure 1: Retention in formal assessment across the first generation of DelftX MOOCs Figure 1 also indicates that the sharpest decrease in participation in formal assessment activities occurs between the first and the second homework assignments. It should be noted that across five courses the first and the second assignments did not follow similar deadlines: e.g. in AeroX the first and the second homework were due within the same week, in SolarX – within the first two several weeks, in NGIx – until the end of the course. We observed that the courses where the time difference between the two first homework assignments deadlines was smaller, the drop in participation is sharper. Regardless of the timeframe, most students disengage from formal assessment after they attempt the first homework. Other insights gained from this simple visualization of the patterns of engagement in formal assessments are that more people attempt exams rather than homework. That can be explained by the differences in assessment design within MOOCs: in courses where homework assignments carry more weight towards the final grade, the participation numbers in on-going homework assessments are higher. In courses where learners had a choice as to how many assignments to complete (e.g. NGIx) they preferred to engage with earlier homework tasks more than with the later ones. Finally, besides the sharp drop between the first and the second homework, we can see the changes in participation after Exam/Test grades are out. There may be different reasons for this: students may be only interested in some parts of the course, or, they may realize that further engagement in formal assessment will not yield a passing grade. To conclude, some similarities were observed upon initial look at the retention in formal assessment. All five courses demonstrated a sharp drop of participation after the learners attempted the first assessment activity. This observation led to the investigation of the relationship between the first formal assessment activity and further participation in the formal credentialing, as will be reported in Study 2. Additionally, we observed vast differences in the drop of the participation rates across courses, suggesting that course contingent factors may be at play. Study 3 refers to the pedagogical analyses conducted for the evaluation of DelftX MOOCs, to address the differences between the courses, as well as contextualize the patterns of engagement with formal assessment, including a possible relation between timing of assessment and retention. Study 2 The second study reported in this paper tested three hypotheses about the relationship between student performance on the first homework assignment (HW1) and their further engagement with formal assessment, including graded homework assignments, exams, and receiving of the certificate at the end of the course. More specifically, we assumed that there is a relation between poor performance on the first graded assignment and disengagement from further graded assignments. It should be noted that poor performance does not imply mistakes only. It could be that a learner started assessment and realized that it was too difficult, and the learner did not have sufficient skills; or the task required more time than the learner had available, which led to disengagement with the task, and thus low score. It could also have been that the assessment allowed only for one attempt, and the results were inputted in the wrong format, causing learners’ frustration. In any case, no matter what caused the poor result, it may have impacted students’ motivation, or informed the students about

the misalignment of their expectations with those of the teaching staff, possibly leading to student re-evaluating the intention and consequently disengaging from formal assessment. Hypothesis A. Students who disengaged from the formal assessment process after the first homework assignment were more likely to receive lower grades. An independent-samples t-test was conducted to compare grades for the first homework for students who went on to complete the second homework and for students who didn’t. Such analysis for the NGIx course could not have been implemented, due to a different grading policy. For the four remaining courses, there was a significant difference (p