Towards Measurement of the Relationship

0 downloads 0 Views 279KB Size Report
Pairwise relations between average results for all modules in a year, normalized .... International Journal of Advanced Computer Science and Applications, 4(9),.
Towards Measurement of the Relationship between Student Engagement and Learning Outcomes at a Bricks-and-Mortar University Carmel Kent*, Chris A. Boulton, Hywel Williams University of Exeter, UK c.kent,c.a.boulton,[email protected] Abstract. The relationship between student engagement and learning outcomes has been extensively studied in the context of online learning. However, it has been less investigated in face-toface learning. In this paper, we describe initial findings from a study of student engagement and outcomes at a ‘bricks-and-mortar’ (BaM) university, where engagement is characterized by a diverse set of systems and agents, spanning both physical and digital spaces. We ask whether the substantial relationship often found in online environments between engagement and outcomes, holds in a BaM setting. We present initial analysis of data traces from various sources, each relating to a different dimension of engagement. Initial results indicate a weak relation between engagement and outcomes, suggesting that this important relationship may be substantively different in face-to-face/BaM and online learning environments. These preliminary findings highlight the need for further research, tackling challenges which are specific to face-toface learning in a bricks-and-mortar university environment. Keywords: Learning analytics · Bricks and mortar · Learning outcome · Engagement

1 Introduction Student engagement and learning outcomes are amongst the most ill-defined and broadly interpreted theoretical concepts, for which there are no currently agreed upon frameworks for operationalization [2, 5]. This vagueness is one of the reasons for the general lack of clarity about the relation between them [16]. Nevertheless, the relationship between engagement and outcomes has been extensively investigated in the context of online learning, in which participation is shown to be highly correlated with various types of learning outcomes [1, 3-4, 7, 11, 20-22]. In purely online learning settings, engagement is typically defined narrowly in terms of student interactions with a Virtual Learning Environment (VLE), usually in the context of a specific course or module. However, student engagement in ‘bricks-and-mortar’ (BaM) institutions, where most teaching is delivered face-to-face, is much less clearly

defined and there are associated difficulties in its measurement and analysis. Thus much less evidence is gathered in order to determine if and how engagement is of value for predicting outcomes in face-to-face learning. Online learning is not analogous to face-to-face learning and each requires different conceptualization and operationalization frameworks [8, 12]. Moreover, students are shown to engage differently when learning in an online learning environment as opposed to BaM environment, also resulting in different learning outcomes. Specifically, this difference might be explained by the nature of online learning, which is more self-regulated [10]. Within a BaM environment, learners interact with a wide variety of systems, some of which relate directly to their course performance (e.g. lectures, assessments, VLEs) while others address learning outcomes in a wider context (e.g. career planning). Comparative study of higher education learning across different contexts and environments is still in its infancy [10] and holds many technical challenges relating to the collection, integration and ethical aspects of data from multiple sources [18]. In this paper, we present initial insights into the relationship between student engagement and learning outcomes in a BaM university. While trying to capture this relationship's flexible and sometimes elusive nature, here we adopt a holistic approach that aims to integrate data captured from various sources and interaction points in order to provide a multidimensional image of student engagement. 1.1 Measuring Engagement The phrase “student engagement” has come to refer to the level of involvement students appear to have within their classes and their institutions in the context of learning [5]. Moore [14] proposed three types of interactivity: learner-content, learner-instructor and learner-learner. We suggest extending this framework and viewing students' engagement at a BaM institution as a multi-dimensional construct entailing the measurement of interactions between the student and various types of resources and agents (such as systems, people and devices) associated with the individual learning experience [19]. For our purpose, an interaction denotes a singular instance or event in which a student uses a resource, and represents a temporal relationship between the student and the resource [6]. For instance, an interaction may be attending a lecture, submitting a quiz, speaking to a lecturer, or accessing the VLE. It is very difficult to separate the net contribution of each type of interaction to the learning process. Even in the field of online learning, where interactions are easier to identify, this debate still remains open [5]. In addition, it is very complicated to study 'engagement' across different learning designs/goals and backgrounds of students.

Thus, in this paper, we execute an initial cross-design analysis, and add demographic parameters, in order to support future work with more fine-grained cohorts. 1.2 Measuring Learning Outcome It is enormously challenging to measure the depth of understanding at a coursespecific learning outcome [17, 19]. Module results usually include assessment tools that are defined by clarifying specific learning objectives [13]. There are important differences between face-to-face and online learning, including the pedagogical basis for assessment. Instructivism, which is common in BaM's face-to-face learning, maintains that knowledge should be transferred directly from the instructor to the learner without further interactions [15]. On the other hand, social constructivism is often implemented in collaborative online learning environments, whereby the teacher is seen as a facilitator between students, content and platforms, and social interactions are more central [9]. In accordance, the definition of learning outcomes might reflect on that difference, and thus face-to-face learning assessment in BaM institutions could be less correlated with interactive behaviors. While we recognize that there are many kinds of learning outcome, in this initial study we focus on student performance as measured by module grades.

2 Method Engagement has been shown to correlate with performance in online learning. In this study, we ask how this relation manifests in a BaM university. More specifically, we attempt to determine what types of interactions and student characteristics can predict specific learning outcomes. Working with data from a traditional BaM university in the UK, we have collected data from various university systems for 30,781 undergraduate students across three academic years commencing in Autumn 2013, 2014 and 2015. Tables 1, 2 and 3 below summarize the variables extracted to operationalize engagement, demographic characteristics and learning outcomes respectively. The systems from which the variables are extracted, as well as basic descriptive statistics, are also presented. Our initial unit of analysis was the aggregate of all interactions involving a specific student in a specific year, resulting in a dataset of 52,553 records.

Table 1. Engagement variables and data sources Variable Number of attended career events Number of signed-up career events Proportion of career events signed-up to that were attended Number of logins Number of logins

Number of logins

Number of library's fines paid Number of logins

Number of submitted evaluations Number of logins Number of papers' views Number of all interactions Number of committee interactions Number of enrolled programs

System Career Events System. Events are optional and cover a wide range of topics. Career Events System

Missing 0

Mean 1.60

St Dv 2.88

0

1.99

3.34

Careers Events System

0

0.46

0.46

Virtual Learning Environment (VLE) Inter Library Loans (Library ILL). Online access to borrow books from other UK libraries.

0

79.72

68.83

0

0.00

0.63

Library. Online access to academic journals and eresources and manage library resource loans. Library

0

0.19

0.70

0

0.12

0.54

MACE (Module and Course Evaluation) system. An optional quality questionnaires for students. MACE

0

1.17

1.45

0

0.83

1.07

Exam's archival system Exam's archival system

0 0

3.29 15.35

6.64 28.20

All systems (VLE, Library ILL, Library, MACE, Exam's archival system) Student's guild (buying tickets to guild's events, holding positions on volunteering project committees) Registration system

0

100.66

87.66

0

0.03

0.18

0

1.07

0.26

Table 2. Demographics variables and data sources Variable

System

Gender

Registration system Registration system Registration system Registration system

Disability (type of disability) National identity Nationality

Missing

Type

Dominant category Female 55.1%

30

Categorical

72

Categorical

5,700

Categorical

No known disability 87.2% British 41.3%

1,584

Categorical

UK 69.8%

Country of domicile Ethnicity Age at enrollment to the university Age at the beginning of the year Living away from home Parents' occupational background

Registration system Registration system Registration system Registration system Registration system Registration system

56

Categorical

England 69%

1,595

Categorical

White 73.9%

265

Numerical

Mean: 19.80

256

Numerical

Mean: 21.04

0

Binary flag

Away: 72.6%

6,339

Categorical

Higher managerial 23.2%

Table 3. Outcome variables and data sources Variable Average number of attempts for all modules in a year Average results for all modules in a year, normalized by credits' weights (i.e. summative ‘end of year result’) Number of failures in all modules in a year Number of pass grades in all modules in a year Proportion of passes out of all passes and failures Number of results which were not agreed in a year Number of agreed upon results in a year Average gap between module result and its class average

System Module Assessment Module Assessment

Missing 2,814

Mean 1.01

St Dv 0.09

8,106

49.80

21.31

Module Assessment Module Assessment Module Assessment Module Assessment Module Assessment Module Assessment

7,697

0.15

0.84

7,697

4.21

2.70

7,697

0.96

0.16

0

0.07

0.31

0

6.84

2.77

8,106

0.003

2.64

3 Findings Here we present our two-step analysis. First, we present the significant pairwise relations found between outcome variables, and engagement or demographics variables. Second, we present a multivariate model to try and predict student success based on features found to be significantly correlated with outcome at our first step. 3.1 Pairwise Relations between Outcome, Demographics and Engagement Since none of our outcome variables are normally distributed, we used Spearman's rank correlation test to find significant relationships between them and any numeric engagement variable. A quick exploration of the engagement variables shows that VLE logins, Past Exam views, Library logins, MACE submissions and event

attendances follow a typical power law distribution as one might expect, where many students use each individual system sparingly and few students use each system often. When correlating outcome with categorical variables, we have used Mann-Whitney U test and Kruskal-Wallis H test. For space limitations, we only show significant relations with the normalized result outcome variables, we are omitting significant relations which were found to be very weak, in addition to some of the post-hoc results. Table 4. Pairwise relations between average results for all modules in a year, normalized by credits' weights variables and between demographics and engagement variables. Significant demographic variables Gender U = 231120953.50**

Selected post-hoc results

Away from home U= 152140073.00** Disability H(10) =168.02**

Away(Med = 60.25)>Local(Med = 52.01) Long standing illness, Mental health, Mobility issues, learning difficulty > Information refused Refused >No disability>Disability

Is Disable flag H(3)=73.89** Country of domicile H(140)=1,554.98** Ethnicity H(18)=627.97** National identity H(7)=360.69** Nationality H(187)=1,880.03** Parents' occupational H(326)=869.74**

Female (Med= 60.16)>Male (Med = 57.03)

Engagement variables MACE- logins (r = 0.262)** MACESubmitted evaluations (r = 0.250)**

White>Arab, Asian, Black

All managers > All routines roles

*Correlation/ difference is significant at the .05 level (two-tailed test) **; significant at the .01 level or below (two-tailed test)

3.2 Multivariate Model to Predict Outcome out of Demographics and Engagement For our regression model, we used the ‘logged’ version of some of the numeric variables in an attempt to make them more normally distributed, which while helpful has not fully solved the problem of non-normality. As there are some students who have not accessed some of the systems at all, we make the transformation x -> log(x+1) which maps 0 to 0 and prevents the problem of trying to use log(0). We have fitted a model to predict the weighted average results for a year, resulting (F(11, 44425) = 512.7, R2 = 0.1126, Adjusted R2 = 0.1124), p< 2.2e-16, Residual standard error=20.08. The parameter estimates and significances are detailed in Table 5 below.

For the categorical variables in the table, the first factor that appears in the data is assumed to have a coefficient of 0 (e.g. “Female” has no effect on our model) and other factors for that category are assigned a coefficient of which the significance is then determined. Table 5. Multivariate model parameters and significances (engagement variables are in bold) Coefficients (Intercept) Gender (Male) Age at beginning of year Away from home Disability Type (Unknown) Is Disable (Yes) log(events attended + 1) Committee interactions log(VLE + 1) log(Past exams + 1) log(Library logins + 1) log(MACE + 1)

Estimate 52.446 -0.239 -0.328 5.973 2.107 -2.834 2.832 8.914 -1.944 0.130 4.842 9.224

Std. Error 0.823 0.19286 0.035 0.224 1.936 0.284 0.132 0.489 0.085 0.067 0.302 0.187

t value 63.751 -1.242 -9.399 26.616 1.088

p-value