The tutor in project-led education: evaluation of tutor performance N. van Hattum-Janssen1, R.M. Vasconcelos2 1
University of Minho, Research Centre in Education, 4710-057 Braga, Portugal (
[email protected]) 2 University of Minho, Council of Engineering Courses, 4810-058 Guimarães, Portugal
Abstract At the University of Minho, project-based learning is becoming part of most first year engineering curricula. Although projects are different, a common element is the changing role of the teachers, as some are supposed to function as tutors instead of lecturers. The precise role of a tutor in project-led education is rather ambiguous to both tutors as well as to students. In order to define this role and inform the tutors about their performance and help them to improve, the Council of Engineering Courses developed a questionnaire with both open and closed items that aims to evaluate the performance of individual tutors in a project semester. After a pilot version at two courses, a revised version was applied at the end of the first semester of 2007/2008. The answers to the open questions provided information about strengths and weaknesses of each tutor. The answers on the closed items helped to gain insight in the performance of a tutor with regard to the functioning of the group, individual learning processes, the progress of the project, the attitudes of the tutor with regard to project-led education and his/her role in the development of critical thinking and problem solving skills for students. Keywords: project-led education, tutor roles, questionnaire
1. INTRODUCTION Project-led engineering education (PLE) [1] was implemented at the Industrial Management and Engineering course of the University of Minho for the first time in the second semester of the academic year 2004/2005. The initiative of a group of teachers of this course led to the implementation of an approach to learning that was no longer merely subject based, but aimed to replace separate and sometimes not clearly related contents of subjects by one project, that would serve as a starting point for all subjects. The main goals that the teaching staff hoped to reach with the implementation of PLE were the improved integration of contents, the emphasis on student-centred learning, the early contact of students with real-life problems and the development of transversal competencies like communication skills, management skills, leadership and conflict management, because students work on the open ended problems in teams. In this teaching and learning approach, the teacher is no longer a lecturer, but is regarded as facilitator of learning. Instead of serving as an intermediate between specific course content and the individual student, a teacher becomes a tutor and is supposed to support the learning process of a group of students that try to develop competencies in areas that the tutor may or may not be familiar with. This role is a new one to most of the teaching staff. Teachers are used to roles that are directly related to their expertise and are less used to roles that are emphasise the support of students working in teams. Powell and Weenk [1] identify a number of tutor roles that can be distinguished in project work, starting with the tutor as setter of the exercise. Deciding on form and content of the project would in this case be a role of the tutor. Another role of a tutor can be the stimulator of the students by showing interest, asking about the why and how of the project, encouraging them to go into more depths and helping them to get through difficult periods, likely to occur in long lasting projects. A tutor can also be a monitor of the learning process, a role in which learning as a group effort is emphasised and the tutor supports the development of the cooperative effort to perform a number of task necessary to successfully conclude the project. The authors finally suggest that a tutor could serve as a technical expert and as an evaluator. They do not prescribe specific tutor roles, but suggest possibilities for a tutor to perform one or more of the roles as described above.
The role of a tutor depends on the project approach that is used, the roles of other staff involved in the project and the degree of interdisciplinarity. If a project is characterised as a project exercise [2], it takes place in the context of one single subject and in this case the teacher and the teaching assistants that are usually linked to the subject also serve as tutors. The degree of interdisciplinarity is reduced in this case. In a project characterised as a project component [2], more than one content area is involved, with a higher degree of interdisciplinarity. Teachers from different areas usually act as tutors. In a project approach, meaning an integrated approach of certain part of the curriculum through a project, the project is no longer separate from the subjects, but the subjects are partly or completely replaced by the project. In this case, tutors are not automatically teachers of specific content areas represented in the project, but are selected as a tutor for one or more student teams, regardless of their individual expertise as a teacher. Based on the features of the project, certain roles are appropriate or less appropriate to a tutor. Being the settler of the exercise, a technical expert or an evaluator of a project, implies content specific knowledge of a tutor that can not always be expected in case of a tutor who is especially appointed as a tutor without being related to a specific content area. On the other hand, the role of a tutor in stimulating cooperative team work, encouraging motivation and monitoring individual and group learning processes becomes more important in an extensive interdisciplinary project. This role is also emphasised by Dolmans et al. [3], who argues that groups that do not show equal input of all members, need extra effort from their tutor. For students, as well as for teachers, the role of the tutor in a project can be rather ambiguous [4]. Students have certain expectations towards their tutor and tutors on the other hand have ideas about their tasks and responsibilities as a tutor, which may not always coincide. A survey carried out among previous and actual tutors at the Production Engineering course identified 12 functions performed by tutors: (1) permanently monitoring the group while reaching project goals (2) monitoring the progress of the project (3) communicating problems to the coordinating team (4) functioning as a privileged communication channel towards the group (5) establishing a close relationship with the student team (6) identifying organizational problem within the team (7) identifying personal problem that reduce individual performance (8) participate in project assessment (reports and presentations) (9) participate in individual assessment of team members (10) contribute to the organisation and coordination of the project (11) guiding students to the relevant lecturer (12) supporting the development of presentations and reports [5]. The tutors also mentioned a number of actions that should not be performed by tutors: (1) interfering in the content of the project (2) helping the team with specific content, related to the specialisation of the tutor (3) making confidential information public (4) disagreeing with other staff (5) breaking the confidence of the student team [5]. The functions, as identified by tutors, were based on their experiences during one or more editions in a project-led learning experience. Formal roles of a tutor were not yet described in the project guide or other information channels linked to the PLE experiences at the Production Engineering course.
2. CONTEXT The teaching staff of the Production Engineering and Management course involved in PLE experiences is constantly trying to improve the way the project works in all its aspects, including the performance of the tutor. It was recognised that the tutors have a fundamental role in the project, as they have the closest contact with student groups and have a more coherent view on what students are doing and what failures and successes they are facing, both related to the goals of the projects as well as from a group communication point of view. There was, though, no consensus on the roles and tasks of a tutor among the teaching staff. Getting a better insight in what could be the roles of a tutor on the one hand and on the other hand evaluating his or her performance became, therefore, the goals of a study that was carried out at the Production Engineering and Management course. All the students of the first year participated in the study, as well as the fourth year students who participated in a different PLE experience, aimed at entrepreneurship. The student teams of the fourth year consisted not only of Production Engineering and Managements students, but also of Polymer Engineering students and Industrial Electronics students.
3. INSTRUMENT A questionnaire was developed, based on the tutor roles as identified by Powell and Weenk [1] and complemented with information from PLE experiences at other Engineering courses of the University of Minho. Studies on the roles of a tutor usually refer to problem-based learning instead of project based experiences [6, 7, 8], mostly taking place within medicine courses and instruments described in such studies are usually more aimed at the group process and less at the project goals that have to be achieved. The first version of the instrument was tested at the Production Engineering and Management course and the Fashion Design and marketing course and led to the removal of a scale on assessment, as both teachers and students agreed that a tutor should not take part in the assessment of his own team. As the reliability of this scale was only ,55, the entire scale was removed. Other items were reformulated, especially those that referred to the tutor as a content expert, as those were too confusing for many students. Both the first and the second version of the instrument include the role of the tutor in reaching the project goals, which distinguished a tutor in problem-based learning from a tutor in project-based learning. The second version consists of six scales and the first scale refers to knowledge on PLE. The second scale is on the attitudes of a tutor in the tutoring process. It contains 11 items that aim to reflect the attitudes of a tutor with regard to PLE, the tutorials and the communication between students and the tutor. The third scale is related to the progress of the student team in the project. It contains 10 items that intend to analyse how students think about the way the tutor monitors the progress of the project. The next scale contains four items on the development of critical thinking and problem solving. The penultimate scale contains seven items of group functioning of the student team, including, among others, items on the discussion of peer and self assessment results. The last scale dedicates three items to the learning process of the individual student. Apart from the 38 closed items, two open items were included to enable specific comment on tutor performance. A last general closed item was also added to make a general quantitative evaluation of the tutor on a 10 point scale. A total of 57 students participated in the tutor evaluation, of which 36 students are first year Production Engineering and Management students and 21 are fourth year students. The latter group consists of an equal distribution of Polymer Engineering students, Industrial Electronics students and Production Engineering and Management students. The student teams for both the first as well as the fourth year included 6 students. The questionnaire was applied at the end of the semester in the last project week. The tutors of the first year - tutor A till F - were all experienced tutor who had served as a tutor in two or more PLE experiences. Two of the tutors of the fourth year also served as a tutor in the first year, both teaching staff of the Production Engineering department. The two other tutors - tutor G and H - were new tutors and are teachers at, respectively, the Industrial Electronics and the Polymer department department.
4. RESULTS The internal consistency of the seven scales of the Tutor Evaluation Questionnaire was estimated by the Cronbach alpha coefficient and generally considered satisfactory, as presented in table 1, although the reliability on the critical thinking and problem-solving scale is rather low, which may be caused by the more abstract and extensive items of this scale. Scale Item Mean SD Cronbach number alpha PLE Knowledge 3 12,75 3,30 ,82 Attitudes 11 45,51 5,59 ,84 Project progress 10 40,38 5,09 ,91 Critical thinking-problem solving 4 16,86 1,96 ,65 Team functioning 7 28,33 4,27 ,83 Individual learning 3 11,30 2,17 ,75 TABLE 1. Reliability analysis of the questionnaire Looking at the results of the questionnaire of the first year students, as shown in Table 1 and 2, the scores on each scale show strengths and weaknesses of each tutor. Although there are many differences between individual tutors, some general remarks can be made. The PLE knowledge scale has the highest score in four groups. The attitudes of
tutors with regard to PLE and the stimulation of critical thinking and problem solving skills are also regarded as strengths of tutors. The support for individual learning processes is, on the other hand, not evaluated very positively. In seven out of ten student teams, it is seen as the weakest aspect of the tutor performance. In this respect, the first year students do not differ from the fourth year students. Individual learning is characterised by a low score in both groups. Scale PLE Knowledge Tutor A Tutor B Tutor C Tutor D Tutor E Tutor F Attitudes Tutor A Tutor B Tutor C Tutor D Tutor E Tutor F Project progress Tutor A Tutor B Tutor C Tutor D Tutor E Tutor F Critical thinking-problem solving Tutor A Tutor B Tutor C Tutor D Tutor E Tutor F Team functioning Tutor A Tutor B Tutor C Tutor D Tutor E Tutor F Individual learning Tutor A Tutor B Tutor C Tutor D Tutor E Tutor F
Min
Max
Mean
SD
12 11 14 10 10 13
14 15 14 15 12 15
12,67 13,67 14,00 12,50 10,67 14,33
1,03 1,75 ,00 1,64 ,82 1,03
39 39 39 42 38 50
50 50 44 53 41 55
45,33 46,50 41,83 47,50 40,25 52,67
3,67 4,23 2,32 3,78 1,50 2,25
30 38 31 34 30 47
44 46 35 48 44 50
39,5 42,17 33,60 39,83 35,00 48,50
4,89 3,31 1,52 4,62 4,98 1,38
14 14 15 15 14 17
19 19 19 18 18 20
16,33 17,00 17,00 16,50 15,00 18,83
1,75 2,00 1,26 1,05 1,55 1,47
27 26 24 23 18 32
33 32 27 32 31 35
29,50 29,17 24,83 27,00 25,67 33,50
2,26 2,32 1,33 3,69 4,37 1,38
9 9 7 8 9 14
13 14 11 13 15 15
11,00 11,83 9,33 10,50 10,83 14,33
1,55 1,83 1,37 2,26 2,14 ,52
TABLE 2. Results Production Engineering 1st year students Looking at the scores in the fourth year group in Table 2, it is noticeable that Tutor G has the highest scores at every scale, compared to the other tutors. Scale
Min
Max
Mean
SD
PLE Knowledge Tutor A Tutor B Tutor G Tutor H Attitudes Tutor A Tutor B Tutor G Tutor H Project progress Tutor A Tutor B Tutor G Tutor H Critical thinking-problem solving Tutor A Tutor B Tutor G Tutor H Team functioning Tutor A Tutor B Tutor G Tutor H Individual learning Tutor A Tutor B Tutor G Tutor H
10 11 13 9
15 14 15 15
11,17 12,6 14,40 11,60
2,04 1,14 ,89 2,30
35 41 50 37
46 52 55 50
39,00 45,80 53,00 42,20
4,05 4,49 2,00 5,07
33 35 42 31
48 49 50 41
39,83 41,20 47,00 36,6
5,71 5,12 3,46 3,78
12 15 19 13
19 18 20 18
16,50 16,00 19,60 16,00
2,43 1,22 ,55 1,87
21 24 34 17
30 33 35 30
26,00 28,40 34,60 25,20
3,52 3,36 ,55 4,97
9 10 12 6
12 13 15 12
10,83 11,20 13,60 9,60
1,17 1,30 1,34 2,61
TABLE 3. Mixed 4th year students Tutor B has the second highest scores on each scale and Tutor H and A are third or fourth on each scale. It could be remarked that more experience as a tutor does not automatically lead to a better perception of student performance of a tutor, as tutor G and H were inexperienced in tutoring PLE projects, whereas A and B have several years of experiences. It must be said though, that the results of Tutor A and B are show differences between the evaluation of their first year and their fourth year student group. With regard to the open items of the questionnaire, it can be remarked that the fourth year students left either one or both questions blank. Those who answered only one of the questions always chose the first one, on the positive aspects of the tutor. The first year students showed similar behaviour, as 19 left one of both questions blank, and in case of one unanswered question, it was always the one on expectations that were not yet met. When analysing the answers to the open questions of the fourth year students, it appears that they would like the tutor be more present at meetings. This comment was made by three students. One student also finds that a tutor should have more attention for who is working more and who is working less within the team. Motivating the team, being available, answering questions, proving contacts of companies and believing in the team were considered important contributions of the tutor to the performance of the student teams, according to the fourth year students. The first year students show a more diverse pattern of answers to the open questions. With regard to the important contributions the tutor made to their team, they comment that their tutor tried to keep them on track when they felt lost, that the tutor gave valuable information on how to format the reports and the presentations, that they received important feedback on the reports, that a critical analysis was provided when necessary and that the team spirit was encouraged by the tutor. The first year students hardly made any clear comment on expectation that had not yet been met. Only being more present at meetings, being slight more active and being more severe could be identified as serious comments.
5. DISCUSSION In this study, the role of the tutor in PLE was subject to analysis through a questionnaire that aimed to evaluate tutor performance. The results of the questionnaire being applied in two distinct groups with experienced and inexperienced tutors show large differences between the tutors of the first year and rather high differences between the tutors of the fourth year. The questionnaires provide useful information to the tutor on his or her own performance and indicate clearly the strengths and weaknesses of each tutor. The results also show that more experience as a tutor does not automatically mean that performance is regarded as better by the student teams. The tutor with the highest scores of the fourth year was a first time tutor, whereas the tutor with the lowest scores of the fourth year was a very experienced tutor. The tutors who served both in the first as well as in the fourth year both obtained higher scores in the first year, although the differences are higher for tutor A than for Tutor B. The seven scales that were used highlighted aspects of tutor performance that clearly differ from traditional teacher performance. The results can help individual tutors to identify their strengths and their weaknesses, but can also serve as a starting point for learning from each other. The items on problem solving and critical thinking need some attention and will probably need to be restated in language that is more accessible for first year engineering students. The questionnaire focuses on specific tutor functions like supporting group processes, monitoring progress and stimulating critical thinking and problem solving skills. As the regular teaching assessment of the University of Minho does not take into account these specific aspects of tutor performance, the instrument is seen as a useful supplement to the standard questionnaire and is already requested by teaching staff of other Engineering courses to help improve their project-based learning experienced. As project-based learning is interpreted differently in different courses, more studies to analyse the usefulness of the instrument in different context would be useful. Before using the questionnaire, tutors were not able to identify opinions of students with regard to their performance as a tutor. Currently, they are receiving information that can help them to evaluate and improve their performance. Although it is common practice at many universities to assess teaching performance and give feedback to teachers in their role as teacher, feedback on tutor performance was not included in teacher assessment processes. The instrument developed in this study helps teachers to analyse their role as a tutor in project-based learning and improve their performance as such, especially in a process that not only has to lead to acquiring certain competencies, but also to a satisfactory final product. References [1] Powell, P. & Weenk, W. Project-led engineering education. Utrecht: Lemma (2003). [2] L. Helle, P. Tynjälä & E. Olkinuora, Project-based learning in post-secondary education – theory, practice and rubber slings shots. Higher Education, 51, 287-314 (2006). [3] D.H.J.M. Dolmans, H.A.P. Wolfhagen, A.J.J.A. Scherpbier, & C.P.M. van der Vleuten, Development of an instrument to evaluate the effectiveness of teachers in guiding small groups. Higher Education, 46, 431-446 (2003). [4] R.M. Lima, D. Carvalho, M.A. Flores & N. van Hattum-Janssen, A case study on project led education in engineering: students’ and teachers’ perceptions. European Journal of Engineering Education, 32, 337-347 (2007). [5] A. C. Alves, F. Moreira, & R. Sousa, R. The role of tutors in project-based learning: three years of experience at the Engineering School of the University of Minho. [O papel dos tutores na aprendizagem baseada em projectos: três anos de experiência na Escola de Engenharia da Universidade do Minho]. In A. Barca, M. Peralbo, A. Porto, B. Duarte da Silva & L. Almeida (Eds.), Libro de Actas do Congresso Internacional Galego-Portugués de PsicoPedagoxía (pp. 1759-1770). A Coruña/Universidade da Coruña: Número extraordinário da Revista Galego-Portuguesa de Psicoloxía e Educación (2007). [6] M. A. Hitchcock and Z.-H. Mylona. Teaching faculty to conduct problem-based learning. Teaching and Learning in Medicine, 12, 52-57 (2000). [7] A.J. Neville. The problem-based learning tutor: teacher? Facilitator? Evaluator? Medical teacher, 21, 393-401 (1999). [8] M. Das, D.J.S. Mpofu, M.Y. Hasan, & T.S. Stewart. Student perceptions of tutor skills in problem-based learning tutorials. Medical Education, 36, 272-278 (2002).
