29. 3. Interdisciplinarity ... as a method for the accreditation of IVET and CVET qualifications in study ... the Module Level Indicator (MLI) and the Learning Outcome Matrix (LOM). .... The English version was developed in the course of the .... used for the assessment of the learning success of a particular learning unit.
Wolfgang Müskens, Wolfgang Wittig, Roland Tutschner, Anja Eilers-Schoof (eds.)
Module Level Indicator
MLI User Guide Assessment of the Level of Competence Orientation
1
Module Level Indicator MLI – User Guide Assessment of the Level of Competence Orientation
Imprint Publisher: Institute Technik und Bildung (ITB) Universität Bremen Am Fallturm 1 28359 Bremen Germany Editors: Wolfgang Müskens, Wolfgang Wittig, Roland Tutschner, Anja Eilers-Schoof Layout: Dirk Stieglitz, ITB, Bremen, Germany Printing: Druckerei Universität Bremen, Bremen, Germany
The project CREDICARE (“Quality-oriented Accreditation of Vocational Learning Outcomes in Health Care and Nursing”) has been funded with support from the European Commission within the “LIFELONG LEARNING PROGRAMME - LEONARDO DA VINCI Transfer of innovation” action. This publication reflects the views only of the author, and the Commission cannot be held responsible for any use which may be made of the information contained therein. Project website: www.credicare-project.eu Bremen 2013
Content Introduction_______________________________________________________________ 4 Development of the MLI_____________________________________________________ 5 Structure of the MLI________________________________________________________ 9 The MLI questionnaire______________________________________________________ 11 The process of MLI assessment_______________________________________________ 17 Results of the MLI_________________________________________________________ 19 Quality criteria of the MLI___________________________________________________ 22 References________________________________________________________________ 24 Annex 1: Glossary__________________________________________________________ 25 Annex 2: The scales of the MLI_______________________________________________ 26 1. Broad and up-to-date knowledge____________________________________________ 26 2. Critical understanding____________________________________________________ 29 3. Interdisciplinarity_______________________________________________________ 32 4. Practice orientation______________________________________________________ 34 5. Problem solving_________________________________________________________ 36 6. Creativity and innovation_________________________________________________ 39 7. Autonomy______________________________________________________________ 41 8. Communicative competence_______________________________________________ 44 9. Consideration of social and ethical issues_____________________________________ 47
3
Introduction The aim of the Leonardo da Vinci project CrediCare has been to compare training programmes in health care and nursing in Estonia, Finland, France, Germany and Ireland on the basis of learning outcomes. In order to implement this comparison of national qualifications, the methodology of the so-called equivalence check was adapted and developed further. This procedure was initially developed from 2005 to 2007 in the course of the ANKOM initiative of the German Federal Ministry of Education and Research (BMBF) as a method for the accreditation of IVET and CVET qualifications in study programmes at universities. The core elements of the transnational comparisons of nursing qualifications in the CrediCare project are the Module Level Indicator (MLI) and the Learning Outcome Matrix (LOM). The function of these tools is on the one hand to determine the level of modules according to the reference levels of the European Qualifications Framework (EQF) and on the other hand to compare modules from different (national) qualifications due to their learning outcomes. While the LOM makes it possible to describe and compare the contents of the modules, learning units or subjects related to a qualification, the 51 items of the MLI, which are derived from the EQF descriptors, allow to identify the levels of these modules. Accordingly, the method of the equivalence check allows for a transparent description and assessment of learning outcomes with the EQF levels serving as the standard of reference. The purpose of the present MLI User Guide is to give a detailed description of the equivalence check methodology for interested practitioners, and especially to introduce potential users into the MLI. The guide is also meant to serve as a reference document for the application and use of the MLI. The first section of this guide describes the background and evolution of the MLI, the aims and objectives associated with the development of the instrument, and the experience gained with its application. In the second section the fundamental concepts, the scales and the overall structure of the MLI are presented. The MLI questionnaire, which consists of 51 items that are grouped in nine scales, is also included here. The third section describes the process of assessing modules, learning units or subjects on the basis of learning outcomes by means of the MLI. Here we give an outline of the reference materials that have to be used for the application of the MLI and explain how the results of the MLI review are interpreted against the background of the European Qualifications Framework. This involves also a discussion of the quality criteria such as validity and reliability that have to be fulfilled by the tool. Finally we would like to point out that a great deal of additional information about the MLI items and scales can be found in the annotated questionnaire in Annex 2, which includes explanations to the items and response options. These “Frequently Asked Questions” reflect the experience gained by the project partners during the period of the CrediCare project and provide guidance for stakeholders and practitioners who wish to use the MLI in practice.
4
Development of the MLI Background From 2005 to 2007 the German Federal Ministry of Education and Research (BMBF) supported the pilot programme ANKOM, which aimed to improve permeability and progression between vocational and higher education (cf. Hartmann & Stamm-Riemer, 2006). In the course of this programme instruments and procedures were to be developed whose purpose was to support the accreditation of vocational learning outcomes in higher education. One of the projects supported by the ANKOM initiative was the pilot project “Qualifikationsverbund Nord-West”, in which the University of Oldenburg and the University of Bremen took part among others. One of the key activities of the project was the development of procedures for a blanket accreditation of vocational learning outcomes in university studies. In the case of a blanket accreditation the extent of accreditation is determined once on the basis of a systematic comparison between the accrediting course of study and the vocational qualification to be accredited. This comparison is labelled “equivalence check”. Once the equivalence of learning outcomes has been established, all holders of the vocational qualification in question can have their learning outcomes recognised in the study programme without the need for further individual assessment. The equivalence check examines as to whether learning units of the receiving programme and the qualification to be accredited correspond to each other in terms of content and level. If such a correspondence can be identified, the learning unit in question is recommended for accreditation. Accordingly, the accreditation process involves the identification of levels of learning units in vocational and higher education. In the course of the ANKOM project “Qualifikationsverbund Nord-West” an instrument for the level assessment of learning units from different areas of education was developed. This instrument was termed Module Level Indicator (MLI) (cf. Gierke & Müskens, 2009). Although the MLI was developed with a view to comparing learning outcomes from different educational sectors, it also allows for the comparative assessment of learning outcomes within one sector, e.g. in higher education. 2006
2007
ANKOM (Accreditation of vocational competences in higher education)
2008
2009
2010
2011
2012
2013 ANKOM III INOS (until 2014)
ANKOM follow-up projects
CREDIVOC - Accreditation of Vocational Learning Outcomes
CrediCare (nursing occupations)
Offene Hochschule Niedersachsen (Open University Lower Saxony)
Competence unit for accreditation
Figure 1. Projects regarding accreditation of prior learning at University Oldenburg.
The focus of the ANKOM project “Qualifikationsverbund Nord-West” (2005-2007) was on qualifications in business occupations and business studies. The Module Level Indicator was developed initially with a view to this vocational discipline (Müskens, 2007).
5
The subsequent project CREDIVOC (Transfer and Mobility through Accreditation of Vocational Learning Outcomes), which was supported from 2007 to 2009 within the Lifelong Learning Programme of the European Union, piloted a transfer of the instrument into the domain of engineering (Müskens, Tutschner & Wittig, 2009). The extension of the instrument to other domains and vocational disciplines (e.g. Finance & Banking, Educational Sciences, Business Informatics) was supported from 2009 to 2012 in the context of the Open University programme of the German province of Lower Saxony (Müskens & Eilers-Schoof, 2013). The MLI was tested by the Carl von Ossietzky University of Oldenburg in cooperation with other German universities and universities of applied sciences (e.g. Braunschweig University of Technology, Jade Hochschule, University of Applied Sciences Bielefeld, University of Bremen). The original version of the MLI was in German. The English version was developed in the course of the CrediCare project, which has been supported since 2011 within the Lifelong Learning Programme.
Objectives of the MLI The development of the MLI addressed several objectives: The MLI was intended to be an instrument for determining the level of learning units from different educational sectors. The instrument was meant to be applicable for learning units related to vocational qualifications, further education certificates, and academic degrees. ■ Unlike qualifications frameworks, the MLI was to refer not to entire qualifications, but to components of qualifications (i.e. modules). ■ The instrument was to be applicable in as many diverse disciplines and occupations as possible. ■ The MLI was to be capable of analysing and evaluating learning units connected with any type of learning and assessment. ■ The MLI was to enable a reliable identification of the level of learning units. ■ The level identified by the MLI should correlate as highly as possible with the level that would be the result if learning units were referenced directly to the EQF. In order to develop the MLI, statements and criteria were collected that have the potential to describe the level of learning in formal learning processes of adults. These statements were drawn from the following sources: ■ the draft European Qualifications Framework for lifelong learning (EQF1), ■ the framework of qualifications for the European Higher Education Area (EHEA Framework)2, and ■ interviews with managers of German enterprises on relevant aspects of professional competence. In 2006 a preliminary version of the MLI (version 1) was published. The instrument was revised on the basis of an analysis of items. The current version 2 of the instrument was subsequently released on 4 July 2007.
1 (EU (2007): European Parliament legislative resolution of 24 October 2007 on the proposal for a recommendation of the European Parliament and of the Council on the establishment of the European Qualifications Framework for lifelong learning (http://www.europarl.europa.eu/sides/getDoc.do?pubRef=-//EP// TEXT+TA+P6-TA-2007-0463+0+DOC+XML+V0//EN) 2 EHEA (2005): The framework of qualifications for the European Higher Education Area. URL: http://www.ond. vlaanderen.be/hogeronderwijs/bologna/documents/QF-EHEA-May2005.pdf
6
European Qualifications Framework for lifelong learning (EQF)
Framework of qualifications for the European Higher Education Area (EHEA Framework)
Interviews with managers on aspects of professional competence relevant for employers
Level of competence orientation of a learning unit (module) from vocational education, adult education or higher education
Module Level Indicator (MLI)
Figure 2. Sources of the MLI items.
Potential application of the MLI The MLI can be applied whenever the level of learning units related to qualifications in vocational education, higher education or adult education has to be determined. Such a level assessment is relevant, for instance, when it comes to the accreditation of vocational learning outcomes in university programmes. In these cases the level assessment by means of the MLI is part of a so-called “equivalence check” between a vocational and an academic qualification. Study module (e.g. production)
Level Contents • Skills • Knowledge
• newly developed instrument: MLI (Module Level Indicator) • reference to the European Qualifications Framework (EQF)
Equivalence check Expert evaluators estimate • to what extent the contents of a study module are covered by the CVET qualification and • whether the level of the competences to be accredited corresponds to the level of the study module
CVET qualification
Figure 3. Equivalence check.
The experience of applying the MLI As explained above, the MLI was originally developed in a project consortium oriented towards economics and business administration. It was used in this context for the accreditation of VET-based competences in university courses.
7
The first level comparisons took blace between modules of a bachelor’s degree in business administration on the one hand and learning units from various advanced vocational qualifications (certificates awarded by the chamber of industry and commerce) in the area of business and commerce on the other. Likewise, learning units related to various certificates of adult education providers were compared with modules of the bachelor programme in question (cf. Müskens, 2007). Some of these learning units had a similar or equal level as those bachelor modules that are typically part of the initial phase and concern the fundamentals of the degree programme. Some particularly comprehensive qualifications from adult education and continuing vocational education featured learning units whose level was equal to average bachelor modules and sometimes even higher (cf. Müskens, 2007). Similar results were observed in the area of engineering. These results were based on the comparison of initial and continuing training programmes in information technology with a bachelor’s degree in business informatics (information systems). In this case, too, the level of certain learning units of the CVET qualification was higher than the corresponding modules of the bachelor’s degree (cf. Hanft & Müskens, 2013). Another level comparison was carried out in the juxtaposition of the vocational qualification of statecertified engineer (staatlich geprüfter Techniker) with a bachelor’s degree in mechanical engineering. The level of some of the vocational learning units was again equal to or higher than the level of the study modules (cf. Müskens, Tutschner & Wittig, 2009). In economics as well as in engineering the MLI scores of the modules and learning units show that there are absolute MLI ranges that are “typical” for the bachelor’s degree and its entry level (cf.The levels, pp. 25). The comparison of a specialist certificate in early childhood education and a bachelor’s degree in inclusive early education also led to corresponding MLI scores for the modules and learning units reviewed (cf. Eilers-Schoof & Müskens, 2013). The absolute scores, however, were notably higher than in economics and engineering. Similar scores were achieved also in learning units in the areas of health and nursing. This can be attributed to the fact that ethical and social considerations play a more important part in these professional areas than, for instance, in engineering. This leads to the conclusion that the MLI in its current version can be used for level comparisons within but not between disciplines or domains. After the usability of the MLI had been tested at the level of various bachelor’s degrees, there was the question as to whether the instrument was applicable for master’s degrees as well. In the field of mediation the MLI was filled in for study modules of a master’s degree and learning units of a comprehensive certificate. The MLI analysis showed comparable scores for the study modules and the learning units, most of them above the bachelor (generic) level. The MLI procedure thus gave a reason to recommend an accreditation at the master’s level (cf. Eilers-Schoof & Müskens, 2013). Beside national qualifications, the MLI was also used for a closer look on international degrees. The equivalence check was realised between an international master’s degree and a German master of innovation management. Some of the learning units could be assigned to the master’s level or to the scope between the bachelor’s and the master’s level.3 Completing the field of current applications, the MLI lead on to an interlocking of vocational qualifications and a bachelor’s degree in the insurance, banking and finance sector. Important advices for the accreditation amount of the vocational qualifications were given and used for the transition to the course of study. Furthermore the course of study was shortened by accreditation. And with the close view on the MLI scales, changes within the content of some study modules were suggested (Hanft & Müskens, 2013). Summarising the different examples given, they show a broad range of different applications of the Module Level Indicator.
3
8
http://www.anrechnung.uni-oldenburg.de/download/Anrechnungsempfehlung_01.pdf
Structure of the MLI The MLI is a clearly structured assessment tool which comprises 51 items. The criteria that are due to be assessed by the assessor refer primarily to the knowledge and skills being taught as well as the method used for the assessment of the learning success of a particular learning unit. 51 MLI items
9 MLI scales Broad and up-to-date kn. Critical understanding Interdisciplinarity Problem solving Practice orientation Creativity and innovation Autonomy
MLI total score M (=level of competence orientation)
Communication Consideration of social and ethical issues
Figure 4. Structure of the MLI.
The 51 items for one particular learning unit are being merged into 9 measurement-theoretically developed and reliable outcome scales: ■ The scale “broad and up-to-date knowledge” describes the scope, profoundness and up-todateness of the knowledge and skills imparted in the learning unit. ■ The scale “critical understanding” describes in how far the theories, models and methods imparted in the learning units are being reflected upon critically. ■ The scale “problem solving” describes if, and in how far, the students are confronted with complex problems in the learning units that are due to be solved by them independently through the application of cognitive and/or practical skills ■ The scale “practical relevance” describes if, and to what extent, the study materials and assessments of learning success are related to the real demands of practice and practical problems. ■ The scale “ability to work independently” describes the scope of independence and assumption of responsibility that is expected from the students in the learning units ■ The scale “consideration of social and ethical issues” describes if, and in how far, social and ethical issues are being picked out as a central theme in the learning units. ■ The scale “interdisciplinarity” describes to what extent a particular learning unit is related to other professions or disciplines, and to what extent it is able to teach students how to handle challenges in interdisciplinary contexts. ■ The scale “innovation” describes if, and in how far, the methods for the assessment of learning success of a learning unit confront students with novel and original problems that require creative approaches for finding a solution.
9
■ The scale “communication” describes to what extent students are being taught how to communicate information, ideas, problems and approaches for their resolution to fellow students, experts of the field and laypeople. Table 1. Scales of the MLI version 2.1 with sample items Knowledge Broad and up-to-date knowledge
The module comprises at least some profound inventory of knowledge that is state of the art within the field of expertise
Critical understanding
The module imparts an awareness of the limitations of the acquired skills and knowledge
Interdisciplinarity
The module comprises interdisciplinary problem statements whose resolution is based on the application of knowledge from various disciplines
Skills (1) Problem solving
The learning requirements and accordingly the examination assignments require a comprehensive application of cognitive and practical skills.
Practical relevance
The module imparts knowledge and skills that can be directly applied in practice
Innovation
The learning requirements comprise the development of new strategic approaches
Skills (2)
10
Ability to work independently
The learning requirements call for independent action and a being proactive
Communication
The students have proven that they are capable of communicating their comprehension of their field of expertise to other individuals
Consideration of ethical and social topics
When resolving a problem, the students demonstrate consideration of others and solidarity with people who might be affected by their actions
The MLI questionnaire MLI-S
Release 2.1 – 2007-08-09
Module Level Indicator Institution: _________________________________________________________ Title of programme or qualification: ________________________________________
● fully correct
● rather correct
● partially correct
● rather not correct
● not correct at all
Title of module / unit: _______________________________________________
Broad and up-to-date knowledge Contents and learning methods 1
The level of the knowledge imparted exceeds the basic general knowledge of adults.
47
The module includes the most important facts, principles, methods and general concepts of the subject matter.
49
The module covers a broad range of theories in the field.
37
The module includes highly specialised knowledge.
10
The knowledge and understanding in this module are at least on the level of an advanced textbook.
19
The module includes some current aspects that are not covered by the standard textbooks in the field.
28
The module includes some in-depth knowledge that reflects the current state of research in the field.
44
The module also covers some of the latest findings in the area, drawing for example on current research papers.
Assessment and examinations 51
The assessment within the module expects students to have knowledge of at least some theories or models in the field.
11
Critical understanding Contents and learning methods 15
The module gives an account of the observations or research findings on which the theories and models presented are based.
48
The module gives at least some suggestion as to how the theories and methods were developed and empirically tested.
46
The module presents alternative approaches or theories which contradict each other.
24
The module gives an idea of those topics or subject matters to which the theories and models presented cannot be applied.
6
The module raises awareness of the boundaries of the knowledge imparted.
33
The module imparts a critical understanding of the theories and principles of the field.
Assessment and examinations 40
The examinations and assessments in the module require learners to critically reflect the theories and principles of the field.
50
The examinations or assessments also refer to the historical context in which theories, models or techniques evolved.
Interdisciplinarity Contents and learning methods 22
The module includes references to other disciplines or occupations.
12
The module includes interdisciplinary topics that have to be addressed on the basis of knowledge from different disciplines.
31
The module informs the learners that the meaning of terms may vary between different disciplines or contexts.
Assessment and examinations 13
The learners are required to present their solutions and ideas in a manner that is understandable for members of other professions or disciplines.
Practice orientation Contents and learning methods 34
The theories and models that have to be learned in the module are explained at the example of practical situations.
16
The module imparts knowledge that is immediately applicable in practice.
41
The module transfers skills that are immediately applicable in practice.
Assessment and examinations
12
7
The requirements of the module include the solution of practical (professional) problems.
25
The learners are required to demonstrate that they are able to solve real practical problems.
Problem solving Assessment and examinations 35
The examinations or assessments include special problems from the subject matter of the module.
42
The students have to do written exercises about complex questions.
26
The assessment tasks require an extensive employment of cognitive or practical skills.
17
In the assessments the learners are confronted with novel requirements.
8
The learners have to demonstrate that they are able to evaluate the effectiveness of the solutions they have developed.
Creativity and innovation Assessment and examinations 30
The examinations and assessments allow for creative ideas and solutions.
3
The examinations and assessments require the students also to work out creative solutions to abstract problems.
4
In the assessments the learners are confronted with novel requirements that cannot be addressed with standard patterns from the course literature.
21
The module requires learners to develop new strategic approaches.
Autonomy Assessment and examinations 36
The module requires learners to make their own independent choices with regard to fundamental methods, tools, material and information for the solution of problems.
18
The module includes complex technical or professional requirements or projects.
27
The module requires learners to act independently and to show self-initiative.
43
The students are required to take responsibility for the fulfilment of work and learning assignments.
9
The students are required to design solutions autonomously in complex and changing environments.
13
Communicative competence Contents and learning methods 14
The students demonstrate that they can communicate their understanding of the field to fellow learners.
5
The students demonstrate that they can communicate their methods for problem solving to fellow learners.
32
The students are required to demonstrate that they are able to exchange opinions with specialists about information, ideas, problems and solutions.
23
The students are able to communicate information, ideas, problems and solutions to experts.
39
The students show that they are able to exchange opinions with laymen about information, ideas, problems and solutions.
Assessment and examinations 45
The requirements of the module include the task to communicate their conclusions, as well as the underpinning information and judgements, in a clear and unambiguous way that reflects the current state of research.
Consideration of social and ethical issues Assessment and examinations
14
29
The requirements of the module include reflection on ethical or social issues.
11
The assessments require learners to consider the ethical consequences of the theories, models and methods they have learned.
20
In problem solving the learners take into account the interests of others and show a sense of solidarity.
2
The assessments require learners to take into account ecological effects (e.g. sustainable use of resources, saving the environment).
38
The assessments include the analysis and scrutinization of social norms.
Please rate the learning outcomes of the module / unit by using the European Qualifications Framework (EQF)4. Place the module / unit on the level whose description approximates the description of the learning outcomes of the module / unit to the greatest extent. Please choose one level, even in cases where there is no complete matching between the level description and the module / unit.
1. Knowledge Which EQF level corresponds to the knowledge of the students? Level5:
Level
Description
1
- Basic general knowledge.
2
- Basic factual knowledge of a field of work or study.
3
- Knowledge of facts, principles, processes and general concepts in a field of work or study.
4
- Factual and theoretical knowledge in broad contexts within a field of work or study.
5
- Comprehensive, specialised, factual and theoretical knowledge within a field of study and awareness of the boundaries of that knowledge.
6
- Advanced knowledge of a field of work or study, involving a critical understanding of theories and principles.
7
- Highly specialised knowledge, some of which is at the forefront of knowledge in a field of work or study, as the basis for critical thinking. - Critical awareness of knowledge issues in a field and at the interface between different fields.
8
- Knowledge at the most advanced frontier of the field and at the interface between fields.
2. Skills Which EQF level corresponds to the skills of the students? Level:
Level
Description
1
- Basic skills required to carry out simple tasks.
2
- Basic cognitive and practical skills required to use relevant information in order to carry out tasks and solve routine problems using simple rules and tools.
3
- A range of cognitive and practical skills required to accomplish tasks and solve problems by selecting and applying basic methods, tools, materials and information.
4
- A range of cognitive and practical skills required to generate solutions to specific problems.
4 The level description is taken from the “Proposal for a RECOMMENDATION OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL on the establishment of the European Qualifications Framework for lifelong learning (presented by the Commission)”, COM(2006) 479 final, Brussels, 5th September 2006. 5 If there is uncertainty about two adjacent levels, you can choose e.g. 3,5 for the middle between 3 and 4.
15
5
- A comprehensive range of cognitive and practical skills required to develop creative solutions to abstract problems.
6
- Advanced skills, demonstrating mastery and innovation, required to solve complex and unpredictable problems in a specialised field of work or study.
7
- Specialised problem-solving skills required in research and/or innovation in order to develop new knowledge and procedures and to integrate knowledge from different fields.
8
- The most advanced and specialised skills and techniques, including synthesis and evaluation, required to solve critical problems in research and/ or innovation and to extend and redefine existing knowledge or professional practice.
3. Competence Which EQF level corresponds to the competence of the students? Level:
Level
Description
1
- Work or study under direct supervision in a structured context.
2
- Work or study under supervision with some autonomy.
3
- Take responsibility for completion of tasks in work or study. - Adapt own behaviour to circumstances in solving problems.
4
- Exercise self-management within the guidelines of work or study contexts that are usually predictable, but are subject to change. - Supervise the routine of work of others, taking some responsibility for the evaluation and improvement of work and study activities.
5
- Exercise management and supervision in contexts of work or study activities where there is unpredictable change. - Review and develop performance of self and others.
6
- Manage complex technical or professional activities or projects, taking responsibility for decision-making in unpredictable work or study contexts.
7
- Manage and transform work or study contexts that are complex, unpredictable and require new strategic approaches. - Take responsibility for contributing to professional knowledge and practice and/ or for reviewing the strategic performance of teams.
8
- Demonstrate substantial authority, innovation, autonomy, scholarly and professional integrity and sustained commitment to the development of new ideas or processes at the forefront of work or study contexts including research.
Many thanks for your support! © Dr. Wolfgang Müskens, Carl von Ossietzky Universität Oldenburg, 26111 Oldenburg – Reprint and duplication or any kind of copy (even of single items) prohibited, unless otherwise expressly agreed with the author.
16
The process of MLI assessment Modularisation In order to evaluate a comprehensive qualification with the help of the MLI, learning units (modules) need to be identified to which the MLI can be applied. A module is always a unity of specific learning phases on the one hand and tests that refer to the outcomes of these phases on the other. As a large part of the MLI items is related to tests and examinations, it is essential to identify the assessment methods of the qualification prior to the MLI review. Each test or examination has to be viewed in terms of the demands put on the learners, the grading systems and the weighting of the components. In the case of written tests anonymous examples of the learners’ answers should be reviewed.
Reference material MLI assessments are typically based on authentic documents related to the learning process and the tests or examinations. Some of these materials are prepared by the teachers or learners themselves and used within the teaching and learning process. The function of the documents is to give evidence of what was learned and how, and how it was tested. Documents from the teaching and learning process might include: ■ presentations or scripts by the teacher, ■ work sheets or written exercises, ■ learners’ answers to test assignments, ■ textbooks. Reference material related to tests and examinations might include: ■ test assignments (for written tests or exams), ■ grading systems (e.g. for oral examinations), ■ exams or papers written by the learners, ■ documentation of group work and/or project work, ■ documentation of case studies. The reference material is the source of information for the analysis of learning units by means of the MLI.
Identification of an expert evaluator The assessment of a learning unit by means of the MLI has to be carried out by an independent expert evaluator. This expert should be well acquainted with the subject matter of the module in question, that is, he or she should normally have a university degree in the discipline concerned. The expert evaluator should be nominated in cooperation with the institution that offers the qualification which is to be evaluated. He or she should be familiar with the teaching and learning methods of the educational sector to which the qualification in question belongs. However, the expert evaluator should not be teaching in the course or programme to be evaluated. In order to safeguard an independent and neutral evaluation, he or she should also not be employed with the institution offering the learning opportunity in question.
17
Problem solving It might happen that some MLI items cannot be clearly rated by the expert evaluator on the basis of the available reference material. In this situation the expert has the following options: ■ If the problem concerns only a few of the 51 MLI items, the expert may skip these items. The items not rated have no impact on the results of the MLI assessment. The MLI scales are then composed on the basis of the remaining items. ■ If several or all items of one MLI scale are concerned, the consequence is that the scale in question cannot be calculated. In this case the total MLI score is calculated on the basis of the remaining scales. ■ If essential information on a larger number of MLI items (related to more than one scale) is missing in the reference material, the expert evaluator should contact the provider of the qualification in question. There might be the possibility to supply additional reference material. In specific cases it is also possible to obtain the missing information by means of interviews with teachers or students of the qualification.
18
Results of the MLI The results of an MLI assessment can be visualised in a bar chart. The MLI diagrams first show the results of the 9 MLI scales. The bar on the right hand side represents the total MLI score (i.e. the level of the module). The values of the MLI scales are normalised. High values indicate that the scale in question is strongly realised. Senior business clerk (voc.) "Marke ng and Distribu on" 8 7 6 5
6,6 5,9 5,2
5,2
4,9
4,9 4,3
4
4,2
4,3 3,8
3 2 1 0
Figure 5. Results of the MLI as bar chart.
In the context of equivalence checks the MLI scores of those modules are reviewed that are to be accredited for or compared with another module. The usual precondition is that the level (the total MLI score) of the module that is considered for accreditation should not be more than 0.5 MLI points below the level of the module to be covered. In the example depicted in Figure 6. Comparison of the MLI profiles of a vocational and a higher education learning unit. the CVET module “Marketing and Sales” is compared with the university module “Marketing”. The aim of the comparison is an accreditation of the vocational module in the university programme. Learners who have completed the CVET programme and enter the university programme shall be exempted from the “Marketing” module. The contents of the vocational module “Marketing and Sales” differ significantly from those of the university module “Marketing”. While in the vocational module the MLI scales with the highest scores are “practice orientation” and “consideration of ethical and social issues”, the highest scores for the university module are found in the scales “critical understanding” and “autonomy”. In spite of the different profiles the total MLI scores of the two modules are close together. The CVET module reaches an MLI score of 4.9 and the bachelor module reaches 5.2. Accordingly, the difference between the wto modules is only 0.3 MLI points. Thanks to this small difference it was possible to recommend the CVET module for accreditation in the bachelor programme.
19
CVET module “Marketing and distibution”
BA module “Marketing”
BA Business Administra on "Marke ng"
Senior business clerk (voc.) "Marke ng and Distribu on" 8
8 7 6
7
6,6 5,9 5,2
5
5,2
6
4,9
4,9 4,3
4
4,2
4,3 3,8
6,0 5,2
5 4
3
3
2
2
1
1
0
0
5,8 5,0
5,3
5,0
5,5 5,0
5,2
3,8
Figure 6. Comparison of the MLI profiles of a vocational and a higher education learning unit.
The levels The total value of the MLI describes the overall level of a learning unit. Both the total value and the individual result scales can be used as a basis for the determination of an award of credit for a module. The MLI values are in line with the EQF scales. Thus, higher values mean a higher academic level. As a result of the hitherto conducted assessment it has to be expected that a clear assignment of an accurate standard for learning units cannot be realized for both Bachelor and Master programmes. Rather, there are level intervals that merge into one another. The results of a MLI assessment can be categorized into 5 different level intervals:
MLI total value < 3.5 The level of the assessed learning unit is considerably below that of a standard Bachelor course unit. Course units of this category should not be awarded with credit for Bachelor or Master programmes. The composition of the MLI scales indicates options for a restructuring of the learning unit which could lead to an up-rating of the MLI level. Such restructurings could include the content of the learning units, the methods for imparting knowledge and/or the method for assessing learning success.
Bachelor entr y level (3.5 < total MLI value < 4.5) The level of the assessed learning unit equates the level of typical course units during the first semesters of a Bachelor programme. Learning units of this category should only be awarded with credit if the combined total of the course units with Bachelor entry level (including the module for the award of credit) does not exceed 60 CPs. This module should not be awarded with credit for Master programmes.
20
Bachelor level (4.5 < total MLI value < 5) The level of the assessed learning unit equates that of a typical course unit during the intermediate phase of a Bachelor programme6. Provided there is an appropriate coverage of content, the learning unit should be awarded with credit for Bachelor programs. The learning unit should not be awarded with credit for Master programmes.
Bachelor-Master intermediate level (5 < total MLI value < 5.5) The level of the assessed learning unit equates that of an advanced Bachelor programme course unit or that of a typical course unit at the beginning of a Master programme. Therefore, an award of credit for Bachelor programmes is possible, provided there is an appropriate coverage of content. An award of credit for Master programmes should only be applied when the combined total of the course units with BachelorMaster intermediate level (including the module for the award of credit) does not exceed 30 CPs.
Master level (5.5 < total MLI value) The level of the assessed learning unit equates that of a standard Master course unit7. The learning units should, therefore, be awarded with credit for Bachelor and Master programmes, provided there is an appropriate coverage of content.
Competence orientation and accreditation
MLI results 8 7
Master level
Competence orientation
6 5
Ba-/Ma- intermediate level Bachelor level
4
Bachelor entry level
3 2 1
Unlimited accreditation for Bachelor and Master programmes Limited accreditation for Master programmes, unlimited accreditation for Bachelor programmes Unlimited accreditation for Bachelor programmes, no accreditation for Master programmes Limited accreditation for Bachelor programmes No accreditation possible
0 Figure 7. MLI scores as level of competence orientation.
6
N=227 Bachelor modules were reviewed by expert evaluators with the help of the MLI. The 25% percentile (1st quartile) was at 4.47, the 75% percentile (3rd quartile) at 5.46. 7 N=15 Master modules were reviewed by expert evaluators with the help of the MLI. The 25% percentile (1st quartile) was at 5.41, the 75% percentile (3rd quartile) at 6.37.
21
Quality criteria of the MLI Reliability In order to determine the reliability of the MLI scales, the internal consistencies of the scales (Cronbach’s alpha) were calculated. On the basis of N=84 module assessments, predominantly in the area of economics and business, medium to high scale reliabilities between α=.76 and α=.95 were achieved. Only the “problem solving” scale had an unsatisfactory internal consistency of α=.62 (Müskens & Gierke, 2009). Module Level Indicator Knowledge Broad and …, α=.87 Critical …, α=.95 Interdisciplinarity , α=.77 Skills
Competences Autonomy, α=.76 Communication, α=.81 Consideration of social and ethical issues , α=.89
Problem solving, α=.62 Practice orientation, α=.88 Innovation , α=.84
Figure 8. Internal consistencies of the MLI scales.
Validity One of the objectives of the MLI construction was to achieve a high correspondence between the MLI score and a direct positioning of the module in question within the European Qualifications Framework for lifelong learning (EQF). Accordingly, the correlation between the total MLI scores and the EQF levels of learning units can be interpreted as the construct validity of the MLI. In order to determine this correlation, ratings of the EQF levels of the learning units was carried out in addition to each of the MLI assessments conducted until now. These ratings were carried out by the expert evaluators subsequently to the MLI assessments. The experts were asked to rate the learning units with regard to the three EQF descriptors (knowledge, skills and competences). The arithmetic mean of these three ratings was interpreted as a direct EQF rating. Based on N= 546 assessments, the result was a Pearson correlation of r=.64 between the total MLI scores and the direct EQF ratings (see Fig. 9).
22
8,00
7,00
6,00
EQF
5,00
4,00
3,00
2,00
1,00 2,00
3,00
4,00
5,00
6,00
7,00
8,00
Total score MLI
Figure 9. Linear regression of EQF ratings on total MLI scores.
With regard to the EQF descriptor “knowledge” and the total MLI score a correlation of r=.51 was identified. The correlation between the descriptor “skills” and the total MLI score was r=.59. The strongest linear relationship was the one between the descriptor “competences” and the MLI score (r=.62). All correlations were found to be highly significant.
23
References Eilers-Schoof, A.; Müskens, W. (2013). Vom Äquivalenzvergleich zur allgemeinen Anrechnungsempfehlung: Eine Weiterentwicklung des Oldenburger Anrechnungsmodells. In: Hanft, A.; Brinkmann, K. (eds.), Offene Hochschulen - Die Neuausrichtung der Hochschulen auf Lebenslanges Lernen, Münster: Waxmann, pp. 248-257. Gierke, W.; Müskens, W. (2009). Der Module Level Indicator - ein Instrument für qualitätsgesicherte Verfahren der Anrechnung. In: Buhr, R.; Freitag, W.; Hartmann, E. A.; Loroff, C.; Minks, K.-H.; Mucke, K.; Stamm-Riemer, I. (eds.), Wege zwischen beruflicher und hochschulischer Bildung, Münster: Waxmann, pp. 134-136. Hanft, A.; Müskens, W. (2013). Anrechnung beruflicher Kompetenzen auf Hochschulstudiengänge: Ein Überblick. In: Hanft, A.; Brinkmann, K. (eds.), Offene Hochschulen - Die Neuausrichtung der Hochschulen auf Lebenslanges Lernen, Münster: Waxmann, pp. 223-234. Hartmann, E. A.; Stamm-Riemer, I. (2006). Die BMBF-Initiative “Anrechnung beruflicher Kompetenzen auf Hochschulstudiengänge” – Ein Beitrag zur Durchlässigkeit des deutschen Bildungssystems und zum Lebenslangen Lernen. Hochschule & Weiterbildung (1), pp. 52-60. Müskens, W. (2007). Anrechnung beruflicher Kompetenzen auf Hochschulstudiengänge - erste Ergebnisse des Modellprojektes “Qualifikationsverbund Nord-West”. In: Hortsch, H. (ed.), Innovationen für die Durchlässigkeit von Studiengängen, Dresdener Beiträge zur Berufspädagogik, 24, pp. 37-49. Müskens, W.; Eilers-Schoof, A. (2013). Neue Wege zwischen beruflicher und hochschulischer Bildung: Das Oldenburger Modell der Anrechnung in der Praxis. In: Hanft, A.; Brinkmann, K. (eds.), Offene Hochschulen – Die Neuausrichtung der Hochschulen auf Lebenslanges Lernen, Münster: Waxmann, pp. 235-247. Müskens, W.; Gierke, W. B. (2009). Gleichwertigkeit von beruflicher und hochschulischer Bildung. Report – Zeitschrift für Weiterbildungsforschung, 32(3), pp. 46-54. Müskens, W.; Tutschner, R.; Wittig, W. (2009). Improving permeability through equivalence Checks: An example from mechanical engineering in Germany. In: Tutschner, R.; Wittig, W.; Rami, J. (Eds.), Accreditation of Vocational Learning Outcomes – European Approaches to Enhance Permeability between Vocational and Higher Education, Impuls, 38, Bonn: BIBB, pp. 10-33.
24
Annex 1: Glossary “Knowledge” – Declarative and procedural/functional (application-oriented) knowledge. A summative term for all contents that are taught within a module. “Model” – A limited representation of (a specific part of) reality. The degree of formalisation may vary. “Theory” – A comprehensive and formally defined model for the description and explanation of a multitude of various phenomena. Theories allow to derive prognostic statements in the form of hypotheses. “Requirements” – All achievements that learners are expected to realise in the course of participating in a module. This may include voluntary teaching and learning activities, or activities for which no marks are given. “Assessment” – Any way of monitoring the performance and learning outcomes of learners. Apart from examinations the term also covers the evaluation of long-term assignments (e.g. project work, case studies, papers). The results may be expressed by numerical marks or by a ‘pass/fail’ distinction. “Examinations” – Temporally focused teaching and learning settings with the aim to record learning outcomes. Examinations may require oral, written or practical productions on the part of the candidates. The results may be expressed by numerical marks or by a ‘pass/fail’ distinction.
25
Annex 2: The scales of the MLI In the following the nine scales of the MLI and the contexts from which they were derived are described in detail. The items that constitute the scales are explained by annotations to the response options.
1. Broad and up-to-date knowledge Sources for the construction of the items EQR – Level 4 „Factual and theoretical knowledge [...] within a field of work or study...“ EQF level 5: “Comprehensive, specialised, factual and theoretical knowledge within a field of study...” EQF level 6: “Advanced knowledge of a field of work or study...” EQF level 7: “Highly specialised knowledge, some of which is at the forefront of knowledge in a field of work or study...” EHEA – First Cycle: “...knowledge and understanding in a field of study that builds upon their general secondary eduation, and is typically at a level that, whilst supported by advanced textbooks, includes some aspects that will be informed by knowledge of the forefront of their field of study” EHEA – Second Cycle: “...knowledge and understanding that is founded upon and extends and/or enhances that typically associated with the first cycle...”
26
Annotations to the items and response options Contents and learning methods 1
The level of the knowledge imparted exceeds the basic general knowledge of adults. What does “basic general knowledge” mean? – The term refers to the knowledge that can normally be expected from a person who has successfully completed the upper secondary level of general education. When should the response option “5 – fully correct” be chosen? 5 – The module is entirely imparting professional knowledge or specialist knowledge in a scientific discipline. When should the response option “3 – partially correct” be chosen? 3 – The module imparts partly professional/scientific knowledge and partly everyday knowledge or school knowledge. When should the response option “1 – not correct at all” be chosen? 1 – The module imparts only school knowledge or everyday knowledge.
47
The module includes the most important facts, principles, methods and general concepts of the subject matter. “the most important facts, principles, methods and general concepts of the subject matter” – The subject matter defined in the title of the module is covered in its entirety, meaning that there is a complete introduction to the relevant facts, principles, methods (procedures) and concepts. 5 – A comprehensive overview of the (module’s) subject matter is given. 3 – The core facts, principles, procedures and concepts are explained at least in brief. 1 – In the description of the subject matter, core elements are missing or described inadequately.
49
The module covers a broad range of theories in the field. “theories” – The term refers to the essential approaches, models or scientific theories underpinning the subject matter of the module. 5 – All relevant theoretical approaches (including contradicting ones, if applicable) are presented in a comprehensive way. 3 – At least some theoretical approaches are presented. 1 – Only one of the underpinning theoretical approaches (or none at all) is presented.
37
The module includes highly specialised knowledge. “highly specialised knowledge” – The term refers to the elaborateness and depth of the knowledge imparted. 5 – All instructing elements of the module include highly specialised knowledge. 3 – The module includes at least some parts or elements with highly specialised knowledge. 1 – No highly specialised knowledge is imparted.
27
10
The knowledge and understanding in this module are at least on the level of an advanced textbook. “advanced textbook” – The term refers to the type of textbook that is typically used in Bachelor programmes. The textbook summarises the state of the art concerning the subject matter of the module or a specific topic related to the same, and refers to current primary sources like research articles in refereed journals. In contrast, textbooks that are used in general schools usually do not comply with this definition. 5 – All of the knowledge imparted is at least on the level of an advanced textbook. 3 – At least a significant part of the knowledge is on the level of an advanced textbook. 1 – No knowledge at the level of an advanced textbook is imparted.
19
The module includes some current aspects that are not covered by the standard textbooks in the field. “some current aspects” – The term refers to aspects that are usually not covered by textbooks because they are taken e.g. from current research publications or media reports. 5 – Current aspects are identified and illustrated by appropriate material (articles, reports). 3 – Current aspects are mentioned in passing. 1 – There is no reference to current aspects at all.
28
The module includes some in-depth knowledge that reflects the current state of research in the field. “current state of research” – This term refers to the state of research as reflected in current primary sources like research articles in refereed journals. 5 – There are several occasions on which knowledge reflecting the current state of research is presented. 3 – There are at least hints to relatively recent research findings. 1 – In the documents there is no reference to current research findings.
44
The module also covers some of the latest findings in the area, drawing for example on current research papers. “current research papers” – The current state of research is typically represented by primary sources, i.e. research articles published in academic journals that are listed e.g. in the “Journal Citation Reports”. Popular magazines, newspapers or trade magazines usually do not publish primary articles that would represent the current state of research. 5 – There are clear and unambiguous references to current primary sources. The current state of research with regard to the topics in question is explained in detail. 3 – There are at least hints to current primary sources (e.g. in the bibliographical annex to the course text). 1 – There is no reference to current research papers.
Assessment and examinations 51
The assessment within the module expects students to have knowledge of at least some theories or models in the field. 5 – The learning requirements explicitly include knowledge of a number of theories or models from the subject matter of the module, or implicitly require the application, transfer or reflection of such theories or models. 3 – The learning requirements are at least connected to the theories or models related to the subject of the module. 1 – The learning requirements do not refer to knowledge of theories or models.
28
2. Critical understanding Sources for the construction of items EQF level 5: “...awareness of the boundaries of that knowledge.” EQF level 6: “...a critical understanding of theories and principles.” EQF level 7: “Critical awareness of knowledge issues in a field...”
Annotations to the items and response options Contents and learning methods 15
The module gives an account of the observations or research findings on which the theories and models presented are based. “observations or research findings” – In the case of theories or models generated by induction it is possible to describe e.g. the genesis, i.e. the observations from which the theories or models were inferred. In the case of deductive theories or models the focus is on findings or observations that are used to test their hypotheses. 5 – Observations or research findings that support the theories and models in question are described in detail. 3 – There are at least hints to empirical foundations or empirical tests of the theories and models presented. 2 – There are at least references to the literature concerning the empirical foundations or tests of the theories and models presented. 1 – In the course materials there is no reference to observations or empirical findings on which the theories and models presented are based (or no theories and models are presented at all).
48
The module gives at least some suggestion as to how the theories and methods were developed and empirically tested. “empirically tested” – Typically hypotheses are derived from theories and tested on the basis of empirical observations or measurements. In the case of methods the success or benefit of their application is evaluated empirically. 5 – The presentation of theories and methods is always accompanied by background information concerning their genesis and information about relevant studies that empirically test the theories. The learners intensively deal with the issue of the empirical foundation of the theories in question. 3 – Parts of the course material discuss the empirical foundation of the methods or theories presented. 1 – No theories and methods are imparted, or their genesis and empirical foundation is not covered.
29
46
The module presents alternative approaches or theories which contradict each other. “approaches which contradict each other” – In many scientific disciplines different models of explanation or alternative methods compete with each other. The analysis of different approaches aims to help the learners achieve their own position in the professional discourse by applying appropriate judgement criteria. 5 – Alternative theories or approaches are presented for at least one of the topics covered by the module. The fact that the approaches contradict each other is obvious or explicitly mentioned. 3 – Alternative approaches are presented for at least one topic. The learners can infer from the presentation that the approaches lead to different conclusions with regard to some questions or problems. 1 – There is no passage in the course materials that would mention alternative approaches. Not even the bibliography refers to approaches that contradict each other.
24
The module gives an idea of those topics or subject matters to which the theories and models presented cannot be applied. “cannot be applied” – The scope of the theories or models is at stake here. Which phenomena can be explained by the approach, and which cannot? Which areas of application are exemplified by empirical studies? 5 – The potential applications and the scope of the relevant theories and models are clearly described. 3 – There are at least hints to problems related to the applicability of the approaches presented. 1 – The potential applications and the scope of the theories and models in question are not covered, or no theories and models are presented at all.
6
The module raises awareness of the boundaries of the knowledge imparted. “boundaries of the knowledge imparted” – The limits of knowledge are determined by the current state of research or development in an area. In science, knowledge is always considered provisional and revisable in principle. In order to develop a sense of the limits of knowledge, the learners need to have basics that enable them to reflect critically the state of research concerning a given topic. 5 – The learners reflect intensively on the limits of the knowledge imparted. 3 – The learners are at least given explicit hints to the limits of the knowledge imparted. 1 – The limits of the knowledge imparted are not treated.
33
The module imparts a critical understanding of the theories and principles of the field. “critical understanding” – The term refers to the awareness that theories and principles in science must always be regarded as provisional and revisable, that the scope of theories and research approaches is limited, that research approaches reflect (explicit and/or tacit) a priori assumptions and the anthropology or world view of the actors involved, that conclusions from these approaches may be wrong, that alternative theories may be capable of explaining the same facts in a more efficient way or predict the same phenomena with greater reliability, etc. 5 – In the presentation of the theories, critical positions on the latter are always duly considered. The fundamental assumptions of the discipline are critically reflected. The learners have the task to scrutinize theories and principles, and to give an (oral or written) account of the limitations of these theories. 3 – The teaching material of the module takes into account at least some critical positions on some of the theories or models imparted. The learners are expected to be familiar with these critical positions. 1 – The teaching material does not refer to critical positions on the theories and models imparted. Learners are not expected to critically reflect the theories or principles imparted.
30
Assessment and examinations 40
The examinations and assessments in the module require learners to critically reflect the theories and principles of the field. “critical reflection” – The term refers to an oral or written discussion concerning the empirical evidence of the validity of the theory and its implications, the underpinning anthropology or world wiew, possible (negative) effects of applying the theory to specific problems, and/or alternative theories and explanations. 5 – In the course of the tests/assessments the learners are explicitly required to critically reflect (orally or in writing) the theories and principles of the discipline or domain. 3 – The tests/assessments give the learners at least the opportunity to critically reflect the theories and principles of the discipline or domain. 1 – The assessment procedures do not require learners to critically reflect the theories and principles of the discipline or no theories and principles are imparted.
50
The examinations or assessments also refer to the historical context in which theories, models or techniques evolved. “historical context” – Era in which the theoretical approach evolved; persons who developed the approach; state of research at the time when the approach evolved; reception of the approach at that time. 5 – In the course of the tests/assessments the learners are explicitly required to describe and reflect the historical context of theories, models or techniques related to the subject matter of the module. 3 – The tests/assessment presuppose knowledge of the historical context of some of the approaches related to the subject matter. 1 – The historical context of the theories, models or techniques imparted does not play a role in the tests/assessments or the module does not impart any theories, models or techniques.
31
3. Interdisciplinarity Sources for the construction of items EQF level 6: “...integrate knowledge from different fields.” EQF level 7: “Critical awareness of knowledge issues [...] at the interface between different fields”
Annotations to the items and response options Contents and learning methods 22
The module includes references to other disciplines or occupations. “other disciplines or occupations” – The term refers to disciplines or occupations that are not part of the main subject of the qualification to be achieved by the learners. Typically these are disciplines or occupations that learners will have to cooperate with after completing their qualification. 5 – The module continuously (i.e. on many occasions) refers to other disciplines or occupations. 3 – The module at least sometimes refers to other disciplines or occupations. 1 – There are no references to other disciplines or occupations at all.
12
The module includes interdisciplinary topics that have to be addressed on the basis of knowledge from different disciplines. “different disciplines” – The term refers to areas of research and development that use to operate separately, i.e. the scientists or developers concerned usually do not cooperate across these areas. When the module was composed, research findings and knowledge from disciplines that normally operate seperately were linked. 5 – It is obvious that essential topics of the module are treated by means of a combination of knowledge, approaches or findings from clearly distinct disciplines. 3 – The approaches, methods or findings described in the course material of the module are clearly originating from different disciplines. 1 – All of the described approaches, methods or findings originate from the same discipline or no approaches, methods or findings are described at all.
31
The module informs the learners that the meaning of terms may vary between different disciplines or contexts. “meaning of terms” – Many technical terms and concepts (e.g. “competence’, ‘network”, “validation”, “consciousness”, “structure”) have different meanings in different academic or vocational disciplines. If such differences are not made the subject of a discussion, the interdisciplinary exchange may become difficult or impossible. 5 – Different meanings of terms are discussed in detail. 3 – There is at least one passage in the course material where the different meaning of terms is explicitly treated. 1 – The possible differences in the meanings of terms are not taken into account.
32
Assessment and examinations 13
The learners are required to present their solutions and ideas in a manner that is understandable for members of other professions or disciplines. “understandable” – The understandability can be improved e.g. by doing without technical terms if possible, by visualising facts and relationships, by explaining the background that is necessary for understanding, and/or by reducing the complexity of results or conclusions with the help of simplified presentations. 5 – The problem of understandability for other disciplines is explicitly discussed in the course material. In addition, there are learning activities or assessment tasks that require learners to present their ideas or solutions in a way that is understandable for members of other professions or disciplines (e.g. presentations in classes or courses from other disciplines). 3 – The problem of understandability for members of other professions or disciplines is discussed at least occasionally. 1 – The problem of understandability for members of other professions or disciplines is not discussed at all or is not relevant for the subject matter of the module.
33
4. Practice orientation Sources for the construction of items EHEA – First Cycle: “Can apply their knowledge and understanding in a manner that indicates a professional approach to their work or vocation...”
Annotations to the items and response options Contents and learning methods 34
The theories and models that have to be learned in the module are explained at the example of practical situations.
16
The module imparts knowledge that is immediately applicable in practice. “knowledge immediately applicable in practice” – This may include factual knowledge that is applicable in practice, but also models or theories that can be used to solve practical problems. 5 – The impartation of knowledge that is immediately applicable in practice is in the focus of the module. 3 – There are at least some passages in the learning materials where knowledge immediately applicable in practice is presented. 1 – The module does not include any knowledge immediately applicable in practice.
41
The module transfers skills that are immediately applicable in practice. “skills that are immediately applicable in practice” – In contrast to item 16 this item is explicitly concerned with skills (i.e. functional knowledge) only. These may include practical or cognitive skills. In either case concrete steps or techniques should be described that can be performed by the learners in the application of the skills in question. 5 – The module includes the impartation of different skills. These are described explicitly. The learners exercise the application of these skills in concrete activities. 3 – Cognitive and/or practical skills are at least explained. 1 – No cognitive or practical skills that are relevant for the topic of the module are described.
Assessment and examinations 7
The requirements of the module include the solution of practical (professional) problems. “solution of practical (professional) problems” – The term refers to written, oral or manual/practical achievements addressing concrete problems. The problems must be related to typical requirements of professional practice. However, they do not necessarily have to be authentic cases. “requirements of the module” – The term refers to tests (with marks) or learning activities (without marks) that are expected from the learners. 5 – The solution of practical problems is in the focus of the learning activities or test assignments. 3 – The solution of practical problems constitutes at least a small part of the learning activities or test assignments. 1 – None of the learning activities or test assignments is concerned with the solution of practical problems.
34
25
The learners are required to demonstrate that they are able to solve real practical problems. “real practical problems” – In contrast to item 7, this item refers to authentic cases, i.e. cases that are not made up or fictitious. “required to demonstrate” – The learners have the task to actually solve the problems (i.e. produce a concrete solution and not just give an account of how a solution could be achieved). 5 – The solution of real practical problems constitutes a significant part of the learning requirements. 3 – The solution of real practical problems constitutes at least a minor part of the learning requirements. 1 – The learning requirements of the module do not include the solution of real (authentic) practical problems.
35
5. Problem solving Sources for the construction of items EQR – level 4: „A range of cognitive and practical skills required to generate solutions to specific problems.“ EQF level 5: “A comprehensive range of cognitive and practical skills required to develop creative solutions to abstract problems.” EQF level 6: “Advanced skills [...] required to solve complex and unpredictable problems in a specialised field of work or study.” EQF level 7: “Specialised problem-solving skills required in research and/or innovation...” EHEA – First Cycle: “...sustaining arguments and solving problems within their field of study;” EHEA – Second Cycle: “...problem solving abilities in new or unfamiliar environments within broader (or multidisciplinary) contexts related to their field of study;”
36
Annotations to the items and response options Assessment and examinations 35
The examinations or assessments include special problems from the subject matter of the module. “special problems” – The term refers to special cases, exceptions or contents that go beyond the ordinary situations in which the methods covered by the module are applied. 5 – The demands put on the learners clearly include special problems whose solution goes significantly beyond the ordinary application of methods or knowledge of the subject matter. 3 – The problems described in the learning requirements include at least hints to special problems. 1 – The learning requirements do not include special problems or do not include problem solving at all.
42
The students have to do written exercises about complex questions. “complex questions” – The term refers to questions that make it necessary to take a multitude of factors into account. The learners are required to consider and describe the interaction of these factors in writing. 5 – The learning requirements include comprehensive, complex tasks that must be answered in writing. 3 – The learning requirements include tasks that have to be answered in writing, are longer than one paragraph and require at least the consideration of more than one factor. 1 – The learning requirements include either no written tasks at all or only questions that can be answered with single sentences or keywords.
26
The assessment tasks require an extensive employment of cognitive or practical skills. “extensive employment” – The learners need to use a broad range of skills or an adequate sample from a broad range of skills when they work on their assignments. “cognitive skills” – The term refers to methods, processes, algorithms and structures for working on a given class of problems. The work on these problems can take place via written or spoken productions, too. “practical skills” – The term refers to skills that go beyond basic key skills (e.g. reading, writing, communication, calculation…) and which are usually acquired or optimised in the module. 5 – The impartation of skills is of paramount importance in the module. Learners are required to demonstrate in the assessments (tests) that they have a broad range of skills at their disposal. 3 – In order to work on the tests successfully, learners need to employ cognitive or practical skills in the sense mentioned above. 1 – The tests do not require the learners to employ specific skills.
17
In the assessments the learners are confronted with novel requirements. “novel requirements” – The assessments/tests require learners to work on tasks or problems which were previously not addressed this way in class. For instance, the learners might be confronted with a need for knowledge transfer that has a new quality, or with a higher level of difficulty. “Novel requirements” are also inherent in project tasks whose requirements can be covered in class only partly or not at all. 5 – The tests frequently and unambiguously confront the learners with demands that are qualitatively new in comparison to the demands that were familiar to the learners before. 3 – The requirements of the tests include at least single aspects that are new to the learners. 1 – The requirements of the tests correspond fully to the contents and exercises of the previous instruction, i.e. they require only the repetition or application of familiar knowledge and methods.
37
8
The learners have to demonstrate that they are able to evaluate the effectiveness of the solutions they have developed. “evaluate the effectiveness” – In the case of complex (practice oriented) problems the (relative) effectiveness of scientifically based solutions is often difficult to identify. Therefore learners need to develop their own evaluation strategies with a view to checking the effectiveness of the proposed solution (and the absence of side-effects). 5 – The learners know approaches and procedures for the evaluation of their solutions and are encouraged by the learning requirements to develop a clear strategy for evaluating the effectiveness of their solutions. (It is not necessary that the evaluation strategy is actually put into practice.) 3 – The need to evaluate the effectiveness of solutions is discussed in the learning materials. There are at least exercises that refer to the application of evaluation procedures. 1 – The evaluation of solutions is not discussed or the module does not include practical problems that would require solutions and their evaluation.
38
6. Creativity and innovation Sources for the construction of items EQF level 5: “A comprehensive range of cognitive and practical skills required to develop creative solutions to abstract problems.” EQF level 6: “Advanced skills, demonstrating mastery and innovation, ...” EQF level 7: “Specialised problem-solving skills required in research and/or innovation ...” EHEA – Second Cycle: “...knowledge and understanding [...] that provides a basis or opportunity for originality in developing and/or applying ideas, often within a research context;”
Annotations to the items and response options Assessment and examinations 30
The examinations and assessments allow for creative ideas and solutions. “creative ideas and solutions” – The learners have the opportunity e.g. to select knowledge and/or methods independently and to use them creatively. 5 – The test assignments explicitly encourage learners to develop creative solutions and ideas. Creative solutions are appreciated in the grading of answers. 3 – The tests at least allow for creative activities of the learners. The structure of the assignments is not so rigid that only one specific approach would lead to a solution. 1 – No creativity whatsoever is required for working on the tests. Learners only need to repeat knowledge or to apply structured approaches (e.g. algorithms, methods, skills).
3
The examinations and assessments require the students also to work out creative solutions to abstract problems. “abstract problems” – The term refers to questions at the level of theories or models as opposed to problems at the manifest level of single practical tasks. The approaches to solving abstract problems have implications for various contexts of application. 5 – The tests require learners to deal with abstract problems at the level of theories or models. The learners have the opportunity to arrive at new solutions, e.g. by means of the creative combination of different approaches, and to reflect on the implications that their solutions might have for different practical contexts. 3 – The test assignments and answers at least touch abstract problems at the level of theories or models. The learners have the opportunity to work creatively on their solutions. 1 – The test assignments and the answers of the learners do not touch abstract problems at the level of theories and models or the problems in question can be solved only by means of routines.
39
4
In the assessments the learners are confronted with novel requirements that cannot be addressed with standard patterns from the course literature. “novel requirements” – The term refers to requirements that go beyond the tasks to be fulfilled in class or in self-learning phases, or which include elements that are qualitatively new for the learners (e.g. transfer to a new area of application or a link between two or more areas of knowledge that were only treated separately before). “that cannot be addressed with standard patterns” – To be successful on their tasks, it is not enough for the learners to apply the techniques, methods or procedures they have learned to the new requirements, or to repeat the knowledge they have acquired. 5 – The requirements of the test assignments clearly have a new quality compared to the demands put on the learners before. The learners have to develop a new approach (procedure) in order to master the tasks. 3 – The requirements of the test assignments include at least single aspects that differ significantly from the previous requirements faced by the learners (e.g. exercises). 1 – The requirements of the test assignments correspond to the examples and exercises already covered in class. The tests can be mastered by applying familiar knowledge, methods or procedures.
21
The module requires learners to develop new strategic approaches. “strategic approaches” – The term refers to long-term planning, e.g. for institutions, enterprises, departments, business divisions, policy areas, economic sectors or academic disciplines. 5 – The learners have the explicit task, taking into account the knowledge they have acquired, to revise existing strategic approaches or to develop new ones. 3 – The learners have the task to apply the imparted approaches or methods to practical topics or problems. Elements of strategic planning are touched. 1 – The learners do not have the task to deal with strategic approaches.
40
7. Autonomy Sources for the construction of items EQR – level 4: „Exercise self-management within the guidelines of work or study contexts that are usually predictable, but are subject to change...“ EQF level 5: “Exercise management and supervision in contexts of work or study activities where there is unpredictable change.” EQF level 6: “Manage complex technical or professional activities or projects, taking responsibility for decision-making in unpredictable work or study contexts” EQF level 7: “Manage and transform work or study contexts that are complex, unpredictable and require new strategic approaches.” EHEA – Second Cycle: „...have the learning skills to allow them to continue to study in a manner that may be largely self-directed or autonomous.“
Annotations to the items and response options Assessment and examinations 36
The module requires learners to make their own independent choices with regard to fundamental methods, tools, material and information for the solution of problems. “solution of problems” – The term refers to the work on assignments (written, oral or practical). 5 – The learners work on comprehensive tasks which frequently require them to select, on their own, appropriate methods, tools, material and information. 3 – The tasks require learners at least occasionally to make independent decisions concerning the choice of methods, tools, material and/or information. 1 – The learning requirements do not include the solution of problems or allow for the solution of problems through the routine application of preset approaches.
41
18
The module includes complex technical or professional requirements or projects. “projects” – The term refers to long-term learning or work assignments that are fulfilled by single learners or learning groups. The learners themselves take at least a part of the responsibility for the selection of methods and resources, and they have the opportunity for creative activity. In the context of the project learners work on a self-defined or assigned task, and the process or outcome is documented or presented by the learners. “complex technical/professional requirements” – The term refers to tasks, exercises or tests which make it necessary to consider a variety of factors and which can only be mastered by employing diverse methods, knowledge or other resources. 5 – The learning requirements are unambiguously complex. They require learners in manifold ways to think and act independently. 3 – The learning requirements include at least exercises or small projects in which learners have the task to determine their activities on their own and to choose appropriate methods or materials. 1 – Independent decisions or activities on the part of the learners are not required at all. The learning requirements can be mastered by clearly identifiable knowledge or methods.
27 “self-initiative” – In order to work on their assignments, the learners have to access resources independently (e.g. search for appropriate methods or literature, or contact interviewees or respondents in a sample). The work steps that are necessary to fulfil the tasks are not completely predetermined by the teachers. 5 – The learners act independently for a longer period of time and to a large extent. They make their own decisions concerning activities and the use of resources. Self-initiative on the part of the learners is absolutely necessary for mastering the tasks. 3 – The learning requirements give the learners some degree of independence within certain limits. The learners have the opportunity to develop self-initiative at least with regard to some aspects of their activity. 1 – The assignments can be mastered without self-intiative with the help of already available resources and routine methods. 43 “take responsibility for the fulfilment of work and learning assignments” – Responsibility means that the non-fulfilment or inadequate fulfilment of the requirements leads to negative effects for third parties or with regard to resources. These effects go beyond the direct effects on the learners themselves (e.g. failure with regard to the course objectives). Responsibility is taken, for instance, in the context of group work, but also through the undertaking to prepare a presentation or in the case of projects whose success or failure has consequences for third parties. 5 – The learners are required to take a high degree of responsibility. If a learner is not successful there are considerable effects for which the learner is held accountable. 3 – The learners are required to take some responsibility. If duties are not fulfilled there are at least minor effects (e.g. for other learners). 1 – The fulfilment or non-fulfilment of learning requirements has no effect on third parties or with regard to resources. Learners are not expected to take responsibility.
42
9
The students are required to design solutions autonomously in complex and changing environments. “in complex and changing environments” – The term refers to learning environments whose requirements are not already certain when the learner starts to work. Examples might be complex simulations, practical projects or research activities where the intermediate results have consequences for the following work steps. “design solutions autonomously” – The learners have the task to develop their own approaches, procedures or methods in order to act successfully within the learning environments. 5 – The learners act in complex and changing learning environments over a longer period of time. They influence several parameters and independently develop solutions that require creativity and self-initiative. 3 – The learning environments are complex and include at least some changing elements that prompt learners to adapt their strategies. The learners have at least the opportunity to follow their own ideas and to develop their own strategies to a limited extent. 1 – The learning environments do not include dynamic elements or are limited to a few parameters that can be controlled or learners are not expected to perform design activities.
43
8. Communicative competence Sources for the construction of items EHEA – First Cycle: “Can communicate information, ideas, problems and solutions to both specialist and non-specialist audiences;” EHEA – Second Cycle: “Can communicate their conclusions, and the knowledge and rationale underpinning these, to specialist and non-specialist audiences clearly and unambiguously;”
Annotations to the items and response options Contents and learning methods 14
The students demonstrate that they can communicate their understanding of the field to fellow learners. “their understanding of the field” – The learners communicate among each other about issues, subjects or problems in their field of knowledge (e.g. in the context of peer teaching, in classroom discussions or by presentations held by the learners). 5 – The communication with peer learners about one’s own understanding of the subject matter is explicitly required and an essential part of the learning requirements in the module. 3 – The learners communicate about their understanding of the field at least in the course of some exercises or classroom discussions. 1 – Neither the classroom teaching nor the tests/assessments involve the opportunity for learners to communicate about their understanding of the field.
5
The students demonstrate that they can communicate their methods for problem solving to fellow learners. “their methods for problem solving” – The term refers to the activities related to the work on practical or professional tasks, e.g. application of techniques, instruments or methods. 5 – There are tests or exercises in which the learners are required to give an (oral or written) description of how they proceed in problem solving. The explicit intention is that the description be understandable for fellow learners. The fellow learners have the opportunity to ask questions and make comments. 3 – The learning requirements ask the learners to describe how they proceed in problem solving. It is at least possible that the other learners understand the description. 1 – The learners do not solve problems or do not present their problem solving activities to other learners.
44
32
The students are required to demonstrate that they are able to exchange opinions with specialists about information, ideas, problems and solutions. “with specialists” – The term refers to persons who are qualified in the discipline or occupation to which the module belongs. In most cases this is the teacher/lecturer responsible for the module. 5 – It is an explicit part of the learning requirements of the module that the learners discuss with the lecturer/teacher or other specialists about information, ideas, problems and solutions (orally or in writing). All learners have to demonstrate that they are able to argue in an adequate way. 3 – The learners engage in problem solving and the learning setting offers at least some opportunities to communicate with teachers or other specialists about information, ideas, problems or solutions. 1 – The learners do not work on practical/professional problems or the learning setting does not allow for communication with teachers or other specialists.
23
The students are able to communicate information, ideas, problems and solutions to experts. “experts” – The term refers to persons who take part in generating knowledge in the subject matter of the module or related disciplines (e.g. scientists). The lecturer responsible for the module might be an expert as well. 5 – The (oral or written) communication with experts is part of the learning requirements of the module. The expert gives the learners feedback concerning the clarity and appropriateness of their description of information, ideas, problems and solutions. 3 – The learners have the task to describe information, ideas, problems and solutions in a way (orally or in writing) that is adequate for communication with experts. 1 – Learners are not expected to be able to describe information, ideas, problems and solutions in a way that is adequate for communication with experts.
39
The students show that they are able to exchange opinions with laymen about information, ideas, problems and solutions. “laymen” – The term refers to persons who are not familiar with the contents and terminology of the module, e.g. members of other occupations or disciplines, Learners from other disciplines or occupations or students from lower semesters/grades can be invited as laymen for the purpose of exercises. 5 – There are tests that explicitly aim to assess the learners’ ability to describe technical approaches, ideas, problems and solutions in a way that is understandable for laymen. 3 – The tests at least bring the learners in contact with laymen. The test assignments frequently prompt learners to exchange information with laymen about technical issues or approaches. 1 – The tests do not require any (written or oral) communication of the learners with laymen.
45
Assessment and examinations 45
The requirements of the module include the task to communicate their conclusions, as well as the underpinning information and judgements, in a clear and unambiguous way that reflects the current state of research. “the current state of research” – The term refers to a level that corresponds to the current scientific discourse in primary journals. Primary journals are scientific publications that are listed e.g. in the SSCI or the SCI. 5 – The learners have the explicit task to communicate (orally or in writing) their conclusions as well as the underlying facts and arguments at the level of the current state of research. The adequacy, clarity and unequivocalness of the learners’ account is rated by a researcher/scientist from the discipline in question. 3 – The learners’ accounts largely reflect the current state of research in the field covered by the module. The learners refer at least to some relatively recent approaches or findings. 1 – The learners are not expected to occupy themselves with current primary literature and to take it into account in their reasoning.
46
9. Consideration of social and ethical issues Sources for the construction of items EHEA – First Cycle: “Have the ability to gather and interpret relevant data (usually within their field of study) to inform judgements that include reflection on relevant social, scientific or ethical issues;” EHEA – Second Cycle: “Have the abiltiy to integrate knowledge and handle complexity, and formulate judgements with incomplete or limited information, but that include reflecting on social and ethical responsibilities linked to the application of their knowledge and judgements;”
Annotations to the items and response options Assessment and examinations 29 The requirements of the module include reflection on ethical or social issues. “reflection on ethical or social issues” – Examples might be the consideration of the possible effects of new developments in the field (technology assessment), issues of discrimination against particular social groups through approaches in the area, reflection on moral standards for the profession, or ethically relevant effects of research activities, e.g. experiments. 5 – In the context of the learning requirements (i.e. in the exercises or tests) the learners have the clear task to reflect on ethical and social issues. The module includes enough opportunity and time for an adequate consideration of these issues. 3 – The learning requirements at least imply a reflection on ethical and social issues. The reflection does not take place in a passive way only (e.g. by reading course material or listening to lectures). 1 – The learners are not required to reflect actively on ethical or social issues in the field covered by the module. 11 The assessments require learners to consider the ethical consequences of the theories, models and methods they have learned. “ethical consequences” – The term refers to effects on third parties, unintended effects, long-term effects, risks, ecological effects, effects on societal processes etc. 5 – The tests explicitly require learners that they consider the ethical consequences of the approaches and methods they have learned or the consideration of the ethical consequences is an explicit part of the grading system to which the tests are subject. 3 – In the tests, learners are at least implicitly required to consider also the ethical consequences of the approaches and methods learned. 1 – The tests do not require learners to consider the ethical consequences of the approaches and methods they have learned.
47
20
In problem solving the learners take into account the interests of others and show a sense of solidarity. “solidarity” – In the module, learners have not only the task to reflect on the ethical implications of the approaches and methods learned but also the task to take the same into consideration in problem solving. For instance, an assignment may include the requirement that solutions should be developed with the smallest possible negative effects on others, or that the evaluation of alternative solutions should take into account the effects on third parties as a relevant criterion. 5 – Consideration of others and solidarity with people affected play a significant role in all problem solving activities of the learners, and are explicitly demanded from learners. 3 – Consideration of others and solidarity with people affected are mandatory at least in some of the problem solving activities or the said attitudes are taken into account in the grading of the learning outcomes. 1 – Consideration of others and solidarity with people affected are not mandatory for the problem solving activities or the learners do not have the task to solve problems.
2
The assessments require learners to take into account ecological effects (e.g. sustainable use of resources, saving the environment). “ecological effects” – The term refers to effects on the environment, nature and/or resources. These effects are viewed not from the perspective of the enterprise, the clients or the economic sector but with a sense of responsibility to the society as a whole. 5 – The tests explictly require the learners to consider and describe ecological consequences, too. 3 – The consideration of ecological effects is at least an implicit part of the test assignments or a an implicit part of the grading system. 1 – The consideration of ecological effects does not play a role in the test assignments.
38
The assessments include the analysis and scrutinization of social norms. “social norms” – The term refers to the expectations within a society towards the (social) behaviour of individuals. Norms depend on societal and cultural conditions, that is, they differ from one society to the next and are subject to social evolution. It is assumed that the models and theories learned in the module have implications for the question as to whether certain norms are adequate. Given these implications, the learners should analyse and critically examine existing social norms. 5 – In the tests the learners have the explicit task to analyse the implications that the approaches and research findings learned as well as their own conclusions have on social norms, and to assess the appropriateness of these norms. 3 – The test assignments or the grading systems of the tests at least imply that an analysis and critical examination of social norms is expected from the learners. 1 – The tests do not involve an analysis of social norms on the part of the learners.
48
