July 2006 Volume 9 Number 3

July 2006 Volume 9 Number 3

Educational Technology & Society An International Journal Aims and Scope Educational Technology & Society is a quarterly journal published in January, April, July and October. Educational Technology & Society seeks academic articles on the issues affecting the developers of educational systems and educators who implement and manage such systems. The articles should discuss the perspectives of both communities and their relation to each other: • Educators aim to use technology to enhance individual learning as well as to achieve widespread education and expect the technology to blend with their individual approach to instruction. However, most educators are not fully aware of the benefits that may be obtained by proactively harnessing the available technologies and how they might be able to influence further developments through systematic feedback and suggestions. • Educational system developers and artificial intelligence (AI) researchers are sometimes unaware of the needs and requirements of typical teachers, with a possible exception of those in the computer science domain. In transferring the notion of a 'user' from the humancomputer interaction studies and assigning it to the 'student', the educator's role as the 'implementer/ manager/ user' of the technology has been forgotten. The aim of the journal is to help them better understand each other's role in the overall process of education and how they may support each other. The articles should be original, unpublished, and not in consideration for publication elsewhere at the time of submission to Educational Technology & Society and three months thereafter. The scope of the journal is broad. Following list of topics is considered to be within the scope of the journal: Architectures for Educational Technology Systems, Computer-Mediated Communication, Cooperative/ Collaborative Learning and Environments, Cultural Issues in Educational System development, Didactic/ Pedagogical Issues and Teaching/Learning Strategies, Distance Education/Learning, Distance Learning Systems, Distributed Learning Environments, Educational Multimedia, Evaluation, HumanComputer Interface (HCI) Issues, Hypermedia Systems/ Applications, Intelligent Learning/ Tutoring Environments, Interactive Learning Environments, Learning by Doing, Methodologies for Development of Educational Technology Systems, Multimedia Systems/ Applications, Network-Based Learning Environments, Online Education, Simulations for Learning, Web Based Instruction/ Training

Editors Kinshuk, Massey University, New Zealand; Demetrios G Sampson, University of Piraeus & ITI-CERTH, Greece; Ashok Patel, CAL Research & Software Engineering Centre, UK; Reinhard Oppermann, Fraunhofer Institut Angewandte Informationstechnik, Germany.

Associate editors Alexandra I. Cristea, Technical University Eindhoven, The Netherlands; John Eklund, Access Australia Co-operative Multimedia Centre, Australia; Vladimir A Fomichov, K. E. Tsiolkovsky Russian State Tech Univ, Russia; Olga S Fomichova, Studio "Culture, Ecology, and Foreign Languages", Russia; Piet Kommers, University of Twente, The Netherlands; Chul-Hwan Lee, Inchon National University of Education, Korea; Brent Muirhead, University of Phoenix Online, USA; Erkki Sutinen, University of Joensuu, Finland; Vladimir Uskov, Bradley University, USA.

Advisory board Ignacio Aedo, Universidad Carlos III de Madrid, Spain; Sherman Alpert, IBM T.J. Watson Research Center, USA; Alfred Bork, University of California, Irvine, USA; Rosa Maria Bottino, Consiglio Nazionale delle Ricerche, Italy; Mark Bullen, University of British Columbia, Canada; Tak-Wai Chan, National Central University, Taiwan; Nian-Shing Chen, National Sun Yat-sen University, Taiwan; Darina Dicheva, Winston-Salem State University, USA; Brian Garner, Deakin University, Australia; Roger Hartley, Leeds University, UK; Harald Haugen, Høgskolen Stord/Haugesund, Norway; J R Isaac, National Institute of Information Technology, India; Paul Kirschner, Open University of the Netherlands, The Netherlands; William Klemm, Texas A&M University, USA; Rob Koper, Open University of the Netherlands, The Netherlands; Ruddy Lelouche, Universite Laval, Canada; Rory McGreal, Athabasca University, Canada; David Merrill, Brigham Young University - Hawaii, USA; Marcelo Milrad, Växjö University, Sweden; Riichiro Mizoguchi, Osaka University, Japan; Hiroaki Ogata, Tokushima University, Japan; Toshio Okamoto, The University of ElectroCommunications, Japan; Gilly Salmon, University of Leicester, United Kingdom; Timothy K. Shih, Tamkang University, Taiwan; Yoshiaki Shindo, Nippon Institute of Technology, Japan; Brian K. Smith, Pennsylvania State University, USA; J. Michael Spector, Florida State University, USA.

Assistant Editors Sheng-Wen Hsieh, National Sun Yat-sen University, Taiwan; Taiyu Lin, Massey University, New Zealand; Kathleen Luchini, University of Michigan, USA; Dorota Mularczyk, Independent Researcher & Web Designer; Carmen Padrón Nápoles, Universidad Carlos III de Madrid, Spain; Ali Fawaz Shareef, Massey University, New Zealand; Jarkko Suhonen, University of Joensuu, Finland.

Executive peer-reviewers http://www.ifets.info/

Subscription Prices and Ordering Information Institutions: NZ$ 120 (~ US$ 75) per year (four issues) including postage and handling. Individuals (no school or libraries): NZ$ 100 (~ US$ 50) per year (four issues) including postage and handling. Single issues (individuals only): NZ$ 35 (~ US$ 18) including postage and handling. Subscription orders should be sent to The International Forum of Educational Technology & Society (IFETS), c/o Prof. Kinshuk, Information Systems Department, Massey University, Private Bag 11-222, Palmerston North, New Zealand. Tel: +64 6 350 5799 ext 2090. Fax: +64 6 350 5725. E-mail: [email protected].

Advertisements Educational Technology & Society accepts advertisement of products and services of direct interest and usefulness to the readers of the journal, those involved in education and educational technology. Contact the editors at [email protected].

Abstracting and Indexing Educational Technology & Society is abstracted/indexed in Social Science Citation Index, Current Contents/Social & Behavioral Sciences, ISI Alerting Services, Social Scisearch, ACM Guide to Computing Literature, Australian DEST Register of Refereed Journals, Computing Reviews, DBLP, Educational Administration Abstracts, Educational Research Abstracts, Educational Technology Abstracts, Elsevier Bibliographic Databases, ERIC, Inspec, Technical Education & Training Abstracts, and VOCED. ISSN ISSN1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forumThe jointly authors retain andthe the copyright forum jointly of the retain articles. the copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or allhard of this copies workoffor part personal or all of or this classroom work for usepersonal is granted or without classroom feeuse provided is granted that without copies are feenot provided made orthat distributed copies are fornot profit made or or commercial distributedadvantage for profit and or commercial that copies advantage bear the full andcitation that copies on thebear firstthe page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned mustbybe others honoured. than IFETS Abstracting must with be honoured. credit is permitted. AbstractingTowith copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post ontoservers, lists, requires or to redistribute prior specific to lists, permission requiresand/or prior a specific fee. Request permission permissions and/or afrom fee. the Request editors permissions at [email protected]. from the editors at [email protected].

i

Guidelines for authors Submissions are invited in the following categories:

• Peer reviewed publications: a) Full length articles (4000 - 7000 words), b) Short articles, Critiques and Case studies (up to 3000 words) • Book reviews • Software reviews • Website reviews

All peer review publications will be refereed in double-blind review process by at least two international reviewers with expertise in the relevant subject area. Book, Software and Website Reviews will not be reviewed, but the editors reserve the right to refuse or edit review. • Each peer review submission should have at least following items: ƒ title (up to 10 words), ƒ complete communication details of ALL authors , ƒ an informative abstract (75-200 words) presenting the main points of the paper and the author's conclusions, ƒ four - five descriptive keywords, ƒ main body of paper (in 10 point font), ƒ conclusion, ƒ references. • Submissions should be single spaced. • Footnotes and endnotes are not accepted, all such information should be included in main text. • The paragraphs should not be indented. There should be one line space between consecutive paragraphs. • There should be single space between full stop of previous sentence and first word of next sentence in a paragraph. • The keywords (just after the abstract) should be separated by comma, and each keyword phrase should have initial caps (for example, Internet based system, Distance learning). • Do not use 'underline' to highlight text. Use 'italic' instead.

Headings Articles should be subdivided into unnumbered sections, using short, meaningful sub-headings. Please use only two level headings as far as possible. Use 'Heading 1' and 'Heading 2' styles of your word processor's template to indicate them. If that is not possible, use 12 point bold for first level headings and 10 point bold for second level heading. If you must use third level headings, use 10 point italic for this purpose. There should be one blank line after each heading and two blank lines before each heading (except when two headings are consecutive, there should be one blank like between them).

Tables Tables should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be provided for every table (below the table) and must be referenced in the text.

Figures Figures should be included in the text at appropriate places and centered horizontally. Captions (maximum 6 to 8 words each) must be provided for every figure (below the figure) and must be referenced in the text. The figures must NOT be larger than 500 pixels in width. Please also provide all figures separately (besides embedding them in the text).

References

• All references should be listed in alphabetical order at the end of the article under the heading 'References'. • All references must be cited in the article using "authors (year)" style e.g. Merrill & Twitchell (1994) • •

or "(authors1, year1; authors2, year2)" style e.g. (Merrill, 1999; Kommers et al., 1997). Do not use numbering style to cite the reference in the text e.g. "this was done in this way and was found successful [23]." It is important to provide complete information in references. Please follow the patterns below:

Journal article Laszlo, A. & Castro, K. (1995). Technology and values: Interactive learning environments for future generations. Educational Technology, 35 (2), 7-13. Newspaper article Blunkett, D. (1998). Cash for Competence. Times Educational Supplement, July 24, 1998, 15. Or Clark, E. (1999). There'll never be enough bandwidth. Personal Computer World, July 26, 1999, retrieved July 7, 2004, from http://www.vnunet.co.uk/News/88174. Book (authored or edited) Brown, S. & McIntyre, D. (1993). Making sense of Teaching, Buckingham: Open University. Chapter in book/proceedings Malone, T. W. (1984). Toward a theory of intrinsically motivating instruction. In Walker, D. F. & Hess, R. D. (Eds.), Instructional software: principles and perspectives for design and use, California: Wadsworth Publishing Company, 68-95. Internet reference Fulton, J. C. (1996). Writing assignment as windows, not walls: enlivening unboundedness through boundaries, retrieved July 7, 2004, from http://leahi.kcc.hawaii.edu/org/tcc-conf96/fulton.html.

Submission procedure Authors, submitting articles for a particular special issue, should send their submissions directly to the appropriate Guest Editor. Guest Editors will advise the authors regarding submission procedure for the final version. All submissions should be in electronic form. The editors will acknowledge the receipt of submission as soon as possible. The preferred formats for submission are Word document and RTF, but editors will try their best for other formats too. For figures, GIF and JPEG (JPG) are the preferred formats. Authors must supply separate figures in one of these formats besides embedding in text. Please provide following details with each submission: ƒ Author(s) full name(s) including title(s), ƒ Name of corresponding author, ƒ Job title(s), ƒ Organisation(s), ƒ Full contact details of ALL authors including email address, postal address, telephone and fax numbers. The submissions should be uploaded at http://www.ifets.info/ets_journal/upload.php. In case of difficulties, they can also be sent via email to (Subject: Submission for Educational Technology & Society journal): [email protected]. In the email, please state clearly that the manuscript is original material that has not been published, and is not being considered for publication elsewhere. ISSN ISSN1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forumThe jointly authors retain andthe the copyright forum jointly of the retain articles. the copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or allhard of this copies workoffor part personal or all of or this classroom work for usepersonal is granted or without classroom feeuse provided is granted that without copies are feenot provided made orthat distributed copies are fornot profit made or or commercial distributedadvantage for profit and or commercial that copies advantage bear the full andcitation that copies on thebear firstthe page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned mustbybe others honoured. than IFETS Abstracting must with be honoured. credit is permitted. AbstractingTowith copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post ontoservers, lists, requires or to redistribute prior specific to lists, permission requiresand/or prior a specific fee. Request permission permissions and/or afrom fee. the Request editors permissions at [email protected]. from the editors at [email protected].

ii

Journal of Educational Technology & Society Volume 9 Number 3 2006

Table of contents Special issue articles Theme: Next Generation e-Learning Systems: Intelligent Applications and Smart Design Editorial: Next Generation e-Learning Systems: Intelligent Applications and Smart Design Demetrios G. Sampson and Peter Goodyear

1-2

A Particle Swarm Optimization Approach to Composing Serial Test Sheets for Multiple Assessment Criteria Peng-Yeng Yin, Kuang-Cheng Chang, Gwo-Jen Hwang, Gwo-Haur Hwang and Ying Chan

3-15

Modeling Peer Assessment as Agent Negotiation in a Computer Supported Collaborative Learning Environment K. Robert Lai and Chung Hsien Lan

16-26

Ontology Mapping and Merging through OntoDNA for Learning Object Reusability Ching-Chieh Kiu and Chien-Sing Lee

27-42

FODEM: developing digital learning environments in widely dispersed learning communities Jarkko Suhonen and Erkki Sutinen

43-55

From Research Resources to Learning Objects: Process Model and Virtualization Experiences José Luis Sierra, Alfredo Fernández-Valmayor, Mercedes Guinea and Héctor Hernanz

56-68

Mining Formative Evaluation Rules Using Web-based Learning Portfolios for Web-based Learning Systems Chih-Ming Chen, Chin-Ming Hong, Shyuan-Yi Chen and Chao-Yu Liu

69-87

Formal Method of Description Supporting Portfolio Assessment Yasuhiko Morimoto, Maomi Ueno, Isao Kikukawa, Setsuo Yokoyama and Youzou Miyadera

88-99

A Systemic Activity based Approach for Holistic Learning & Training Systems Hansjörg von Brevern and Kateryna Synytsya

100-111

Full length articles Campus Laptops: What Logistical and Technological Factors are Perceived Critical? Robert Cutshall, Chuleeporn Changchit and Susan Elwood

112-121

Students’ Preferences on Web-Based Instruction: linear or non-linear Nergiz Ercil Cagiltay, Soner Yildirim and Meral Aksu

122-136

Student-Generated Visualization as a Study Strategy for Science Concept Learning Yi-Chuan Jane Hsieh and Lauren Cifuentes

137-148

On Improving Spatial Ability Through Computer-Mediated Engineering Drawing Instruction Ahmad Rafi, Khairul Anuar Samsudin and Azniah Ismail

149-159

Massively multiplayer online games (MMOs) in the new media classroom Aaron Delwiche

160-172

A Web environment to encourage students to do exercises outside the classroom: A case study Laurence Capus, Frédéric Curvat, Olivier Leclair and Nicole Tourigny

173-181

The effect of LEGO Training on Pupils’ School Performance in Mathematics, Problem Solving Ability and Attitude: Swedish Data Shakir Hussain, Jörgen Lindh and Ghazi Shukur

182-194

The Effects of a Computer-Assisted Interview Tool on Data Quality Justus J. Randolph, Marjo Virnes, Ilkka Jormanainen and Pasi J. Eronen

195-205

Data for School Improvement: Factors for designing effective information systems to support decisionmaking in schools Andreas Breiter and Daniel Light

206-217

ISSN 1436-4522 1436-4522.(online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum&ofSociety Educational (IFETS). Technology The authors & Society and the (IFETS). forum The jointly authors retainand thethecopyright forum jointly of theretain articles. the Permissionoftothe copyright make articles. digital Permission or hard copies to make of part digital or all orof hard thiscopies work for of part personal or allorofclassroom this work use for is personal grantedorwithout classroom fee provided use is granted that copies without arefee notprovided made or that distributed copies for profit are not made or commercial or distributed advantage for profitand or that commercial copies bear advantage the fulland citation that copies on the bear first page. the full Copyrights citation onfor thecomponents first page. Copyrights of this workfor owned components by others of than this work IFETS owned must by be honoured. others thanAbstracting IFETS mustwith be honoured. credit is permitted. Abstracting To with copy credit otherwise, is permitted. to republish, To copy to post otherwise, on servers, to republish, or to redistribute to post on to lists, servers, requires or to prior redistribute specifictopermission lists, requires and/or priora specific fee. Request permission permissions and/orfrom a fee. theRequest editors permissions at [email protected]. from the editors at [email protected].

iii

Component Exchange Community: A model of utilizing research components to foster international collaboration Yi-Chan Deng, Taiyu Lin, Kinshuk and Tak-Wai Chan

218-231

A Visualization Tool for Managing and Studying Online Communications William J. Gibbs, Vladimir Olexa and Ronan S. Bernas

232-243

An ICT-mediated Constructivist Approach for increasing academic support and teaching critical thinking skills Dick Ng’ambi and Kevin Johnston

244-253

ISSN ISSN1436-4522 1436-4522. (online) © International and 1176-3647 Forum (print). of Educational © International Technology Forum of & Educational Society (IFETS). Technology The authors & Society and (IFETS). the forumThe jointly authors retain andthe the copyright forum jointly of the retain articles. the copyright Permission of the to make articles. digital Permission or hard copies to make of digital part or or allhard of this copies workoffor part personal or all of or this classroom work for usepersonal is granted or without classroom feeuse provided is granted that without copies are feenot provided made orthat distributed copies are fornot profit made or or commercial distributedadvantage for profit and or commercial that copies advantage bear the full andcitation that copies on thebear firstthe page. full Copyrights citation on the for components first page. Copyrights of this work for owned components by others of this than work IFETS owned mustbybe others honoured. than IFETS Abstracting must with be honoured. credit is permitted. AbstractingTowith copy credit otherwise, is permitted. to republish, To copy to otherwise, post on servers, to republish, or to redistribute to post ontoservers, lists, requires or to redistribute prior specific to lists, permission requiresand/or prior a specific fee. Request permission permissions and/or afrom fee. the Request editors permissions at [email protected]. from the editors at [email protected].

iv

Sampson, D. G., & Goodyear, P. (2006). Next Generation e-Learning Systems: Intelligent Applications and Smart Design (Guest Editorial). Educational Technology & Society, 9 (3), 1-2.

Next Generation e-Learning Systems: Intelligent Applications and Smart Design (Guest Editorial) Demetrios G. Sampson Department of Technology Education and Digital Systems, University of Piraeus and Cether for Research and Technology - Hellas, 150 Androutsou Street, Piraeus, 18235, Greece [email protected]

Peter Goodyear CoCo Research Centre, Education Building, A35, University of Sydney, NSW 2006, Australia [email protected] This special issue of Educational Technology & Society presents a selection of papers from the 5th IEEE International Conference on Advanced Learning Technologies (ICALT2005) that was held in Kaohsiung, Taiwan in July 2005. There were 409 submissions to that conference. The acceptance rate for full papers was 23% and for short papers 30%. There were 205 papers published in the proceedings, along with descriptions of 55 posters and 32 workshop papers. We had the great honour of being co-chairs of the Programme Committee for that conference and it has been a difficult but rewarding task to decide from the great number of good papers presented at that conference. The conference theme represents a confluence of two combinations of technology and intelligence and their application to the design of (web-based) educational systems. On the one hand, we have the increasingly sophisticated embedding of intelligent technologies in educational computing applications. On the other, we see a need for more ‘savvy’ approaches to learning technology design: so that emerging technology can better serve the real needs of its users, rather than their “anticipated” needs. Advanced Learning Technologies (ALT) or Technology-enhanced Learning (TeL), at its best, is a field propelled by a creative tension – coupling an open-minded exploration of the educational affordances of each new technology with a rigorous demand for evidence to back up claims about potential benefit. Sometimes technological innovation seems to be in the driving seat, and some sceptics complain about “technological solutions in search of educational problems”. At other times, demands for evidence and the tight constraints of established evaluation methods can make the field appear as if it is moving nowhere. In the short term, this can be frustrating and partly demotivating. But one thing we have learned is that ALT progresses in the longer-term. While it may appear to meander back and forth, on a long view it generally seems to flow in the right direction. Thus, each of the papers selected here needs to be seen as contributing to this general flow. As in any healthy field, the papers differ in their central concerns and may even seem to be heading in different directions – but these are eddies making up the stream, rather than signs of a field in disarray. We present them to you as worthwhile contributions in their own right, but also for what they say about the general flow of ideas. The first paper in the collection, entitled “A Particle Swarm Optimization Approach to Composing Serial Test Sheets for Multiple Assessment Criteria” is by Yin, Hwang, Chang, Hwang and Chan. Their topic is central to education and forms a dominant theme within this special issue, being concerned with assessment of student learning. Assessment is a technically challenging area, hugely important for the student and potentially very time consuming for the teacher. Consequently there is a strong interest in automating or partially automating aspects of assessment work. Problems arise in computer-based test construction if very large item banks are in use and there are multiple constraints to be satisfied (such as length of test, restrictions on multiple use of items, gauging item difficulty to suit the level of the learner being assessed, etc.) This paper explores the use of particle swam optimisation approach in test sheet construction. The authors draw on recent modelling techniques from statistical biology, in particular, swarm modelling algorithms, and demonstrate aspects of the efficacy and the usability of a test-construction methodology that builds on such algorithms. Lai & Lan, in their paper entitled “Modelling Peer Assessment as Agent Negotiation in a Computer Supported Collaborative Learning Environment” are also interested in assessment and the use of technology to assist in the assessment process, but their context is radically different. They are concerned with facilitating peer assessment, rather than automating assessment per se. Peer assessment has some demonstrable value, especially as a way of getting students to engage more closely with ideas about how we make judgements about what is known and worth knowing. But it can also be fraught with difficulty. Lai & Lan describe a novel approach to helping ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

1

students negotiate in the peer-assessment process, through the use of mediating agents. Their paper is exemplary in combining technical innovation with demonstration of both learning benefits and user acceptance of the approach. The context for the work reported by Kiu and Lee in their paper entitled “Ontology Mapping and Merging through OntoDNA for Learning Object Reusability” is the rapid evolution of interest in repositories of reusable learning objects. This is an area of great technical, economic and pedagogical interest, though one would have to remark that much more attention is being paid to supply-side than to demand-side issues. That said, the potential of learning object repositories is such that serious progress is needed in methods for enhancing the interoperability of repositories. Kiu and Lee’s work on ontologies is an impressive case in point. They present a framework for automated ontology mapping (the OntoDNA framework) and demonstrate its significance for interoperability questions. Suhonen & Sutinen, in their paper entitled “FODEM: developing digital learning environments in sparse learning communities” shift our attention to distance learning provision and in particular are concerned with the needs of people in sparsely populated areas. Their focus is firmly on ‘smart design’ and they introduce and illustrate a methodology for designing digital learning environments that takes into account the needs of widely distributed learner groups. Their approach – the Formative Development Method, FODEM – focuses attention on needs analysis, implementation through rapid prototyping and formative evaluation; it is capable of providing timely and usable information about learner needs and about how well those needs are being understood and addressed. Sierra, Fernández-Valmayor, Guinea & Hernanz, in their paper entitled “From Research Resources to Learning Objects: Process Model and Virtualization Experiences” once more address the issue of reusable learning objects. In this case, their concern is with enabling wider educational access to materials that exist in museum collections. An important part of the problem is virtualization – in a sense, shifting the artefacts from the material to the digital world. The paper describes a process model for virtualization, building on the practical experience of the authors in working with domain experts in museums. An interesting aspect of this approach is that it is realistically conservative in its assumptions about how much effort, domain experts can contribute to such work. By this way, the authors attempt to avoid the fact that quite a lot of activity in the field of virtualization and/or re-purposing has made unrealistic assumptions about the skills and time available to domain experts – e.g. for producing metadata – and has consequently failed. Chen, Hong, Chen & Liu, in their paper entitled “Mining Formative Evaluation Rules Using Web-based Learning Portfolios for Web-based Learning Systems” bring us to assessment once more. This time the focus is on formative rather than summative assessment. They take a data mining approach to extracting evidence about learning from students’ online portfolios. Their method involves a combination of neuro-fuzzy network and Kmeans algorithm for logically determining membership functions, and a feature reduction scheme to discover a manageably small set of simplified fuzzy rules for evaluating learning performance based on the material gathered in student learning portfolios. Their goal is to allow teachers to redistribute their time, concentrating on tasks where they can uniquely add value to the educational process. Morimoto, Ueno, Kikukawa, Yokoyama and Miyadera, in their paper entitled “Formal Method of Description Supporting Portfolio Assessment” stay with the topic of portfolio assessment. This time, the point is to provide appropriate support to teachers and learners who can have problems understanding how best to engage with the portfolio assessment process. In particular, teachers need support in designing for portfolio assessment – e.g. determining the type of assessment portfolios that are needed. The contribution of this paper is to provide a way of formally mapping between lesson forms and portfolios. Finally, von Brevern & Synytsya, in their paper entitled “A Systemic Activity based Approach for holistic Learning & Training Systems” look at the connections between work activity and learning activity in corporate settings. Their contribution is rooted in a Systemic-Structural Theory of Activity, which supports a more holistic conceptualisation of learning and working, such that (among other things) technical systems can be designed to support these in an integrated rather than a fragmenting way. This Special Issue of Educational Technology & Society collected eight papers from the 5th IEEE International Conference on Advanced Learning Technologies (ICALT2005) in a single volume. Technology-enhanced Assessment on one hand and Reusable Learning Resources from the other, have been two different areas of focus, following a current international trend towards deeper investigations in these topics. With our capacity, as Guest Editors of this volume, we hope that the readers of ET&S shall appreciate the contributions of this collection towards the research of the Next Generation e-Learning Systems. 2

Yin, P.-Y., Chang, K.-C., Hwang, G.-J., Hwang, G.-H., & Chan, Y. (2006). A Particle Swarm Optimization Approach to Composing Serial Test Sheets for Multiple Assessment Criteria. Educational Technology & Society, 9 (3), 3-15.

A Particle Swarm Optimization Approach to Composing Serial Test Sheets for Multiple Assessment Criteria Peng-Yeng Yin and Kuang-Cheng Chang Department of Information Management, National Chi Nan University, Pu-Li, Nan-Tou, Taiwan 545, R.O.C.

Gwo-Jen Hwang Department of Information and Learning Technology, National University of Tainan, 33, Sec. 2 Shulin St.,Tainan city 70005, Taiwan, R.O.C. [email protected] Tel: 886-915396558 Fax: 886-6-3017001

Gwo-Haur Hwang Information Management Department, Ling Tung University, Taichung, Taiwan 40852, R.O.C.

Ying Chan Graduate Institute of Educational Policy and Leadership, Tamkang University Tamsui, Taipei County, Taiwan 251, R.O.C.

ABSTRACT To accurately analyze the problems of students in learning, the composed test sheets must meet multiple assessment criteria, such as the ratio of relevant concepts to be evaluated, the average discrimination degree, difficulty degree and estimated testing time. Furthermore, to precisely evaluate the improvement of student’s learning performance during a period of time, a series of relevant test sheets need to be composed. In this paper, a particle swarm optimization-based approach is proposed to improve the efficiency of composing near optimal serial test sheets from very large item banks to meet multiple assessment criteria. From the experimental results, we conclude that our novel approach is desirable in composing near optimal serial test sheets from large item banks and hence can support the need of evaluating student learning status.

Keywords Computer-assisted testing, serial test-sheet composing, particle swarm optimization, computer-assisted assessment

1. Introduction As the efficiency and efficacy of the deployment of computer-based tests have been confirmed by many early studies, many researchers in both technical and educational fields have engaged in the development of computerized testing systems (Fan et al., 1996; Olsen et al., 1986). Some researchers have even proposed computerized adaptive testing, which uses prediction methodologies to shorten the length of the test sheets without sacrificing their precision (Wainer, 1990). A well-scrutinized test is helpful for teachers wanting to verify whether students well digest relevant knowledge and skills and for recognition of students’ learning bottlenecks (Hwang et al., 2003a). In a computerized learning environment, which provides students with greater flexibility during the learning process, information concerning the student learning status is even more important (Hwang, 2003a). The key to a good test depends not only on the subjective appropriateness of test items, but also on the way the test sheet is constructed. To continuously evaluate the learning performance of a student, it is usually more desirable to compose a series of relevant test sheets to meet a predefined set of assessment criteria such that those test sheets in the same series will not contain identical test items (or contain only an acceptable percentage of overlapped test items). Because the number of test items in an item bank is usually large and the number of feasible combinations to form test sheets thus grows exponentially, an optimal test sheet takes enormous time to build up (Garey & Johnson, 1979). Previous investigation has even shown that a near-optimal solution is difficult to find when the number of candidate test items is larger than five thousand (Hwang et al., 2003b), not to mention the composition of a series

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

3

of relevant test sheets from larger item banks for evaluating the improvement of student’s learning performance during a period of time. To cope with the problem in composing optimal serial test sheets from large item banks, a particle swarm optimization (PSO)-based algorithm (Kennedy and Eberhart, 1995) is proposed to optimize the selection of test items to compose serial test sheets. By employing this novel approach, the allocation of test items in each of the serial test sheets will meet the needs of multiple criteria, including the expected testing time, the degree of difficulty, the expected ratio of unit concepts, and the acceptable percentage of overlapped test items among test sheets to approximate the optimal allocation. Based on this approach, an Intelligent Tutoring, Testing and Diagnostic (ITED III) system has been developed. Experimental results indicated that the proposed approach is efficient and effective in generating near-optimal compositions of serial test sheets that satisfy the specified requirements.

2. Background and Relevant Researches In recent years, researchers have developed various computer-assisted testing systems to more precisely evaluate student’s learning status. For example, Feldman and Jones (1997) attempted to perform semi-automatic testing of student software using Unix systems; Rasmussen, et al. (1997) proposed a system to evaluate student learning status on computer networks while taking Feldman and Jones’ progress into consideration. Additionally, Chou (2000) proposed the CATES system, which is an interactive testing system developed in a collective and collaborative project with theoretical and practical research on complex technology-dependent learning environments. Unfortunately, although many computer-assisted testing systems have been proposed, few of them have addressed the problem of finding a systematic approach for composing test sheets that satisfy multiple assessment requirements. Most of the existing systems construct a test sheet by manually or randomly selecting test items from their item banks. Such manual or random test item selection strategies are inefficient and are unable to meet multiple assessment requirements simultaneously. Some previous investigations showed that a well-constructed test sheet not only helps in the evaluation of student’s learning status, but also facilitates the diagnosis of the problems embedded in the learning process (Hwang, 2003a; Hwang et al. 2003a; Hwang 2005). Selecting proper test items is very critical to constitute a test sheet that meets multiple assessment criteria, including the expected time needed for answering the test sheet, the number of test items, the specified distribution of course concepts to be learned, and, most importantly, the maximization of the average degree of discrimination (Hwang et al, 2005). Since satisfying multiple requirements (or constraints) when selecting test items is difficult, most computerized testing systems generate test sheets in a random fashion. Hwang et al. (2003b) proposed a multiple-criteria where test sheet-composing problem is formulated as a dynamic programming model (Hillier and Lieberman, 2001) to minimize the distance between the parameters (e.g., discrimination, difficulty, etc.) of the generated test sheets and the objective values subject to the distribution of concept weights. A critical issue arising from the use of a dynamic programming approach is the exceedingly long execution time required for producing optimal solutions. As the time-complexity of the dynamic programming algorithm is exponential in terms of input data, the execution time will become unacceptably long if the number of candidate test items is large. Consequently, Hwang et al. (2005) attempted to solve the test sheet-composing problem by optimizing the discrimination degree of the generated test sheets with a specified range of assessment time and some other multiple constraints. Nevertheless, in developing an e-learning system, it is necessary to conduct a long-term assessment of each student; that is, only optimizing a test sheet is not enough for such long-term observation for the student. Therefore, a series of relevant test sheets with multiple assessment criteria need to be composed for such a continuously learning performance evaluation. As the problem is much more difficult than that of composing a single test sheet, a more efficient and effective approach is needed. In this paper, a particle swarm optimizationbased algorithm is proposed to find quality approximate solutions in an acceptable time. A series of experiments will be also presented to show the performances of the novel approach.

3. Problem Description In this section, a mixed integer programming model (Linderoth and Savelsbergh, 1999) is presented to formulate the underlying problem. In order to conduct a long-term observation on the student’s learning status, a series of K relevant test sheets will be composed. The model aims at minimizing the differences between the average 4

difficulty of each test sheet and the specified difficulty target, with a specified range of assessment time and some other multiple constraints. Assume K serial test sheets with a specific difficulty degree will be composed out of a test bank consisting of N items, Q1, Q2, …, QN. To compose test sheet k, 1 ≤ k ≤ K, a subset of nk candidate test items will be selected. Assume that in total M concepts will be involved in the K tests. With the specified course concepts to be learned, say Cj, 1 ≤ j ≤ M , each test item is relevant to one or more of them. For example, to test the multimedia knowledge of students, Cj might be “MPEG”, “Video Streaming” or “Videoon-Demand”. We shall call this problem the STSC (Serial Test Sheets Composition) problem. In the STSC problem, we need confine the similarities between each pair of tests. Such a constraint imposed upon each pair of tests k and l, 1 ≤ k, l ≤ K, is specified by parameter f, which indicates that any two tests can have at most f items in common. The variables used in this model are given as follows: ¾ Decision variables xik, 1 ≤ i ≤ N and 1 ≤ k ≤ K: xik is 1 if test item Qi is included in test sheet k; 0, otherwise. ¾ Coefficient di, 1 ≤ i ≤ N : degree of difficulty of Qi. ¾ Coefficient D, target difficulty level for each of the serial test sheets generated. ¾ Coefficient rij, 1 ≤ i ≤ N , 1 ≤ j ≤ M : degree of association between Qi and concept Cj. ¾ ¾ ¾ ¾ ¾

Coefficient ti, 1 ≤ i ≤ N : expected time needed for answering Qi. Right hand side hj, 1 ≤ j ≤ M: lower bound on the expected relevance of Cj for each of the K test sheets. Right hand side l: lower bound on the expected time needed for answering each of the K test sheets. Right hand side u: upper bound on the expected time needed for answering each of the K test sheets. Right hand side f: the maximum number of identical test items between two composed test sheets;

Formal definition of the STSC Model:

⎛ ⎜ ⎜ Minimize Zk = ⎜ ∑ ⎜ 1≤ k ≤ K ⎜ ⎝ subject to: N

∑r x ij

ik

i =1 N

N

∑

p

d i x ik

i =1 N

∑

− D x ik

i =1

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

1

p

≥ h j , 1 ≤ j ≤ M , 1 ≤ k ≤ K;

(1)

∑t x

ik

≥ l , 1 ≤ k ≤ K;

(2)

∑t x

ik

≤ u, 1 ≤ k ≤ K ;

(3)

i

i =1 N

i

i =1 N

∑x x

ij ik

≤ f , 1 ≤ j ≠ k ≤ K;

(4)

i =1

In the above formula, constraint set (1) indicates the selected test items in each generated test sheet must have a total relevance no less than the expected relevance to each concept assumed to be covered. Constraint sets (2) and (3) indicate that total expected test time of each generated test sheet must be in its specified range. Constraint set (4) indicates that no pair of test sheets can contain more than f identical test items.

⎛ ⎜ ⎜ In the objective function, Zk = ⎜ ∑ ⎜ 1≤ k ≤ K ⎜ ⎝

N

∑

p

d i x ik

i =1 N

∑

i =1

− D x ik

⎞ ⎟ ⎟ ⎟ ⎟ ⎟ ⎠

1

p

is the p-norm between the average

difficulty degree of each test sheet from the target difficulty degree specified by the teacher. In particular, the objective function indicates the absolute distance when p = 1, and it calculates the root squared distance when p = 2. Therefore, the objective of this model seeks to select a number of test items such that the average difficulty 5

of each generated test sheet is closest to the target difficulty value D. Without loss of generality, we let p = 2 for simulation. The computation complexity for obtaining the optimal solution to the STSC problem is analyzed as follows. The number of possible combinations of test items for composing a single test sheet is ⎛⎜ N ⎞⎟ where Ω is the range

∑⎜ i∈Ω

⎟ ⎝i⎠

for number of test items that could be answered within the specified time frame [l, u], while the parameters hj and f will affect the number of feasible solutions in those combinations. For composing K serial test sheets, it K requires a computation complexity of ⎛⎜ ⎛⎜ ⎛ N ⎞ ⎞⎟ ⎞⎟ which is extremely high. Hence, seeking the optimal solution O⎜ ∑ ⎜ ⎟ ⎟ ⎜ ⎜⎝ i∈Ω ⎜⎝ i ⎟⎠ ⎟⎠ ⎟ ⎠ ⎝

to the STSC problem is computationally prohibitive.

4. PSO-based Algorithm for Serial Test Sheet Composition Linderoth and Savelsbergh (1999) conducted a comprehensive computational study manifesting that the mixed integer programming problems are NP-hard, which implies that composing optimal serial test sheets from a large item bank is computationally prohibitive. To cope with this difficulty, a particle swarm optimization (PSO)based algorithm is proposed to find quality approximate solutions with reasonable time. A. STSCPSO (Serial Test Sheets Composition with PSO) Algorithm The PSO algorithm was developed by Kennedy and Eberhart (1995). It is a biologically inspired algorithm which models the social dynamics of bird flocking and fish schooling. Ethologists find that a swarm of birds/fishes flock synchronously, change direction suddenly, scatter and regroup iteratively, and finally stop on a common target. The collective intelligence from each individual not only increases the success rate for food foraging but also expedites the process. The PSO algorithm facilitates simple rules simulating bird flocking and fish schooling and can serve as an optimizer for nonlinear functions. Kennedy and Eberhart (1997) further presented a discrete binary version of PSO for combinatorial optimization where the particles are represented by binary vectors of length d and the velocity represents the probability that a decision variable will take the value 1. PSO has delivered many successful applications (Eberhart and Shi, 1998; Yoshida et al., 1999; Shigenori et al., 2003). The convergence and parameterization aspects of the PSO have also been discussed thoroughly (Clerc and Kennedy, 2002; Trelea, 2003). In the followings, a PSO-based algorithm, STSCPSO (Serial Test Sheets Composition with PSO approach), is proposed to find quality approximate solutions for the STSC problem. Input: N test items Q1, Q2, …, Qn, M concepts C1, C2, …, Cm, the target difficulty level D, and the number of required test sheets, K. Step 1. Generate initial swarm Since all decision variables of the STSC problem take binary values (either 0 or 1), a particle in the STSCPSO algorithm can be represented by x = x11 x 21 ⋅ ⋅ ⋅ x N 1 x12 x 22 ⋅ ⋅ ⋅ x N 2 ⋅ ⋅ ⋅ x1K x2 K ⋅ ⋅ ⋅ x NK , which is a vector of NK

[

]

binary bits where xik is equivalent to 1 if test item Qi is included in test sheet k and 0 otherwise. Due to the constrains (2) and (3) with test time, the number of selected items in any test sheet is bounded in [l max{t i }, u min{t i }] . Hence, we should enforce the integrity rule: i =1~ N

i =1~ N N i =1 ik

l max{t i } ≤ ∑ x ≤ u min{t i }, ∀k = 1,2,..., K , during every step of our algorithm. To generate the i =1~ N

i =1~ N

initial swarm, we randomly determine the number of items for each test sheet according to the integrity rule. The selection probability of each item is based on the selection rule which gives higher selection probability to the items that have closer difficulty level to the target. In particular, the selection probability of item Qi is defined as

(S − d

i

− D ) S where S is a constant. As such the initial swarm contains solutions that have good objective

6

values but may violate constraint sets. Then the particle swarm evolves to quality solutions that not only optimize the objective function but also meet all of the constraint sets. Step 2. Fitness evaluation of particles The original objective function of the STSC problem measures the quality of a candidate solution which meets all the constraints (1)-(4). However, the particles generated by the PSO-based algorithm may violate one or more of these constraints. To cope with this problem, the merit of a particle is evaluated by incorporating penalty terms into the objective function if any constraint is violated. The penalty terms corresponding to separate constraints are described as follows. ¾ α penalty for violating concept relevance bound constraint K

M

⎛

N

⎞

⎝

i =1

⎠

α = ∑∑ ⎜ h j − ∑ rij xik ⎟ . k =1 j =1

¾

This term sums up relevance deficit of selected test items to the specified relevance lower bound of each concept over all test sheets. β penalty for violating test time bound constraint K

⎛

⎛

N

⎞

⎛

N

⎞⎞

k =1

⎝

⎝

i =1

⎠

⎝

i =1

⎠⎠

β = ∑ ⎜⎜ max⎜ l − ∑ t i xik ,0 ⎟ + max⎜ 0, ∑ t i xik − u ⎟ ⎟⎟ .

¾

This term penalizes the case where the expected test times are beyond the specified lower bound or upper bound. γ penalty for violating common item constraint

⎛

N

⎞

γ = ∑ ⎜ ∑ xij xik − f ⎟ . j ≠k

⎝ i =1

⎠

This term penalizes the case where the number of common items between two different tests exceeds the threshold f. ¾

Function J(⋅) for evaluating the fitness of a particle x Minimize

J ( x ) = Z k + w1α + w2 β + w3γ .

w1, w2, and w3 denote relative weights for the three penalty terms. As such the fitness of a particle x accounts for both of quality (objective value) and feasibility (penalty terms). The smaller the fitness value, the better the particle. Step 3. Determination of pbesti and gbest using the bounding criterion In the original PSO, the fitness evaluation of particles which is a necessity for determination of pbesti and gbest is the most time-consuming part. Here we propose a bounding criterion to speed up the process. We observe that the fitness value of a particle is only used for determination of pbesti and gbest, but not directly used for velocity update. Since Zk and J(⋅) are both monotonically increasing functions, we can use the fitness of the incumbent pbesti as a fitness bound and terminate the fitness evaluation of the ith particle when the intermediate fitness value has exceeded the bound. Also, only those pbesti that have been updated at the current iteration need to be compared with gbest for its possible updating. The use of bounding criterion can save the computational time significantly. Step 4. Update of velocities and particles The updating of velocities and particle positions follow the discrete version of PSO, i.e., the velocity is scaled into [0.0, 1.0] by a transformation function S(⋅) and is used as the probability with which the particle bit takes the value 1. In this paper, we adopt the linear interpolation function, S (vij ) =

vij

2 vmax

+ 0.5 , to transform velocities

into probabilities.

7

C. An Illustrative Example Herein, an illustrative example for the STSCPSO algorithm is provided. Assume that two test sheets with target difficulty level D = 5 are required to be generated from 10 test items. The 10 test items are relevant to 3 concepts, and the relevance association (rij) between each test item and each concept is shown in Table 1. The estimated answering time (ti) and difficulty degree (di) for the 10 test items are tabulated in Table 2. Let h1 = 2, h2 = 2, h3 = 1, l = 10, u = 16, f = 3, w1 = w2 = w3 = 0.01. The algorithm proceeds as follows. Table 1. Relevance association between each test item and each concept C1 C2 C3 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10

1 0 0 0 0 0 1 1 0 0

0 1 0 1 0 1 0 1 0 1

0 0 1 0 1 0 0 1 0 0

Table 2. Estimated answering time and difficulty degree for each test item ti di Q1 4 0.5 Q2 3 0.9 Q3 3 0.1 Q4 5 0.7 Q5 3 0.4 Q6 2 0.5 Q7 4 0.2 Q8 3 0.6 Q9 5 0.3 Q10 4 0.5 Initial swarm generation Let the algorithm proceed with a swarm of two particles. To generate the initial swarm, the range for the feasible number of selected items in a test sheet is first determined using the integrity rule: N l max{t i } ≤ ∑ xik ≤ u min{t i } . Hence, each test sheet can select 2 to 8 items from the test item bank. i =1~ N

i =1

i =1~ N

According to our particle representation scheme, each particle is represented as a binary vector with 20 bits and is generated based on the selection rule S − d i − D S which gives higher selection probability to the items

(

)

that have closer difficulty level to the target D. It is observed from Table 2 that test items Q1, Q6, and Q10 will have the highest selection probability. With the integrity and selection rules, the initial swarm can be generated as shown in the first generation in Figure 1. Particle 1 selects items Q1, Q6, and Q8 for the first test sheet, and chooses Q1, Q4, Q5, Q8, and Q10 for the second test sheet. As for particle 2, the first test sheet consists of Q2, Q4, Q8, and Q10 and the second test sheet is composed of Q1, Q3, Q7, and Q9.

Generation 1:

Generation 2:

test sheet 1

test sheet 2

Zk

α

β

γ

J

particle 1 1000010100

1001100101

0.05

0

3

0

0.08

particle 2 0101000101

1010001010

0.285

3

0

0

0.315

particle 1 0100010100

1000100101

>0.08

-

-

-

-

particle 2 1000000100 1000101101 0.078 1 5 Figure. 1. Swarm evolution and the corresponding fitness evaluation

0

0.138 8

Particle fitness evaluation The particle fitness evaluation function

J ( x ) = Z k + w1α + w2 β + w3γ consists of objective value and

penalty terms which can be easily computed. In particular, particle 1 attains an objective value of 0.05 and incurs β penalty of 3 because the expected test time exceeds the upper limit, resulting a fitness value of 0.08. While particle 2 has an objective value of 0.285 and incurs α penalty of 3 due to the deficit of concept relevance. The fitness of particle 2 is thus 0.315. For the initial swarm, the personal best experience (pbesti) is the current particle itself. Since the fitness value of particle 1 is smaller, it is considered as gbest. The fitness values of pbesti and gbest will be used as bounds to expedite the process of next generation. Update of velocities and particles As for particle 1, the incumbent particle is equivalent to pbesti and gbest, resulting in the same vij values as previous ones and thus the same probabilities. Only a very small number of bits will be changed. Assume that particle 1 replaces Q1 by Q2 for the first test sheet and removes Q4 for the second test sheet (see generation 2 in Figure 1). For the case of particle 2, pbesti is the particle itself, but gbest is equivalent to particle 1, vij will be changed at the bits where pbesti and gbest are different. In essence, particle 2 will be dragged, to some extent, toward particle 1 in the discrete space. Assume that particle 2 becomes [10000001001000101101]. Use of bounding criterion for determining pbesti and gbest Now we proceed with the fitness evaluation for the two particles. For the case of particle 1, we find that the intermediate objective value during computation has already exceeded the current bound (0.08); hence, the computation is terminated according to the bounding criterion. There is also no need to derive penalty terms. As such the computational time is significantly reduced. As for particle 2, it attains an objective value of 0.078 and incurs α and β penalties of 1 and 5, resulting in a fitness of 0.138. Compared to incumbent pbest2 (fitness = 0.315), the fitness is improved, so pbest2 is updated to current particle 2; while gbest is not changed (fitness = 0.08). The STSCPSO algorithm iterates this process until a given maximum number of iterations has passed and gbest is considered as the best solution found by the algorithm.

5. Experiment and Discussion The STSCPSO approach has been applied to the development of an Intelligent Tutoring, Evaluation and Diagnosis (ITED III) system, which contains a large item bank for many science courses. The interface of the developed system and the experiments for evaluating the performance of the novel approach are given in the following subsections. A. System Development The teacher interface of ITED III provides a step-by-step instruction to guide teachers in defining the goal and parameters of a test. In the first step, the teacher is asked to define the type and date/time of the test. The test type might be “certification” (for performing a formal test), personal practice (for performing a test based on the to-be-enhanced concepts of each student), or group practice (for performing a test to evaluate the computer knowledge of a group of students). Each student is asked to take the test, using an assigned computer in a monitored computer room, and is allowed to answer the test sheet only within the specified test date/time range. In the following steps, the teachers are asked to define the parameters for composing the test sheets, such as the lower bound and upper bound of the expected test times (i.e., l and u), and the lower bound on the expected relevance of each concept or skill to be evaluated (i.e., hj). Figure 2 demonstrates the third step for defining a test with lower bound and upper bound of the expected test times being 60 minutes and 80 minutes, respectively. Moreover, in this step, the teacher is asked to define the lower bounds on the expected relevance of the concepts for the test, which are all set to 0.9 in the given example.

9

Figure. 2. Teacher interface for defining the goal and parameters of a test The entire test sheet is presented in one Web page with a scroll bar for moving the page up and down. After submitting the answers for the test items, each student will receive a scored test result and a personalized learning guide, which indicates the learning status of the student for each computer skill evaluated. Figure 3 is an illustrative example of a personalized learning guide for evaluating the skills of web-based programming. Such a learning guidance has been shown to be helpful to the students in improving their learning performance if remedial teaching or practice can be conducted accordingly (Hwang, 2003a; Hwang et al. 2003a; Hwang 2005).

Figure. 3. Illustrative example of a personalized learning guide 10

B. Experimental Design To evaluate the performance of the proposed STSCPSO algorithm, a series of experiments have been conducted to compare the execution times and the solution quality of three competing approaches: STSCPSO algorithm, Random Selection with Feasible Solution (RSFS), and exhaustive search. The RSFS program generates the test sheet by selecting test items randomly to meet all of the constraints, while the exhaustive search program examines every feasible combination of the test items to find the optimal solution. The platform of the experiments is a personal computer with a Pentium IV 1.6 GHz CPU, 1 GB RAM and 80G hard disk with 5400RPM access speed. The programs were coded with C# Language. To analyze the comparative performances of the competing approaches, twelve item banks with number of candidate items ranging from 15 to 10,000 were constructed by randomly selecting test items from a computer skill certification test bank. Table 3 shows the features of each item bank. Item bank 1 2 3 4 5 6 7 8 9 10 11 12

Table 3. Description of the experimental item banks Number of test Average Average expected answer time items difficulty of each test item (minutes) 15 0.761525 3.00000 20 0.765460 3.25000 25 0.770409 3.20000 30 0.758647 2.93333 40 0.720506 2.95000 250 0.741738 2.94800 500 0.746789 2.97600 1000 0.751302 2.97200 2000 0.746708 3.00450 4000 0.747959 3.00550 5000 0.7473007 2.99260 10000 0.7503020 3.00590

The experiment is conducted by applying each approach twenty times on each item bank with the objective values (Zk) and the average execution time recorded. The lower bounds and upper bounds of testing times are 60 and 120 minutes, respectively, and the maximal number of common test items between each pair of test sheets is 5. To make the solutions hard to obtain, we set the target difficulty level D = 0.5 which sufficiently deviates from the average difficulty of item banks. The STSCPSO algorithm is executed with 10 particles for 100 generations. The execution time of RSFS is set the same as that of the STSCPSO algorithm, while the maximal execution time of the exhaustive search is set to 7 days to obtain the optimal solution.

N 15 20 25 30 40 250 500 1000 2000 4000 5000 10000

Table 4 Experimental results STSCPSO RSFS Average Zk Average Zk Time (sec) Time (sec) 0.44 50 0.62 50 0.44 63 0.70 63 0.44 80 0.66 80 0.44 102 0.68 102 0.40 134 0.76 134 0.36 815 0.54 815 0.28 1805 0.68 1805 0.22 3120 0.46 3120 0.18 6403 0.60 6403 0.16 12770 0.62 12770 0.16 15330 0.54 15330 0.14 21210 0.56 21210

Optimum Solution Zk Average Time (day) 0.42 2 days > 7 days -

11

Table 4 shows the experimental results of objective values (Zk) and execution times using the three methods. We observe that the exhaustive search method can only obtain the optimal solution for the smallest test item bank with N = 15 since the computation complexity of the STSC problem is extremely high as described in Section III. As for the STSCPSO algorithm and the RSFS, approximate solutions to all of the test item banks can be obtained with reasonable times ranging from 50 seconds to 5.89 hours, but the solution quality delivered by the STSCPSO algorithm is significantly better. In particular, for N = 15, the objective value obtained by the STSCPSO algorithm is 0.44 which is very close to the optimal value (0.42), while the objective value obtained by the RSFS is 0.62. For the other cases with larger item banks, the superiority of the STSCPSO algorithm over the RSFS becomes more prominent as the size of the item banks increases. Figure 4 shows the variations of the objective value obtained using the STSCPSO algorithm and the RSFS. The objective value derived by the RSFS fluctuates, while the objective value derived by the STSCPSO algorithm constantly decreases since more candidate test items can be selected to construct better solutions as the test bank is larger.

Figure. 4. Variations of the objective value as the size of test item banks increases Figure 5 shows the fitness value obtained by gbest of the STSCPSO algorithm as the number of generations increases for the test item bank with N = 1000. We observe that the global swarm intelligence improves with a decreasing fitness value as the evolution proceeds. This validates the feasibility of the proposed particle representation and the fitness function fits the STSC problem scenario.

Figure. 5. Variations of the objective value as the size of test item banks increases

12

To analyze the convergence behavior of the particles, we testify whether the swarm evolves to the same optimization goal. We propose the information entropy for measuring the similarity convergence among the particles as follows. Let pij be the binary value of the jth bit for the ith particle, i = 1, 2, …, R, and j = 1, 2, …, NK, where R is the swarm size. We can calculate probj as the conditional probability that value one happens at the jth bit given the total number of bits that take value one in the entire swarm as follows.

∑ p = ∑ ∑ R

prob j

i =1

R

i =1

ij

NK

.

p h =1 ih

The particle entropy can be then defined as

Entropy = −∑ j =1 prob j log 2 ( prob j ) . NK

The particle entropy is smaller if the probability distributions are denser. As such, the variations of particle entropy during the swarm evolution measure the convergence about the similarity among all particles. If the particles are highly similar to one another, the values of the non-zero probj would be high, resulting in denser probability distributions and less entropy value. This also means the swarm particles reach the consensus about which test items should be selected for composing the test sheets. Figure 6 shows the variations of particle entropy as the number of generations increases. It is observed that the entropy value drops drastically during the first 18 generations since the particles exchange information by referring to the swarm’s best solution. After this period, the entropy value is relatively fixed due to the good quality solutions found and the high similarity among the particles, meaning the particles are resorting to the same high quality solution as the swarm converges.

Figure 6. The particle entropy as the number of generations increases

6. Conclusions and Future work In this paper, a particle swarm optimization-based approach is proposed to cope with the serial test sheet composition problems. The algorithm has been embedded in an intelligent tutoring, evaluation and diagnosis system with large-scale test banks that are accessible to students and instructors through the World-Wide Web. To evaluate the performance of the proposed algorithm, a series of experiments have been conducted to compare the execution time and the solution quality of three solution-seeking strategies on twelve item banks. Experimental results show that serial test sheets with near-optimal average difficulty to a specified target value can be obtained with reasonable time by employing the novel approach. For further application, collaborative plans with some local e-learning companies are proceeding, in which the present approach is used in the testing and assessment of students in elementary school and junior high schools.

13

Acknowledgement This study is supported in part by the National Science Council of the Republic of China under contract numbers NSC-94-2524-S-024-001 and NSC 94-2524-S-024 -003.

References Chou. C. (2000). Constructing a computer-assisted testing and evaluation system on the World Wide Web-the CATES experience. IEEE Transactions on Education, 43 (3), 266-272. Clerc, M., & Kennedy, J. (2002). The particle swarm explosion, stability, and convergence in a multidimensional complex space. IEEE Transaction on Evolutionary Computation, 6, 58-73. Eberhart, R. C., & Shi, Y. (1998). Evolving artificial neural networks. In Proceedings of the International Conference on Neural Networks and Brain, PL5-PL13. Fan, J. P., Tina, K. M., & Shue, L. Y. (1996). Development of knowledge-based computer-assisted instruction system. The 1996 International Conference Software Engineering: Education and Practice, Dunedin, New Zealand. Feldman, J. M., & Jones, J. Jr. (1997). Semiautomatic testing of student software under Unix(R). IEEE Transactions on Education, 40 (2), 158-161. Garey, M. R., & Johnson, D. S. (1979). Computers and Intractability: A Guide to the Theory of NPCompleteness, San Francisco, CA: Freedman. Hillier, F. S., & Lieberman, G. J. (2001). Introduction to Operations Research, 7th Ed., New York: McGrawHill. Hwang, G. J. (2003a). A concept map model for developing intelligent tutoring systems. Computers & Education, 40 (3), 217-235. Hwang, G. J. (2003b). A Test Sheet Generating Algorithm for Multiple assessment criteria. IEEE Transactions on Education, 46 (3), 329-337. Hwang, G. J., Hsiao, J. L., & Tseng, J. C. R. (2003a). A Computer-Assisted Approach for Diagnosing Student Learning Problems in Science Courses. Journal of Information Science and Engineering, 19 (2), 229-248. Hwang, G. J., Lin, B. M. T., & Lin, T. L. (2003b). An effective approach to the composition of test sheets from large item banks. 5th International Congress on Industrial and Applied Mathematics, Sydney, Australia, 7-11 July, 2003. Hwang, G. J. (2005). A Data Mining Algorithm for Diagnosing Student Learning Problems in Science Courses. International Journal of Distance Education Technologies, 3 (4), 35-50. Hwang, G. J., Yin, P. Y., Hwang, G. H., & Chan, Y. (2005). A Novel Approach for Composing Test Sheets from Large Item Banks to Meet Multiple Assessment Criteria. The 5th IEEE International Conference on Advanced Learning Technologies, Kaohsiung, Taiwan, July 5-8, 2005. Kennedy, J., & Eberhart, R. C. (1995). Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks, IV, 1942-1948. Kennedy, J., & Eberhart, R. C. (1997). A discrete binary version of the particle swarm algorithm. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, 4104-4108. Linderothm J. T., & Savelsbergh, M. W. P. (1999). A computational study of search strategies for mixed integer programming. INFORMS Journal on Computing, 11 (2), 173-187.

14

Olsen, J. B., Maynes, D. D., Slawson, D., & Ho, K. (1986). Comparison and equating of paper-administered, computer-administered and computerized adaptive tests of achievement. The Annual Meeting of American Educational Research Association, California, April 16-20, 1986. Rasmussen, K., Northrup, P., & Lee, R. (1997). Implementing Web-based instruction. In Web-Based Instruction, Khan, B. H. (Ed.), Englewood Cliffs, NJ: Educational Technology, 341–346 Shigenori, N., Takamu, G., Toshiku, Y., & Yoshikazu, F. (2003). A hybrid particle swarm optimization for distribution state estimation. IEEE Transaction on Power Systems, 18, 60-68. Trelea, I. C. (2003). The particle swarm optimization algorithm: convergence analysis and parameter selection. Information Processing Letters, 85, 317-325. Wainer, H. (1990). Computerized Adaptive Testing: A Primer, Lawrence Erlbaum Associates, Hillsdale, NJ. Yoshida, H., Kawata, K., Fukuyama, Y., & Nakanishi, Y. (1999). A particle swarm optimization for reactive power and voltage control considering voltage stability. In Proceedings of the International Conference on Intelligent System Application to Power Systems, 117-121.

15

Lai, K. R., & Lan, C. H. (2006). Modeling Peer Assessment as Agent Negotiation in a Computer Supported Collaborative Learning Environment. Educational Technology & Society, 9 (3), 16-26.

Modeling Peer Assessment as Agent Negotiation in a Computer Supported Collaborative Learning Environment K. Robert Lai Department of Computer Science and Engineering, Yuan Ze University, Taiwan [email protected]

Chung Hsien Lan Department of Computer Science and Engineering, Yuan Ze University, Taiwan [email protected] ABSTRACT This work presents a novel method for modeling collaborative learning as multi-issue agent negotiation using fuzzy constraints. Agent negotiation is an iterative process, through which, the proposed method aggregates student marks to reduce personal bias. In the framework, students define individual fuzzy membership functions based on their evaluation concepts and agents facilitate student-student negotiations during the assessment process. By applying the proposed method, agents can achieve mutually acceptable agreements that avoid the subjective judgments and unfair assessments. Thus, the negotiated agreement provides students with superior assessments, thereby enhancing learning effectiveness. To demonstrate the usefulness of the proposed framework, a web-based assessment agent was implemented and used by 49 information management students who submitted assignments for peer review. Experimental results suggested that students using the system had significantly improved learning performance over three rounds of peer assessment. Questionnaire results indicated that students believed that the assessment agent provided increased flexibility and equity during the peer assessment process.

Keywords Peer assessment, collaborative learning, agent negotiation, assessment agent, fuzzy constraint

1. Introduction Peer assessment is a widely adopted technique that can be applied to improve learning processes (Arnold et al., 1981; Falchikov, 1995; McDowell and Mowl, 1996; Freeman, 1995; Strachan and Wilcox, 1996). Computerbased environments typically enable students to develop their individual learning portfolios and conveniently assess those of their peers. Numerous researchers have investigated the effectiveness of computer-based peer assessment systems in various learning scenarios (Lin, 2001; Davies and Berrow, 1998). Davies developed a computer program to adjust the scores awarded to the coursework of others and encourage students to diligently and fairly review the coursework of their fellow students (Davies, 2002). Kwok and Ma generated a Group Support Systems (GSS) for collaborative and peer assessment (Kwok and Ma, 1999). Rada applied a Many Using and Creating Hypermedia system (MUCH) to solve exercise problems and submit solutions for peer review (Rada, 1998). Rapid development of Internet technologies has spawned extensive use of online web learning in education. Some researchers have explored the feasibility of Internet supported peer assessment (Davis and Berrow, 1998; Zhao, 1998; Liu et al., 1999; Lin et al., 2001). For example, Sitthiworachart designed a web-based peer assessment system that successfully assists students in developing their understanding of computer programming (Sitthiworachart and Joy, 2003). Liu et al. developed a Networked Knowledge Management and Evaluation System (NetKMES) for an instructional method that emphasizes expressing ideas via oral presentations and writing, accumulated wisdoms via discussion, critical thinking via peer assessment, and knowledge construction via project work. Student’s achievement increased significantly as a result of the peer assessment process and the number of students willing to take part in learning activities also significantly increased (Liu, 2003). Knowledge acquisition software was utilized for peer assessment systems to discover the conceptual framework and evaluation schemes for students during peer assessment. Instructors and students go online to exchange their understanding of criteria and portfolios, allowing them to reflect and thereby enhance the learning process (Ford et al., 1993; Liu et al., 2002). Computer-based peer assessment systems enhance the effectiveness of student-student and student-instructor communication and assist students in thinking reflectively. In the peer assessment process, each student assumes the role of a researcher who submits assignments as well as a reviewer who comments on their peers’ assignments (Rogoff, 1991). Therefore, students are involved in the learning and assessment processes. ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

16

Numerous studies have demonstrated that students benefit significantly from peer assessment; however, some students have revealed that peers may lack the ability to evaluate each others’ work or not take their role seriously, allowing friendships, entertainment value, etc., to influence the marks they give to peer coursework. Furthermore, students often lack experience in peer assessment and encounter difficulties in interpreting assessment criteria. These obstacles often result in subjective and unfair assessments and, thus, students’ reflection and learning effectiveness are not enhanced. This work presents a novel approach that models collaborative learning as multi-issue agent negotiation using fuzzy constraints. Agent negotiation (Pruitt, 1981) is an iterative process through which the proposed methodology aggregates students’ marks to reduce personal bias. In this framework, students define individual fuzzy membership functions based on their evaluation concepts and agents facilitate student-student negotiations during the assessment process. By applying this method, agents can reach mutually acceptable agreements that overcome the unfair assessment as a result of students’ various degree of understanding the assessment criteria. Thus, the negotiated agreement improves the assessment process and learning effectiveness. The remainder of this paper is organized as follows. Section 2 introduces the concept of peer assessment. Section 3 describes the overall framework of the assessment agent and its computational model. Then, a walk-through example is utilized to illustrate the application of the proposed methodology. Section 4 presents the experimental design, analytical results, and evaluation of the questionnaire results. Finally, Section 5 draws the conclusion.

2. Peer Assessment Assessment is a learning tool. However, most traditional assessment methods often encourage “surface learning,” which is characterized by information memorization and comprehension (Sitthiworachart and Joy, 2003). Falchikov defined peer assessment as “the process whereby groups rate their peers” (Falchikov, 2001). It can include student involvement not only in the final judgement made of student work, but also in the prior setting of criteria and the selection of evidence of achievement (Biggs, 1999). Peer assessment is an interactive assessment method that enhances student interpretation and reflection, enabling instructors to improve their understanding of student performance. This method moves coursework assessment from instructor-centered to student-centered. Based on previous research (Ford et al., 1993; Liu et al., 2002; Sitthiworachart and Joy, 2003), students are capable of learning how to criticize the work of their peers and accept peer criticism, thereby developing their critical thinking skills and self-reinforcement through peer assessment. Peer assessment requires cognitive activities such as reviewing, summarizing, clarifying, providing feedback, diagnosing errors and identifying missing knowledge or deviations (Van Lehn et al., 1995). However, based on the framework described by Sitthiworachart (Sitthiworachart and Joy, 2003) and Liu (Liu et al., 1999), the peer assessment process can be divided into four separate stages (Fig. 1). During stage 1, students complete their assignments on their own time and then submit assignments. During stage 2, peer reviewers assess peer assignments and then discuss their marks with the other students in their group who marked the same assignments. After the reviewers generate preliminary scores and feedback (stage 2), each student in the stage 3 evaluates the quality of marks given by the other markers. Finally, during stage 4, the results, which are aggregates of all marks, are sent to the original author who then revises the original assignment based on peer feedback. By this process, students typically develop a serious attitude toward their coursework. Stage 1 Do assignment

Stage 3 Stage 4 Stage 2 Mark quality of Show results Do peer assessment marking exercise Mark & Feedback Submit the assignment Scores & Feedback Figure 1. Peer assessment process

Thus, peer assessment entails formative work reviewing to provide feedback and summative grading. Moreover, peer assessment also integrates a feedback mechanism into the peer assessment process. Some studies indicate that receiving feedback is correlated with effective learning (Van Lehn et al., 1995; Bangert-Drowns et al., 1991). 17

Web-based peer assessment has recently gained popularity. Liu, who implemented a networked peer assessment system (NetPeas) to support peer assessment (Liu et al., 1999), indicated that web-based peer assessment positively affected learning effectiveness (Yu et al., 2004). A network peer assessment model allows students to learn or construct knowledge by submitting assignments and receiving comments from peers to improve their work. During the assessment process, students evaluate peer assignments via the Web, which ensures anonymity and positively affects students’ willingness to critique their peers’ work. Furthermore, web-based peer assessment allows instructors to monitor student progress at any point during the assessment process. That is, instructors can determine how well an assessor or assessee performs and constantly monitor the assessment process whereas this is nearly impossible during ordinary peer assessment when several rounds are involved (Chiu et al., 1998). Topping et al. described the potential advantages of peer assessment, including the development of the skills of evaluating, justifying, and using discipline knowledge (Topping et al., 2000). They also suggested that peer assessment can increase post hoc reflection, improves student ability to generalize knowledge to new situations, and promotes self-assessment and self-awareness. In addition to the potential advantages of peer assessment, some students feel negatively about peer assessment as markers are also competitors (Lin et al., 2001). During the peer assessment process, students may utilize different assessment criteria that produce different assessment results. Additionally, students often lack the ability to evaluate each other’s work or may apply judgments that are excessively subjective. Zhao also pointed out that students commonly believe that only instructors have the ability and knowledge required to effectively evaluate work and provide critical feedback (Zhao, 1998). This study employed a novel negotiation mechanism to balance individual assessment standard and utilized web technologies to implement this system to enhance student anonymity, reduce transmission costs and increase student interaction. Thus, the computational methodology employed combines fuzzy constraints and agent negotiation to overcome such difficulties. An assessment agent is deployed to support web-based peer assessment.

3. Assessment Agent 3.1 Methodology A novel methodology that provides computational support to elicit the assessment criteria and results is proposed. This methodology relies on fuzzy membership functions and agent negotiation to construct an assessment agent. Negotiation is a process of cooperative and competitive decision making between two or more entities. Agent negotiation is an iterative process through which a joint decision is made by two or more agents in order to reach a mutually acceptable agreement (Pruitt, 1981). Thus, the student, whose coursework is marked, can negotiate with markers using the assessment agent to reach a final assessment. Figure 2 presents the framework of peer assessment.

Assessment criteria

Student A’s Portfolio

Student B Self-assessment criteria

Student A Student A’s Portfolio

Final assessment results

Assessment Agent

Assessment criteria

Student A’s Portfolio

Student C

Assessment criteria

Student A’s Portfolio

Student D Figure 2. The framework of peer assessment

In this framework, students are free to construct personal assessment criteria when assessing portfolios. Assessment criteria, which consist of several evaluation concepts, are used to measure the qualities of students’ portfolios. Figure 2 shows a group of four students, A, B, C and D. Student A’s portfolio is assessed by students 18

B, C and D, who rate the portfolio by defining fuzzy constraints after identifying their own evaluation concepts. The role of the assessment agent is to negotiate the assessment of students A, B, C and D and to achieve an agreement. Based on this process, the assessment agent is considered a distributed fuzzy constraint network (Lai and Lin, 2004). Assessment criteria are regarded as negotiation issues or constrained objects and student assessments are fuzzy constraints. Figure 3 presents the workflow of the assessment agent. No Define Fuzzy constraints

Apply negotiation strategy

Negotiation (Offer generation) (Offer evaluation)

Reach an agreement

Yes Self-reflection and improvement

Produce the final results

Figure 3. The workflow of the assessment agent

In the first step of the assessment process, students evaluate the portfolios submitted by the other students and use their own fuzzy membership functions to mark. These fuzzy membership functions for assessment criteria are regarded as fuzzy constraints. After defining fuzzy constraints, the assessment agent applies concession and/or trade-off strategies to negotiate. Trough offers generation and evaluation, if an agreement cannot be reached, the fuzzy constraints or negotiation strategies must be adjusted. Conversely, if an agreement is reached, the interests of all students are considered to produce the final results. Then, the student submitting portfolio to be assessed by other students receives final scores, understands the peer assessments, and can reflect upon the assessment and revise the portfolio. During the negotiation process, each agent begins negotiating by proposing an ideal offer. However, when an offer is unacceptable to the other agents, these agents make concessions using a concession strategy or derive new alternatives using a trade-off strategy to move toward an agreement. Adopted from the framework in (Lai and Lin, 2004), the fuzzy constraint-based negotiation context is formalized as follows. ¾ ℜ is a set of agents involved in the negotiation, ℜ p ∈ ℜ is the one of members in ℜ where 1 ≤ p ≤ m and ¾ ¾ ¾

m is the number of ℜ . C p is a distributed fuzzy constraint network that represents an agent ℜ p . p Π C p is the intent of a distributed fuzzy constraint network C and represents the set of all potential

agreements for agent ℜ p . μ Π (u ) is the overall degree of satisfaction reached with a solution u. Cp

μΠ

wp

Cp

(1)

= min (( μC p (u )) q ) q =1,.., n

q

where n is the number of negotiation issues, wqp is the weight of issue q in agent ℜ p , and μ (.) is the C p q

degree of satisfaction for agent ℜ p and issue q. ¾ The process of negotiation is a series of determining how agents evaluate and generate alternatives from a possible designated space. ℑ(ℜ,α Π p ) denotes to find a final agreement for all agents in ℜ C i from α Π p . If ℑ(ℜ,α Π p ) holds, the negotiation is complete and terminates; otherwise, threshold i

C

i

C

α i will move to next lower threshold α i +1 and repeatedly applies ℑ(ℜ,α Π C ) to achieve an agreement. i +1

p

As the next lower threshold α over issue q is smaller than the minimal satisfaction degree δ q for issue q, the set of potential agreements over issue q would be j Π k and that of other issues is α Π k . C i +1 δ C q i +1

j

j

Then, ℑ(ℜ,α Π p ) will be false and the negotiation terminates until the next lower threshold α i +1 is C i lower than the overall minimal satisfaction degree, that is, α i +1 < arg min δ q . q =1..n

19

¾

ΨC p :α i Π C p → [0,1] is an evaluation function that represents the aggregate satisfaction value of

agent ℜ p for the potential agreement in α Π p . The aggregate satisfaction value is the measure of human C i preference. Given an offer (or counteroffer) u, the aggregate satisfaction value of u for agent ℜ p can be defined as 1 n wp (2) ( μC p (u )) q ∑ q =1 q n In a concession strategy, agents generate new proposals to achieve a mutual satisfactory outcome by reducing their demands. p ℘up is a set of feasible concession proposals at the threshold α qp for agent p ΨC p (u ) =

¾

αq

and it is defined as αqp

¾

℘up ={v | (μCp (v) ≥αqp ) ∧(Ψp (v) = Ψp (u) −r)}

(3)

where r is the concession value. In a trade-off strategy, agents can explore options for achieving a mutual satisfactory outcome by reconciling their interests. Agents can propose alternatives from a certain solution space and the degrees of satisfaction for constraints associated with the alternative are greater than or equal to an acceptable threshold. p Φ up is a set of feasible trade-off proposals at threshold α qp for the alternatives αq

of agent p and is defined as αqp

¾

Φup ={v | (μC p (v) ≥αqp ) ∧ (Ψp (v) = Ψp (u))}

(4)

where u is the latest offer. Agents maximize their individual payoffs and maximize the outcomes for all agents. Thus, a normalized Euclidean distance can be applied in measuring the similarity between alternatives to generate a best offer. The similarity function is defined as

∑

n q=1

(μ C p (v) −μ C p (u' ) + pC p (u' ))2

(5) n where n is the number of fuzzy constraints for agent p on issues, v is a feasible trade-off proposal of agent p, U′ is the set of counteroffers made by other agents, u ' = arg v ' max v '∈U ' ( μ C p ( v ) − μ C p ( v ' )) , μC p (v ) and μC p (u ' ) denote the satisfaction degree of qth fuzzy Θ (v,U ′) = 1 − p

q

q

q

q

q

q

q

constraint associated with v and u′ for agent p, and p (u ' ) is the penalty from the qth dissatisfied fuzzy C p q

constraint associated with offer u′ made by agent p. ¾

u* is the expected trade-off proposal for the next offer by agent p and is defined as u * = arg v (max v∈ Φ Θ p (v, U ' )) p

αq

where α qp is the highest possible threshold such that

α qp

p u

(6)

p p Φup ≠ {} and Θ (v,U ′) > Θ (u,U ′) .

By applying a similarity function, the best proposal that can be accepted by all participators will be found. The negotiation integrates the interests of all agents. If a final solution that satisfies all participators exists during a negotiation, a negotiation process will succeed. Otherwise, negotiation terminates and all agents adjust their interests prior to a new negotiation process. Multi-issue agent negotiation is supported in the assessment process. In particular, the core methodology of the assessment agent focuses on using fuzzy constraints to represent personal interests and applies negotiation strategies in making concessions or trade-offs between different possible values for negotiation issues. By applying constraints to express negotiation proposals, the model can execute the negotiation with increased efficiency and determine final results for overall assessments. 3.2 Illustrative Example To demonstrate the application of this approach, the following scenario in peer assessment is considered. The peer assessment system is designed to assist information management undergraduates to learn database design methodologies during Database System course. Following the instruction in a conventional classroom, the instructor assigned a database design project as homework. Students were required to submit portfolios that analyze the assigned project and the design methodology used to solve the database development problem. 20

Solving this problem involved the following database development concepts: defining, constructing, manipulating, protecting and sharing databases. In this example, three students (students I, J and K) were in a group. Student K submitted his portfolio and students I and J assessed independently student K’s portfolio. Students I, J and K were allowed to construct their own fuzzy membership functions for the same evaluation concepts such as Completeness, Security and Flexibility to assess the portfolio. Following the workflow presented in Fig. 3, the assessment process was divided into the following steps. Defining the Fuzzy Constraints The instructor first explained each evaluation concept to the students. Students I, J and K defined membership functions for each evaluation concept. Each membership function was considered as a fuzzy constraint. After assessing student K’s portfolio, students I, J and K illustrated their own fuzzy constraints (Figures 4(a)(b)(c)).

(a) Student I

(b) Student J

(c) Student K

Figure 4. Fuzzy constrains Student I specified the assessments as Fuzzy constraints, namely Low Completeness, Median Security and Median Flexibility. Student J defined Median Completeness, Low Security and Median Flexibility as fuzzy constraints and student K defined High Completeness, High Security, and High Flexibility as fuzzy constraints. Suppose students I, J and K adopted concession and trade-off strategies. Using Fuzzy Constraints to negotiation Agents I, J and K represent students I, J and K respectively. Negotiation in this example is a multi-issue negotiation among agents I, J and K. Agreement is achieved when all participants agree. Agents I, J and K took turns attempting to reach an agreement. Agent I proposed its assessments u1I = (60,70,70) related to Completeness, Security and Flexibility at threshold α 1I = 1 (Fig. 4(a)). However, according to (1), μ J (u1I ) = 0 C

and μ K (u1I ) = 0 , agents J and K did not accept u1I as an agreement. Subsequently, agent J proposed its offer C u1J = (70,60,70) at threshold α 1J = 1 . However, agents I and K also did not accept u1J as an agreement. Agent K

then proposed its offer u1K = (90,90,85) at threshold α1K = 1 . However, agents I and J still did not accept u1K as an agreement. Furthermore, assuming agent K adopted the fixed concession strategy and had no expected proposal at threshold α1K = 1 , agent K lowered its threshold to next threshold α1K = 0.9 and created a new set of feasible proposals as v2Ka = (90,90,83), v2Kb = (90,88,85), v2Kc = (88,90,85)

According to (5), the similarity among these feasible proposals was computed by agent K as Θ K (v 2Ka , u ' ) = 0.361, Θ K (v 2Kb , u ' ) = 0.366 , Θ K (v 2Kc , u ' ) = 0.374 21

Thus, agent K selected the most likely acceptable solution v2Kc = (88,90,85) as the offer for agents I and J. This procedure of offer evaluation and generation for agents I, J and K continued until an agreement was reached or no additional solutions were proposed. Through several rounds, negotiation finally reached an agreement over (completeness, security, flexibility) at (76, 76, 75), of which the satisfaction degree for agent I is (0.2, 0.4, 0.5); that of the agent J is (0.4, 0.2, 0.5), and that of the agent K is (0.3, 0.3, 0.5). The agreement reached u=(76,76,75), of which Ψ K (u ) = 0.367 , Ψ I (u) = 0.367 , Ψ J (u ) = 0.367 , u K ( u ) = 0.3 , u I (u ) = 0.2 and C

C

uC J ( u ) = 0.3 shows that the proposed approach, involving fuzzy constraint relaxation and similarity, helped the

assessment agent in arranging assessment criteria to meet each agent’s needs, and assisted agents in reaching an agreement that maximizes overall degree of satisfaction for assessments in the multi-issue negotiation.

4. Experiment and Results This experiment examined the usability and effectiveness of the assessment agent, a computational model that emphasizes critical thinking via peer assessment. Ninety-two university students in their junior year majoring in information management and enrolling in a Database System course were selected as study subjects. These students, who attended two sessions, were assigned to 26 teams, and each team was assigned to implement a database design project and undergo two examinations. However, the teams in each session were randomly divided into two groups. One group was the experimental group that participated in peer assessment using the web-based assessment agent, whereas the other group did not take part in any peer assessment activities. Therefore, 13 teams in the experimental group were assigned to group A and the remaining teams were in group B. The experimental process consisted of the following three steps. Analysis of Groups’ Difference The instructor gave all students an examination that tested knowledge of course content to determine whether the learning status of the two groups differed. After students completed the examination, the tests were marked by an instructor and analyzed using t-test analysis (Table 1). Table 1. The analysis of the test score difference between the two groups Session Group N Teams Mean t-value p-value Session 1 Group A 27 7 71.11 -0.005 0.995 Group B 22 7 71.14 Session 2 Group A 22 6 58.05 -0.172 0.864 Group B 21 6 59.10 Level of significance α =0.05 No significant difference in learning status was noted between the two groups (Table 1) as both p-values exceed the level of significance α . Therefore, the 13 selected teams were assigned to use the web-based assessment agent for peer assessment. Analysis of Teams’ Performance and Correlation During the peer assessment process, the instructor asked each team to design a project utilizing database theory that comprised preliminary plan, requirement analysis, conceptual database design, logistical database design, physical database design and implementation. All teams were instructed to do the following rounds. During round 1, they learned about database design theory and practiced on a case that needed to be assessed prior to attempting their projects. When all teams finished round 1, the 13 experimental teams submitted their projects and moved on to peer and self assessment using the evaluation concepts provided by the instructor via the assessment agent. Figure 5 displays the student interface of the assessment agent in defining fuzzy constraints. The instructor marked the submitted projects. After each team submitted fuzzy constraints for each project and obtained a negotiated agreement, these teams were allowed to revise their own projects and proceed to round 2 (preliminary plan and requirement analysis). Similarly, each team and the instructor completed the work. The overall 22

assessment proceeded until round 3 (the conceptual, logistical, physical database design and the implementation) was completed.

Figure 5. The student interface in defining fuzzy constraints Pearson’s correlation analysis is adopted to compare the correlations between peer and instructor marks. Student assessments are significantly and positively correlated with the instructors’ assessments for each evaluation concept utilized in the 3 rounds of assessment (Table 2). Table 2. Correlation analysis between instructor and student marks Round 1 Round 2 Round 3 Evaluation Concepts Correlation Correlation Correlation Completeness 0.457* 0.619* 0.752** Correctness 0.534* 0.745** 0.623* Originality 0.413* 0.712** 0.676* *: p < 0.05, **: p < 0.01 Paired t-test analysis for performance during the three rounds indicates that the improvement in learning for the 13 teams was significant (Table 3). Students improved their performance from round 1 to round 2, and especially from round 1 to round 3. Table 3. Performance analysis over the 3 rounds Round 1 and 2 Round 2 and 3 Round 1 and 3 Evaluation Concepts t-value p-value t-value p-value t-value p-value Completeness 5.84 2.89E-05 0.60 0.278 5.04 1.13E-04 Correctness 5.66 3.88E-05 1.25 0.116 5.25 7.82E-05 Originality 4.32 4.18E-04 0.39 0.348 4.66 2.22E-04 Level of significance α =0.05 Furthermore, as the difference cannot be perceived in round 1, the instructor’s assessment is utilized in round 2 and round 3 to evaluate the performance of the two groups. By t-test analysis, the difference between the two groups was significant (Table 4). Table 4. Performance analysis for the two groups Round 2 Round 3 Evaluation Concepts t-value p-value t-value p-value Completeness 2.02 0.029 2.24 0.019 Correctness 2.47 0.012 2.38 0.014 Originality 1.46 0.082 3.09 0.003 Level of significance α =0.05 23

Analysis of Individual Performance The instructor then gave all students a post-assessment examination related to course content and the project to determine whether each student’s performance in the 13 experimental teams differed from that of students in the other teams. Table 5 presents the differences of individual performance via the paired t-test and t-test. Analytical results indicate that students who participated in peer assessment had acquired more knowledge than students who did not. Furthermore, quantitative scores also indicate that students in the experimental group improved their performance via peer assessment. Session Session 1

Session 2

Table 5. Performance analysis for each student Group Examination N Mean t-value p-value Group A First 27 71.11 -4.43 0.000 Group A Second 27 78.22 Group B First 22 71.14 -0.32 0.375 Group B Second 22 71.36 Group A Second 27 78.22 1.79 0.039 Group B Second 22 71.36 Group A First 22 58.05 -6.52 0.000 Group A Second 22 71.77 Group B First 21 59.10 -1.59 0.063 Group B Second 21 62.86 Group A Second 22 71.77 1.90 0.003 Group B Second 21 62.86 Level of significance α =0.05

Evaluation of Questionnaire Results Following the experiment, students provided feedback via a questionnaire containing twelve questions. A 5point Likert scale was employed to grade responses. Questionnaire results indicate that students regarded the assessment agent as a satisfactory approach for flexibly assessing peer assignments, receiving fair feedback, and improving their performance. The questionnaire results are listed as follows: (A high mean indicates that students agree with the statement.) ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾ ¾

The assessment model is easily used. (Mean=4.02) The assessment model is flexible. (Mean=4.00) The assessment model is fair and, therefore, personal bias can be reduced. (mean=3.66) The assessment model is complex. (Mean=2.95) Students are provided with additional opportunities to reflect and improve their assignments via peer assessments. (Mean=4.39) The assessment standard can be constructed while reviewing peers’ assignments. (Mean=4.14) Peer assessment assists students in learning more than does instructor assessment. (Mean=4.05) Peer assessment is time and effort consuming. (Mean=3.22) Peer feedback encourages students to reflect self-assessment results. (Mean=4.05) Peer feedback enables students to improve their own assignments. (Mean=4.00) Peer assessment enhances the degree of seriousness students ascribe to their work. (Mean=3.82)

Overall, students agreed that the assessment agent benefits learning evaluation. (Mean=3.93) Although some students considered it time and effort consuming, most students believed that the system helped them to reflect on and improve their learning activities. Additionally, students felt that by relying on fuzzy membership functions and agent negotiation, the assessment model was flexible, fair and easy to use.

5. Conclusion This study has presented a computational model for peer assessment learning. By using this model, students are able to reach an agreement for peer assessment via agent negotiation. Experimental results indicate that this model significantly improved student performance. Students also agreed that the assessment agent is flexible and benefits the learning process. Based on these results, the assessment agent, which relies on fuzzy constraints and agent negotiation to facilitate peer assessment, has the following important merits. 24

First, the assessment agent can produce an objective assessment by considering assessments of all markers and the self-marker. Second, the level of flexibility increases by using fuzzy constraints. Furthermore, agent negotiation augments interaction among students. Third, the student assessed is provided with individual scores, rather than overall results based on assessment concepts. Thus, students can reflect on the representation over each assessment concept and thereby think more deeply to improve the quality of the portfolio. Fourth, final scores integrate all assessments and, thus, the assessment method overcomes the assessment bias. The scores achieved through this process of peer assessment are more reliable than scores assigned by an individual. In addition to eliminating individual bias, student learning effectiveness is enhanced through interaction with students. The assessment agent benefits student learning in many ways. The web-based assessment agent provides a higher degree of anonymity and lower transmission costs than traditional peer assessment processes. Furthermore, it eliminates time and location constraints and effectively offers interaction and feedback. Finally, although the proposed methodology has yielded promising results in promoting learning effectiveness, considerable work remains to be done, including further development of the methodology to allow students to select their own evaluation concepts, comparing assessment agents against other peer assessment methodologies, and experimental application of the methodology to other domains.

Acknowledgements This research is supported in part by the National Science Council of R.O.C. through Grant Number NSC 942745-E-155-007-URD and by the Center for Telecommunication Research of YZU. The authors also thank the anonymous reviewers for their helpful comments on this paper.

References Arnold, L., Willoughby, L., Calkins, V., Gammon, L., & Eberhart, G. (1981). Use of peer evaluation in the assessment of medical students. Journal of Medical Education, 56 (1), 35-42. Bangert-Drowns, R. L., Kulick, C. L. C., Kulick, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61, 213-238. Biggs, J. (1999). Teaching for Quality Learning at University, Buckingham: SRHE and Open University Press. Chiu, C. H., Wang, W. R., & Yuan, S. M. (1998). Web-based collaborative homework review system. Proceedings of ICCE’98, 474–477. Davies, P. (2002). Using agent reflective self-assessment for awarding degree classifications. Innovations in Education and Teaching International, 39 (4), 307-319. Davis, R., & Berrow, T. (1998). An evaluation of the use of computer supported peer review for developing higher-level skills. Computers and Education, 30, 111-115. Falchikov, N. (2001). Learning Together: Peer tutoring in higher education, London: Routledge Falmer. Falchikov, N. (1995). Peer feedback marking: developing peer assessment. Innovations in Education and Training International, 32, 175-186. Ford, K., Bradshaw, J., Adams-Webber, J., & Agnew, N. (1993). Knowledge acquisition as a constructive modeling activity. International Journal of Intelligent Systems, 8 (1), 9-32. Freeman, N. (1995). Peer assessment by groups of group work. Assessment and Evaluation in Higher Education, 20, 289-300. Kwok, R. C. W., & Ma, J. (1999). Use of a group support system for collaborative assessment. Computers and Education, 32 (2), 109-125.

25

Lai, K. R., & Lin, M. W. (2004). Modeling agent negotiation via Fuzzy constraints in E-Business. Computational Intelligence, 20 (4), 624-642. Lin, S. S. J., Liu, E. Z. F., & Yuan, S. M. (2001). Web-based peer Assessment feedback for students with various thinking styles. Journal of Computer-Assisted Learning, 17 (4), 420-432. Lin, S. S. J., Liu, E. Z. F., Chiu, C. H., & Yuan, S. M. (2001). Web-based peer assessment: attitude and achievement. IEEE Transaction on Education, 4 (2), 211. Liu, C. C., Liu, B. J., Hui, T. A., & Horng, J. T. (2002). Web based peer assessment using knowledge acquisition techniques: tools for supporting contexture awareness. International Conference on Computers in Education, Auckland, New Zealand, 4-6 December, 2002. Liu, E. Z. F. (2003). Networked knowledge management and evaluation system for CSCL: A case study. The Hawaii International Conference on Business, Hawaii, USA. Liu, E. Z. F., Lin, S. S. J., Chiu, C. H., & Yuan, S. M. (1999). Student participation in computer sciences courses via the Network Peer Assessment System. Advanced Research in Computers and Communications in Education, 2, 744-747. McDowell, L., & Mowl, G. (1996). Innovative assessment. In Gibbs, G. (Ed.), Improving student learning through assessment and evaluation, Oxford: The Oxford Centre for Staff Development, 131-147. Pruitt, D. G. (1981). Negotiation behavior, New York: Academic Press. Rada, T. (1998). Efficiency and effectiveness in computer-supported peer-peer learning. Computers and Education, 30 (3), 134-146. Rogoff, B. (1991). Social interaction as apprenticeship in thinking: guidance and participation in spatial planning. In L. B. Resnick, J. M. Levine, & S. Teasley (Eds.), Perspectives on socially shared cognition, Washington: APA Press, 349-383. Sitthiworachart, J., & Joy, M. (2003). Web-based peer assessment in learning computer programming. The 3rd IEEE International Conference on Advanced Learning Technologies, 180-184. Strachan, I. B., & Wilcox, S. (1996). Peer and self assessment of group work: developing an effective response to increased enrollment in a third year course in microclimatology. Journal of Geography in Higher Education, 20 (3), 343-353. Topping, K. J., Smith, E. F., Swanson, I., & Elliot, A. (2000). Formative peer assessment of academic writing between postgraduate agents. Assessment & Evaluation in Higher Education, 25 (2), 149-166. Van Lehn, K. A., Chi, M. T. H., Baggett, W., & Murray, R. C. (1995). Progress report: Towards a theory of learning during tutoring, Pittsburgh: Learning Research and Development Center, University of Pittsburgh. Yu, F. Y., Liu, Y. H., & Chan, T. W. (2004). A networked question-posing and peer assessment learning system: a cognitive enhancing tool. International Journal of Educational Technology Systems, 32, 211-226. Zhao, Y. (1998). The effects of anonymity on computer-mediated peer review. International Journal of Educational Telecommunications, 4 (4), 311-345.

26

Kiu, C.-C., & Lee, C.-S. (2006). Ontology Mapping and Merging through OntoDNA for Learning Object Reusability. Educational Technology & Society, 9 (3), 27-42.

Ontology Mapping and Merging through OntoDNA for Learning Object Reusability Ching-Chieh Kiu Faculty of Information Technology, Multimedia University, Jalan Multimedia, 63100 Cyberjaya, Malaysia [email protected]

Chien-Sing Lee Faculty of Information Technology, Multimedia University, Jalan Multimedia, 63100 Cyberjaya, Malaysia [email protected] ABSTRACT The issue of structural and semantic interoperability among learning objects and other resources on the Internet is increasingly pointing towards Semantic Web technologies in general and ontology in particular as a solution provider. Ontology defines an explicit formal specification of domains to learning objects. However, the effectiveness to interoperate learning objects among various learning object repositories are often reduced due to the use of different ontological schemes to annotate learning objects in each learning object repository. Hence, structural differences and semantic heterogeneity between ontologies need to be resolved in order to generate shared ontology to facilitate learning object reusability. This paper presents OntoDNA, an automated ontology mapping and merging tool. Significance of the study lies in an algorithmic framework for mapping the attributes of concepts/learning objects and merging these concepts/learning objects from different ontologies based on the mapped attributes; identification of a suitable threshold value for mapping and merging; an easily scalable unsupervised data mining algorithm for modeling existing concepts and predicting the cluster to which a new concept/learning object should belong, easy indexing, retrieval and visualization of concepts and learning objects based on the merged ontology.

Keywords Learning object, Semantic interoperability, Ontology, Ontology mapping and merging, Semantic Web

Introduction The borderless classroom revolves not only around the breaking down of geographical and physical borders but also, of cultures and knowledge. Realizing this vision however, requires orienting next-generation e-learning systems to address three challenges: first, refocusing development of e-learning systems on pedagogical foundations; second personalizing human-computer interaction in a seamless manner amidst a seemingly faceless technological world and third increasing interoperability among learning objects and among technological devices (Sampson, Karagiannidis & Kinshuk, 2002). A corresponding ontological solution, which addresses all these three issues, is Devedzic’s (2003) GET-BITS framework for constructing intelligent Web-based systems. Ontology is applied at three levels. The first is to provide a means to specify and represent educational content (domain knowledge as well as pedagogical strategies). The second application of ontology is to formulate a common vocabulary for human-machine interaction, human-human interaction and software-to-software agent interaction in order to enable personalization in presentation of learning materials, assessment and references adapted to different student models. The third application lies in systematization of functions (Mizoguchi & Bordeau, 2000) to enable intelligent help in learning systems, especially in collaborative systems. The concerns mentioned above point towards Semantic Web technologies as solutions. Layers in Semantic Web technologies are as shown in Figure 1 (Arroyo, Ding, Lara, Stollberg & Fensel, 2004). These layers appear to provide the solution to the third aspect mentioned by Sampson et al and Devedzic, which is also the focus in this paper. At the top most layer is the Web of trust, which relies on the proof and logic layers. The proof layer verifies the degree of trust assigned to a particular resource through security features such as digital signatures. The logic layer on the other hand, is formalized by knowledge representation data, which provides ontological support to the formation of knowledge representation rules. These rules enable inferencing and derivation of new knowledge. Basic description of data (metadata) is defined in the Dublin core such as author, year, location and ISBN. RDF (Resource Description Framework) schema and XML (eXtensible Markup Language) schema both provide means for standardizing metadata descriptions. RDF’s object-properties-values (OAV) building block creates a semantic net of concepts whereas XML describes grammars to represent document structure either through data type definitions (DTD) or XML schemas. Finally, all resources are tagged with uniform resource ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

27

identifiers to enable efficient retrieval. Unicode, a 16-bit character scheme, creates machine-readable codes for all languages.

Problem Definition and Background Background to the Problem The Semantic Web layers discussed above can be applied to the retrieval and reuse of learning objects as shown in Figure 2 (Qin & Finneran, 2002). According to Mohan and Greer (2003), “learning objects (LO) are digital learning resources which meet a single learning objective that may be reused in a different context”. Basic metadata to describe a learning object are defined in the Dublin core followed by contextual information that allows reuse of a learning object in different contexts. However, metadata expressed in XML addresses only lexical issues. XML does not allow interpretation of the data in the document. XML allows only lexical interoperability.

Figure 1. The Semantic Web stack

Figure 2. Representation granularity Furthermore, metadata schemas are user-defined. Due to differences in metadata specifications, metadata standardization initiatives such as ADL’s Sharable Content Object Resource Model (SCORM) and IEEE’s Learning Object Model (LOM) are introduced. However, LO reusability is complicated due to differences in LO metadata standards. For example, LO metadata standards such as IEEE LOM and Dublin Core are RDF binding whereas SCORM is XML binding. Hence, there are admirable efforts to map different learning object standards (Verbert, Gasevic, Jovanovic & Duval, 2005) in order to increase interoperability between learning object repositories. Significance of this work is the retrieval and assembly of parts of learning objects to create new learning objects suitable for different learning objectives and contexts. 28

On a higher layer of abstraction, additional constraints to schemas are introduced through W3C’s Web Ontology Language (OWL). OWL extends RDF’s OAV schemas using DAML (DARPA Agent Markup Language) and OIL (Ontology Inference Layer). OWL creates a common vocabulary by adding constraints to class instances, property value, domain and range of an attribute (property) by providing interoperability functions such as sameClassAs and differentFrom (Mizoguchi, 2000). However, there are also differences among ontologies. This paper deals with semantic and structural heterogeneity. Ontological semantic heterogeneity arises from two scenarios. In the first scenario, ontological concepts for a domain are described with different terminologies (synonymy). For instance, in Figure 3, the terms booking and reservation; client and customer are synonymous but termed differently. In the second scenario, different meanings are assigned to the same word in different contexts (polysemy). On the other hand, different taxonomies cause structural heterogeneity among ontologies (Euzenat et al., 2004; Noy & Musen, 2000; deDiego, 2001; Ramos, 2001; Stumme & Maedche, 2001). Instances of structural differences in Figure 3 are between the concepts airline and duration.

Figure 3. Semantic differences and structural differences between ontologies Hence, there is a need for ontology mapping and merging. Ontology mapping involves mapping the structure and semantics describing learning objects in different repositories whereas ontology merging integrates the initial taxonomies into a common schematic taxonomy. [As such, in this paper the term merged ontology is used interchangeably with the term shared ontology]. Problem Statement Four major issues constrain efforts toward ontology mapping and merging tasks: 1. Semantic differences: Some ontological concepts describe similar domain with different terminology (synonymy and polysemy). This results in overlapping domains. Thus, there is a need for mapping tools to interpret metadata descriptions from a lexical and semantic perspective to resolve the problems of synonymy and polysemy. 2. Structural differences: Structure in ontology mapping and merging refers to the structural taxonomy associating concepts (objects). Different creators annotate LOs with different ontological concepts. This creates syntactical differences, necessitating merging tools, which are able to capture the different taxonomies and merge the taxonomies into a common taxonomy. 3. Scalability in ontology mapping and merging: This is true especially for large ontological repositories where the growth of LO repositories over the network can be explosive. 4. Lack of prior knowledge: Prior knowledge is needed for ontology mapping and merging using supervised data mining methods. However, such knowledge is not always available. Thus, unsupervised methods are needed as an alternative in the absence of prior knowledge.

29

Research Questions We aim to design an automated and dynamic ontology mapping and merging framework and algorithm to enable syntactical and semantic interoperability among ontologies. Our research questions are: 1. Semantic interoperability: 2. How can we capture attributes describing concepts (objects) in order to map the attributes of concepts from different ontologies? 3. What is the best threshold value for automated mapping and merging between two ontological concepts based on experiments on 4 different ontologies? 4. Structural interoperability: How can we use these captured attributes to create a shared (merged) taxonomy from different ontologies? 5. Scalability: Which unsupervised data mining technique is sufficiently easy to scale? 6. Lack of prior knowledge for knowledge management: Which unsupervised data mining technique is easy to use for modeling existing concepts in the database and for predicting the cluster to which a new concept should be categorized into? Significance of the Study The contributions of this paper are: 1. An algorithmic framework for mapping attributes from concepts in different ontologies and for merging concepts from different ontologies based on the mapped attributes. 2. Determination of a suitable threshold value for automated mapping and merging 3. An easily scalable unsupervised data mining technique that enables modeling of existing concepts in the database and for predicting the cluster to which a new concept should be categorized into in order to enhance the management of knowledge. 4. Easy indexing and retrieval of LOs based on the merged ontology. 5. Easy visualization of the concept space based on formal context. The outline for this paper is as follows: First, we will present related work in ontology mapping and merging. This is followed by a discussion on the OntoDNA framework; the OntoDNA algorithm, simulation results on 4 different ontologies; and comparison with Ehrig and Sure’s (2004) integrated approach to ontology mapping in terms of precision, recall and f-measure. Next, we present how OntoDNA is applied in the CogMoLab integrated learning environment, in particular, in the OntoVis authoring tool. Finally, we conclude with future directions.

Related Work on Ontology Mapping and Merging Ontology mapping is a precursor to ontology merging. As mentioned earlier, ontology mapping is the process of finding the closest semantic and intrinsic relationship between two or more existing ontologies of corresponding ontological entities such as concept, attribute, relation, instance and etc. Ontology merging however, is the process of creating new ontologies from two or more existing ontologies with overlapping or similar parts (Klein, 2001). Given ontology A (preferred ontology) and ontology B as illustrated in Figure 4, ontology mapping is used to discover intrinsic semantics of ontological concepts between ontology A and ontology B. Concepts between ontologies may be related or unrelated. The ontology merging process looks at the semantics between ontologies and restructures the concept-attribute relations between and among concepts in order to merge the different ontologies.

Figure 4. Ontology Mapping and Merging Process 30

Ontology mapping and merging methods mainly can be categorized into two approaches, the concept-based approach and the instance-based approach. Concept-based approaches are top-down approaches, which consider concept information such as name, taxonomies, constraints and relations and properties of concept elements for ontology merging. On the other hand, instance-based approaches are bottom-up approaches, which build up the structural hierarchy based on instances of concepts and instances of relations (Gomez-Perez, Angele, FernandezLopez, Christophides, Stutt, & Sure, 2002). Some ontology mapping and merging systems, namely, Chimaera, PROMPT, FCA-Merge and ODEMerge are described in this section. Chimaera is a merging and diagnostic ontological environment for light-weight ontologies. The ontologies are automatically merged if the linguistic match is found. Otherwise, name resolution lists are generated, suggestion the terms from different ontologies to guide users in the merging process. The name resolution lists consists the candidate to be merged and taxonomic relationships that are yet to be merged into the existing ontology (McGuinness, Fikes, Rice, & Wilder, 2000). Similarly, PROMPT provides semi-automatic guided ontology merging. PROMPT is plugged into Protégé 2000. PROMPT’s ontology merging process is interactive. It guides the user through the ontology merging process. PROMPT identifies matching class names and iteratively performs automatic updates. PROMPT also identifies conflicts and makes suggestions on means to remove these conflicts to the user (Noy & Musen, 2000). A fully automated merging tool, ODEMerge is integrated with WebODE. ODEMerge performs supervised merging of concepts, attributes and relationships from two different ontologies using synonym and hypernym tables to generate the merged ontology. It merges ontologies with the help of corresponding information from the user. The user can modify results derived from the ODEMerge process (de-Diego, 2001; Ramos, 2001; Gomez et al., 2002). In contrast to Chimaera, PROMPT and ODEMerge, which are top-down approaches, FCA-Merge is a bottom-up ontology merging approach using formal concept analysis and natural language processing techniques. Given source ontologies, FCA-Merge extracts instances from a given set of domain-specific text documents by applying natural language processing techniques. The concept lattice, a structural result of FCA-Merge, is derived from the extracted instances using formal concept analysis. The result is analyzed and merged with the existing ontology by the ontology engineer (Stumme & Maedche, 2001).

Differences between OntoDNA and Related Work OntoDNA utilizes Formal Concept Analysis (FCA) to capture attributes and the inherent structural relationships among concepts (objects) in ontologies. FCA functions as a preprocessor to the ontological mapping and merging process. Semantic problems such as synonymy and polysemy are resolved as FCA captures the structure (taxonomy) of ontological concepts as background knowledge to resolve semantic interpretations in different contexts. Table 1. Comparison between OntoDNA and four ontology mapping and merging systems Problem addressed

Top-down Yes

Level of mapping

Approach Is the tool integrated in other ontology tool? Which? Concept definitions and slot values Taxonomies Instances of concepts Knowledgerepresentation supported Suggestion provided by the method

Chimarea Merging

Methodology or techniques supported

PROMPT Mapping and Merging Top-down Yes. Prompt Suite

9

9

9

9

ODEMerge Merging

FCA-Merge Merging

Top-down Yes. WebODE

Bottom-up No. It is a method

9

OntoDNA Mapping and Merging Top-down No.

9 9

9

9

9

No.

No.

No.

Lexicons

Name resolution lists are generated to guide users in the merging process -

A list of suggested concepts to be merged

-

-

Lexicons – string matching No.

-

-

Natural language processing and

Conceptual clustering (FCA)

31

conceptual clustering (FCA) Type of output

Merged ontology

Merged ontology

Merged ontology

Level of user interaction

-

Adjusting the system suggestion

Supply synonym and polysemy files for merging

Level of automaticity

Semi-automated

Semi-automated

Fully Automated

Merged ontology in concept lattice The produced merged ontology is fine-tuning by the user Fully Automated

Unsupervised data mining (SOM and k-means) Merged ontology in concept lattice -

Fully Automated

A hybrid unsupervised clustering technique, Self-Organizing Map (SOM) – k-means (both explained in Vesanto & Alhoniemi, 2000; Kiu & Lee, 2005) is employed by OntoDNA to organize data and reduce problem size prior to string matching in order to address semantic heterogeneity in different contexts more efficiently. A priori knowledge is not required in unsupervised techniques as the unsupervised clustering results are derived from the natural characteristics of the data set. SOM organizes data, clustering more similar objects together, while kmeans is used to reduce the problem size of the SOM map for efficient semantic heterogeneous discovery. Most of the mentioned tools are based on syntactic and semantic matching heuristics. User interaction on the ontology merging process is required to generate the merged ontology. However, OntoDNA provides automated ontology mapping and merging using unsupervised clustering techniques and string matching metric to generate the merged ontology. Similar to the above tools, OntoDNA allows the user to modify system-generated choices if he or she wants to. However, if the option is not selected, then the output from the system is automatically updated. Table 1 summarizes these comparisons in terms of the problem addressed, the type of approach, the level of mapping, the level of automation, the type of output and the presence or absence of knowledge-representation supported in ontology mapping and merging systems.

Figure 5. OntoDNA framework for ontology mapping and merging 32

OntoDNA Framework and Algorithm OntoDNA Framework The OntoDNA automated ontology mapping and merging framework is depicted in Figure 5. The OntoDNA framework enables learning object reuse through Formal Concept Analysis (FCA), Self-Organizing Map (SOM) and K-means incorporated with string matching based on Levenshtein edit distance. Ontological Concept Ontology is formalized as a tuple O: = (C, SC, P, SP, A, I), where C is concepts of ontology and SC corresponds to the hierarchy of concepts. The relationship between the concepts is defined by properties of ontology, P whereas SP corresponds to the hierarchy of properties. A is axioms used to infer knowledge from existing knowledge and I is instances of concept (Ehrig & Sure, 2004). Clustering Techniques Formal Concept Analysis (FCA) is a conceptual clustering tool used for discovering conceptual structures of data (Ganter & Wille, 1997; Lee & Kiu, 2005). To allow significant data analysis, a formal context is first defined in FCA. Consequently, the concept lattice is depicted according to the context to represent the conceptual hierarchy of the data. As shown in Table 2, the formal context for learning objects is contextualized. The concepts of the ontology are filled in the matrix rows and the corresponding attributes and the matrix columns represent relations of the concepts. A ‘1’ indicates the binary relation that the concept g has the attribute m. The source ontology and target ontology is first formalized using Formal Concept Analysis, followed by semantic discovery through string matching using Levenshtein edit distance.

description

title

publishedOn

abstract

softCopyURI

softCopyFormat

softCopySize

institution

volume

organization

school

chapter

publisher

journal

counter

type

note

keyword

pages

number

booktitle

series

address

edition

author

firstAuthor

editor

relatedProject

softCopy

PhdThesis Misc SoftCopy TechReport MastersThesis InBook InProceedings InCollection

version

Table 2. Example of a formal context for an ontology

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

1 1 0 1 1 1 1 1

To accelerate the ontology mapping and merging process, the source ontology is fed to the self-organizing map (SOM). The SOM is an unsupervised neural network used to cluster the data set according to similarity. SOM compresses complex and high-dimensional data to lower-dimensional data, usually to a two-dimensional grid. The most similar data are grouped together in the same cluster. The SOM clustering algorithm provides effective modeling, prediction and scalability to cluster data. An unsupervised clustering technique, k-means is applied on the learnt SOM to reduce the problem size of the SOM cluster. K-means iteratively divides a data set to a number of clusters and minimizes the error function. To compute the optimal number of clusters k for the data set, the Davies-Bouldin validity index is used (Vesanto & Alhoniemi, 2000; Kiu & Lee, 2004). The determination of the best number of k-clusters to be used based on the Davies-Bouldin index provides scalability to the modeling of ontological concepts. Figure 6 illustrates the trained SOM, k-means clustering of the trained SOM based on k = 3, and the clustering of new ontological concepts to the SOM cluster which best matches it. In the event of new concepts, they will be clustered into the same cluster as their best matching units. Subsequently, the new concepts are merged based on semantic similarity defined by Levenshtein edit distance and the older version of the source ontology will be dynamically updated with the new concept. 33

(c) (a) (b) Figure 6. (a) The trained SOM for the ontology, (b) K-Means clustering of trained SOM and (c) New Ontological Concepts to SOM’s BMU (boxed) String Matching Metric String matching based on Levenshtein edit distance is found to provide the best semantic similarity metric as demonstrated in the empirical experiment section. As such, we have applied it in the OntoDNA to measure similarity between two strings. Given two ontological elements, ontological element A, OE1 and ontological concept B, OE2, the edit distance calculated between OE1 and OE2, is based on simple editing operations such as delete, insert and replace. The result returned by string matching is in the interval range [0, 1], where 1 indicates similar match and 0 dissimilar match. OntoDNA Algorithm The terms used in the OntoDNA framework are first explained: ¾ Source ontology OS: Source ontology is the local data repository ontology ¾ Target ontology OT: Target ontology refers to non-local data repository ontology ¾ Formal context KS and KT: Formal context KS is the formal context representation of the conceptual relationship of the source ontology OS, meanwhile formal context KT is the formal context representation of the conceptual relationship of the target ontology OT. ¾ Reconciled formal context RKS and RKT: Reconciled formal context RKS and RKT are formal context with normalized intents of source and target ontological concepts’ properties. The prototypical implementation of the automated mapping and merging process illustrated in Figure 5 above is explicated below: Input : Two ontologies that are to be merged, OS (source ontology) and OT (target ontology) Step 1 : Ontological contextualization The conceptual pattern of OS and OT is discovered using FCA. Given an ontology O: = (C, SC, P, SP, A), OS and OT are contextualized using FCA with respect to the formal context, KS and KT. The ontological concepts C are denoted as G (objects) and the rest of the ontology elements, SC, P, SP and A are denoted as M (attributes). The binary relation I ⊆ G x M of the formal context denotes the ontology elements, SC, P, SP and A corresponding to the ontological concepts C. Step 2 : Pre-linguistic processing A similarity calculation, Levenshtein edit distance is applied to discover correlations between KS and KT attributes (ontological properties). The computed similarity value of ontological properties at or above threshold value is persisted in the context, else it appends into the context to reconcile the formal context, KS and KT. RKS and RKT are used as input for the next step.

34

Step 3

: Contextual clustering SOM and k-means are applied to discover semantics of ontological concepts based on the conceptual pattern discovered in the formal context, KS and KT. This process consists of two phases: (a) Modeling and training Firstly, SOM is used to model the formal context RKS to discover the intrinsic relationship between ontological concepts of the source ontology OS. Subsequently, k-means clustering is applied on the learnt SOM to reduce the problem size of the SOM cluster to the most optimal number of k clusters based on the Davies-Bouldin validity index. (b) Testing and prediction In this phase, new concepts from the target ontology OT are discovered by SOM's best-matching unit (BMU). SOM's BMU clusters the formal concepts RKT into its appropriate cluster without need for prior knowledge of internal ontological concepts. Step 4 : Post-linguistic processing The clusters, which contain the target ontological concepts, are evaluated by Levenshtein edit distance to discover the semantic similarity between ontological concepts in the clusters. If the similarity value between the ontological concepts are at or above threshold value, the target ontological concepts are dropped from the context (since they are similar to the source ontological concepts) and the binary relations I ⊆ G x M are automatically updated in the formal context. Else, the target ontological concept is merged with the source ontology. Finally a compounded formal context is generated. Output : Merged ontology in a concept lattice is formed. Mapping and Merging of Ontology OT to Ontology OS

Mapping between source ontology OS and target ontology OT is needed prior to merging ontology OT with ontology OS. Mapping between ontological elements (ontological concepts or ontological properties) is required to resolve semantic overlaps between OS and OT. To perform ontology mapping between OS and OT, each ontological element in source ontology OS is examined against each ontological element in the target ontology OT. Hence, the mapping algorithm of OntoDNA runs in O(nm) time where n and m are the length of the source and target ontological elements. The structure and naming conventions of source ontology OS are preserved in the mapping and merging process. OntoDNA addresses two types of mapping: matched case and unmatched case (or non-exact-match). A matched case occurs in one-to-one mapping, where the source ontological element correlates with a target ontological element. Meanwhile, the unmatched case (or non-exact-match) exists when: 1. there is no semantic overlap; where a source ontological element has no correlation with any target ontological element (no mapping) 2. there are semantic overlaps between many elements, where a source ontological element has correlation with more than one target ontological element (one-to-many mapping) In OntoDNA, simple mapping and complex mapping algorithms are adopted to map the target ontology OT to the source ontology OS. The simple mapping algorithm is implemented to handle one-to-one mapping and also in cases where there are no semantic overlaps. The simple mapping algorithm uses lexical similarity to perform mapping and merging between ontologies. The simple mapping algorithm is outlined as follows: Given source ontological element OelementSi and target ontological element OelementTj, apply lexical similarity measure (LSM) to map the target ontology OT to the source ontology OS at threshold value, t, where elements i and j = 1, 2, 3, …, n. a) map (OelementTj l OelementSi), if LSM(OelementSi, OelementTj) ≥ t; b) the target ontological element, OelementTj is mapped to (integrated with) the source ontological element, OelementSi and the naming convention and structure of the source ontological element, OelementSi are preserved. c) merge (OelementTj l OS), if LSM(OelementSi, OelementTj) < t; d) the target ontological element, OelementTj is merged (appended) to the source ontology and the naming convention and structure of the target ontological element, OelementTj are preserved. A complex mapping algorithm is used to handle one-to-many mapping, where multiple matches between the target ontological elements to a source ontological element are resolved based on relative frequency of instances of the ontological elements. The complex mapping algorithm is outlined as follows: 35

Given source ontological element OelementSi and its instances IelementSi and target ontological element OelementTj and its instances IelementTi, lexical similarity measure (LSM) and relative frequency of instances (fI) are applied to map the target ontology OT to the source ontology OS at threshold value, t, where ontological elements i and j = 1, 2, 3, …, n. a) map (OelementTj l OelementSi), if LSM(OelementSi, OelementTj) ≥ t AND LSM(OelementSi, OelementTj+1) ≥ t, where LSM(OelementSi, OelementTj) = LSM(OelementSi, OelementTj+1), if fI(IelementSi, IelementTj) > fI(IelementSi, IelementTj+1); the target ontological element, OelementTj is mapped to the source ontological element, OelementSi and the naming convention and structure of the source ontological element, OelementSi are preserved. For example, given threshold value, t = 0.8, target ontological elements, B, C, D, source ontological element, X, map (X l B), map (X l C) and map (X l D). If LSM = 0.8678 (above threshold value) and relative frequency, fI of ontological properties to each mapping is 7, 8 and 5 respectively, map C (highest frequency match) to the source ontological element, X.

Empirical Experiment The objective of this experiment is a) to investigate which lexical measure, i.e. string matching, Wordnet or a combination of string matching and Wordnet will provide more accurate semantic similarity results. The Levenshtein edit distance measure is used for string matching (Cohen, Ravikumar & Fienberg, 2003) whereas the Leacock-Chodorow for Wordnet linguistic matching (Pedersen, Patwardhan & Michelizzi, 2004). b) to identify the best threshold value for semantic similarity discovery to automate the ontology mapping and merging process. The mapping results generated by OntoDNA are compared against human mapping to evaluate the precision of the system. For this experiment, we used a threshold value from 0.6 to 1.0 for the evaluation. Other comparative experimental results highlighting OntoDNA’s degree of accuracy are found in Kiu & Lee (in press). Data Sets Four pairs of ontologies were used for evaluation. These ontologies and the human mapping results can be obtained from http://www.aifb.uni-karlsruhe.de/WBS/meh/mapping/. The details of the paired ontologies are summarized in Table 3. ¾ Pair 1, Pair 2 and Pair 3 are SWRC (Semantic Web Research Community) ontologies, which describe the domain of universities and research. SWRC1a contains about 300 entities including concepts, properties and instances. Meanwhile the three ontologies, SWRC1b, SWRC1c and SWRC1d are small ontologies, each of them containing about 20 entities. ¾ Pair 4 ontologies describe Russia. Each ontology contains about 300 entities. The ontologies are created by students to represent the content of two independent travel websites about Russia. Table 3. Ontologies used in the experiment Experiment

Ontology 1

Ontology 2

# Concepts

# Properties

# Total

Pair 1 Pair 2 Pair 3 Pair 4

SWRC1a SWRC1a SWRC1a Russian2a

SWRC1b SWRC1c SWRC1d Russian2b

62 59 60 225

143 144 143 122

205 203 203 347

Manual Mapping 9 6 3 215

Evaluation Metrics Information retrieval metrics such as precision, recall and f-measure are used to evaluate the mapping result from OntoDNA against human mapping (Do, Melnik, & Rahm, 2002). The objective and formula for each of these metrics are indicated below:

36

Precision: measures the number of correct mapping found against the total number of retrieved mappings. number of correct found mapping precision = Eq. 1 number of retrieved mappings Recall: measures the number of correct mapping found comparable to the total number of existing mappings. number of correct found mapping recall = Eq. 2 number of existing mappings F-measure: combines measure of precision and recall as single efficiency measure. 2 x precision x recall f - measure = Eq. 3 precision + recall

Results and Discussion The result of each semantic similarity approach is presented in Table 4. In terms of recall and precision, Levenshtein edit distance measure yields better results against the Leacock-Chodorow similarity measure and the combined WordNet-string matching approach in discovering correlation between two candidate mappings. String matching using Levenshtein edit distance generates an accuracy rate of 93.33% comparable to the human experts’ assessment as shown in Figure 7. Table 4. Comparison of lexical measures for 4 paired ontologies based on thresholds 0.6 to 0.1 Threshold Precision Recall F-Measure

t=0.6 0.6667 0.4444 0.5333

Threshold Precision Recall F-Measure

t=0.6 0.7500 0.5000 0.6000

Threshold Precision Recall F-Measure

t=0.6 0.5000 0.6667 0.5714

Threshold Precision Recall F-Measure

t=0.6 1.0000 0.1767 0.3004

Threshold Precision Recall F-Measure

t=0.6 0.7292 0.4470 0.5013

String Matching Measure (SM) Pair 1 t=0.7 t=0.8 t=0.9 0.8000 1.0000 1.0000 0.4444 0.2222 0.5556 0.5714 0.3636 0.7143 Pair 2 t=0.7 t=0.8 t=0.9 0.6667 0.7500 0.7500 0.3333 0.5000 0.5000 0.4444 0.6000 0.6000 Pair 3 t=0.7 t=0.8 t=0.9 1.0000 1.0000 1.0000 0.6667 0.3333 0.3333 0.8000 0.5000 0.5000 Pair 4 t=0.7 t=0.8 t=0.9 1.0000 0.9831 0.9752 0.1814 0.5395 0.5488 0.3071 0.6967 0.7024 Average t=0.7 t=0.8 t=0.9 0.8667 0.9333 0.9313 0.4065 0.3988 0.4844 0.5307 0.5401 0.6292

t=1.0 1.0000 0.2222 0.3636 t=1.0 0.7500 0.5000 0.6000 t=1.0 1.0000 0.3333 0.5000 t=1.0 0.9762 0.5721 0.7214 t=1.0 0.9315 0.4069 0.5463

WordNet Matching Measure (WN) Pair 1 t=0.6 t=0.7 t=0.8 t=0.9 t=1.0 0.3333 0.5714 0.6667 1.0000 1.0000 0.2222 0.4444 0.4444 0.2222 0.4444 0.2667 0.5000 0.5333 0.3636 0.6154 Pair 2 t=0.6 t=0.7 t=0.8 t=0.9 t=1.0 0.7500 0.8000 0.7500 0.6667 0.8000 0.5000 0.6667 0.5000 0.3333 0.6667 0.6000 0.7273 0.6000 0.4444 0.7273 Pair 3 t=0.6 t=0.7 t=0.8 t=0.9 t=1.0 0.2857 0.3333 0.2857 1.0000 1.0000 0.6667 0.6667 0.6667 0.3333 0.3333 0.4000 0.4444 0.4000 0.5000 0.5000 Pair 4 t=0.6 t=0.7 t=0.8 t=0.9 t=1.0 0.7778 0.7500 0.6667 0.9091 1.0000 0.0326 0.0279 0.0186 0.0465 0.0465 0.0625 0.0538 0.0362 0.0885 0.0889 Average t=0.6 t=0.7 t=0.8 t=0.9 t=1.0 0.5367 0.6137 0.5923 0.8939 0.9500 0.3554 0.4514 0.4074 0.2339 0.3727 0.3323 0.4314 0.3924 0.3491 0.4829

Precision

Recall

t=0.6 0.5000 0.4444 0.4706 t=0.6 0.8000 0.6667 0.7273 t=0.6 0.2857 0.6667 0.4000 t=0.6 1.0000 0.0465 0.0889 t=0.6 0.6464 0.4561 0.4217

Combined Measure (SM + WN) Pair 1 t=0.7 t=0.8 t=0.9 0.5714 0.5714 1.0000 0.4444 0.4444 0.2222 0.5000 0.5000 0.3636 Pair 2 t=0.7 t=0.8 t=0.9 0.8000 0.6000 0.7500 0.6667 0.5000 0.5000 0.7273 0.5455 0.6000 Pair 3 t=0.7 t=0.8 t=0.9 0.1667 0.2857 1.0000 0.3333 0.6667 0.3333 0.2222 0.4000 0.5000 Pair 4 t=0.7 t=0.8 t=0.9 0.9231 1.0000 0.9524 0.0558 0.0372 0.1860 0.1053 0.0717 0.3113 Average t=0.7 t=0.8 t=0.9 0.6153 0.6143 0.9256 0.3751 0.4121 0.3104 0.3887 0.3793 0.4437

t=1.0 1.0000 0.2222 0.3636 t=1.0 0.6667 0.3333 0.4444 t=1.0 1.0000 0.6667 0.8000 t=1.0 0.9643 0.2512 0.3985 t=1.0 0.9077 0.3683 0.5017

F-Measure

1.0000 0.8000 0.6000 0.4000 0.2000

String Matching Measure

WordNet Matching Measure

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

0.0000

Combined Measure

Figure 7. Average values of lexical measures 37

Generally, a threshold value of 0.8 or above improves OntoDNA’s precision and recall. Based on the individual ontology mapping performance, threshold value 0.8 provides the best measurement for ontology Pairs 1, 2 and 3 in terms of precision. However the recall values for the mapping are affirmative at the threshold value 0.7 as evidenced by ontology Pairs 1 and 3. The graphical representation of the experimental results for Levenshtein edit distance measure is shown in Figure 8, Figure 9 and Figure 10. 1.20

1.20

1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

0.20

0.20 0.00

0.00 Pair 1

Pair 2

Pair 3

Pair 1

Pair 4

Pair 2

Pair 3

Pair 4

Threshold = 0.7

Threshold = 0.6

(b)

(a) 1.20

1.20

1.00

1.00

0.80

0.80

0.60

0.60

0.40

0.40

0.20

0.20

0.00

0.00 Pair 1

Pair 2

Pair 3

Pair 4

Pair 1

Pair 2

Threshold = 0.8

Pair 3

Pair 4

Threshold = 0.9

(c)

(d)

1.20 1.00 0.80

Precision

0.60

Recall

0.40

F-Measure

0.20 0.00 Pair 1

Pair 2

Pair 3

Pair 4

Threshold = 1.0

(e) Figure 8. Precision, recall and f-measure of the paired ontologies at threshold value 0.6 to 1.0

1.20

Precision

Recall

F-Measure

1.00 0.80 0.60 0.40 0.20

Pair 1

Pair 2

Pair 3

Pair 4

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

0.00

Average

Figure 9. Average mapping results based on the threshold value

38

70%

Matching Accuracy

60% 50% 40% 30% 20% 10%

Pair 1

Pair 2

Pair 3

Pair 4

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

t=1.0

t=0.9

t=0.8

t=0.7

t=0.6

0%

Average

Figure 10. Improvement in matching accuracy of mapped ontologies The average mapping result for all four pairs (Figure 9) shows that the threshold value 0.8 generates the best precision and recall. It contributes towards improvement in the f-measure and matching accuracy in mapping (Figure 10). Therefore, threshold value 0.8 will be adopted to automate the ontology mapping and merging process. However, we desire to perform more experiments to justify the current threshold value for ontological domains other than that of the academic domain. OntoDNA and Ontology Mapping (An Integrated Approach) Comparison Evaluation Result The statistics for Ehrig and Sure’s (2004) ontology mapping on measures at cut-off using a neural net similarity strategy is extracted and compared with that of OntoDNA in terms of precision, recall and f-measure as shown in Table 5 below. The statistics indicate the best results obtained in terms of precision among metric measures and similarity strategies. Table 5. Summarized of precision, recall and f-measure

SWRC Russia2

Precision 0.9167 0.9752

OntoDNA Recall 0.4630 0.5488

Ontology Mapping (Ehrig & Sure, 2004) Precision Recall F-Measure 0.7500 0.6667 0.7059 0.7763 0.2822 0.4140

F-Measure 0.6048 0.7024

OntoDNA provides better precision compared to Ehrig and Sure’s ontology mapping tool (Figure 11). OntoDNA shows significant improvement in terms of recall and f-measure for the Russia2 ontology. This is evidence that OntoDNA can effectively address structural complexities and different ontological semantics. It is also noted that precision and recall relationships are inverse; increase in precision tends to result in decrease in recall, and vice-verse (Soboroff, 2005). There is tradeoff between precision and recall. Hence, it is up to the designer to decide on a suitable level of tradeoff.

precision, recall & f-measure

OntoDNA vs. Ontology Mapping (Ehrig & Sure, 2004) 1.2000

OntoDNA

Ontology Mapping (Ehrig & Sure, 2004)

1.0000 0.8000 0.6000 0.4000 0.2000 0.0000 SWRC

Russia2 precision

SWRC

Russia 2 recall

SWRC

Russia2

f-measure

Figure 11. Comparison result between OntoDNA with Ehrig and Sure’s ontology mapping 39

Learning Object Interoperability in CogMoLab with OntoDNA CogMoLab is an integrated learning environment comprising of the OntoID authoring tool (Lee & Chong, 2005), OntoVis authoring/visualization tool (Lee & Lim, 2005), Merlin agent-assisted collaborative concept map (Lee & Kuan, 2005) and OntoDNA (formerly named as OntoShare) ontology mapping and merging tool (Lee & Kiu, 2005). In CogMoLab, the student model (Lee, in press) forms the base component to provide intelligent adaptation with the applications to offer interactive environments for supporting student-learning tasks and instructors’ designing tasks. Currently, we plan to use OntoDNA to enable retrieval of resources from external repositories to enrich resources in OntoID, Merlin and OntoVis (Figure 12).

Figure 12. General view of learning objects interoperability with OntoDNA

Figure 13. Visualization of concepts through the OntoVis Concepts and instances in the OntoVis (Lim & Lee, 2005) are designed based on Formal Concept Analysis’ formal context. Currently, concepts and attributes are keyed in manually by the instructor into OntoVis’ formal context and visualized as shown in Figure 13. Hence, we also aim to enable visualization of the merged ontology through the OntoVis. 40

Conclusion This paper has described the OntoDNA framework for automated ontology mapping and merging and for dynamic update of the new ontological concepts in the existing knowledge base or database. The utilization of OntoDNA to interoperate ontologies for learning object retrieval and reuse from local and external learning object repository in CogMoLab has been explained. The merged ontology can be visualized through the OntoVis authoring/visualization tool in the form of a composited concept lattice. For future work, we will experiment with the discovery of instances from the ontological clusters through SOMk-means to enable more efficient query. We also desire to evaluate the performance of the OntoDNA on other ontological domains in terms of matching accuracy and threshold value. In the ASEAN seminar on e-learning, participating representatives have agreed to share knowledge and expertise with regards to human resource development through Information and Communications Technologies (ICT). One of the means for sharing knowledge is through the establishment of an ASEAN repository of learning objects. We hope that the OntoDNA will be able to contribute towards interoperability among standards and schemas and enrich the teaching and learning process, especially with the sharing of various cultures, not only in ASEAN but also with communities of practice in other parts of the world.

References Arroyo, S., Ding, Y., Lara, R., Stollberg, M., & Fensel, D. (2004). Semantic Web Languages - Strengths and Weakness. International Conference in Applied computing (IADIS04), Lisbon, Portugal. Cohen, W., Ravikumar, P., & Fienberg, S. (2003). A Comparison of String Distance Metrics for Name-matching tasks. In IIWeb Workshop held in conjunction with IJCAI, retrieved June 20, 2006 from http://www.cs.cmu.edu/~wcohen/postscript/ijcai-ws-2003.pdf. de-Diego, R. (2001). Método de mezcla de catálogos electrónicos, Final Year Project, Facultad de Informática de la Universidad Politécnica de Madrid. Spain. Devedzic, V. (2003). Key issues in next-generation Web-based education. IEEE Transactions on Systems - Man, and Cybernetics, Part C, 33 (3), 339-349. Do, H., Melnik, S., & Rahm, E. (2002). Comparison of schema matching evaluations. In Proceedings of the 2nd International.Workshop on Web Databases (German Informatics Society). Ehrig, M., & Sure, Y. (2004). Ontology Mapping - An Integrated Approach. Lecture Notes in Computer Science, 3053, 76-91. Euzenat, J., Bach, T. L., Barrasa, J., Bouquet, P., Maynard, D., Stamou, G., Stuckenschmidt, H., Zaihrayeu, H., Hauswirth, M., Ehrig, M., Jarrar, M., Shvaiko, P., Dieng-Kuntz, R., Hernández, R. L., Tessaris, S., & Acker, S. V. (2004). D2.2.3: State of the art on current alignment techniques, retrieved April 5, 2006 from http://knowledgeweb.semanticweb.org/. Ganter, B., & Wille, R. (1997). Applied Lattice Theory: Formal Concept Analysis, retrieved June 10, 2006 from http://citeseer.ist.psu.edu/rd/0%2C292404%2C1%2C0.25%2CDownload/http://citeseer.ist.psu.edu/cache/papers/ cs/14648/http:zSzzSzwww.math.tu-dresden.dezSz%7EganterzSzconcept.pdf/ganter97applied.pdf. Gomez-Perez, A., Angele, J., Fernandez-Lopez, M., Christophides, V., Stutt, A., & Sure, Y. (2002). A survey on ontology tools, OntoWeb deliverable 1.3. Universidad Politecnia de Madrid. Kiu, C. C., & Lee, C. S. (2004). Discovering Ontological Semantics using FCA and SOM, In Proceedings of M2USIC2004, Cyberjaya, Malaysia, 7-8 October 2004. Kiu, C. C., & Lee, C. S. (2005). Discovering Ontological Semantics for Reuse and Sharing of Learning Objects in a Contextual Learning Environment. The 5th International Conference on Advanced Learning Technology (ICALT2005), July, 8-9 2005, Kaohsiung, Taiwan. Kiu, C. C., & Lee, C. S. (2006). A Data Mining Approach for Managing Shared Ontological Knowledge. The 6th International Conference on Advanced Learning Technology, July 5-7, 2006, Kerkrade, The Netherlands. 41

Klein, M. (2001). Combining and relating ontologies: an analysis of problems and solutions. In Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI-01), Workshop: Ontologies and Information Sharing, USA. Lee, C. S. (in press). Diagnostic, predicting and compositional modeling with data mining in integrated learning environments. Computers & Education, Elsevier. Lee, C. S., & Chong, H. R. (2005). Synergistic design considerations for continuing education: Refocusing on instructional design. WSEAS Transactions on Advances in Engineering Education, 2 (4), 294-304. Lee, C. S., & Kiu, C. C. (2005). A concept-based graphical –neural approach to ontological interoperability, WSEAS Transactions on Information Systems and Applications, 2 (6), 761-770. Lee, C. S., & Kuan, C. L. (2005). Intelligent scaffolds for collaborative concept mapping, WSEAS Transactions on Information Systems and Applications, 2 (8), 1157-1166. Lee, C. S., & Lim, W. C. (2005). Visualization for course modeling, navigation and retrieval: Towards contextual learning and cognitive affordance. WSEAS Transactions on Advances in Engineering Education, 2 (4), 347-355. Lim, W. C., & Lee, C. S. (2005). Knowledge discovery through composited visualization, navigation and retrieval. Lecture Notes in Artificial Intelligence, 3735, 376-378. McGuinness, D. L., Fikes, R., Rice, J., & Wilder, S. (2000). An environment for merging and testing large Ontologies. Proceeding 7th International Conference on Principles of Knowledge Representation and Reasoning (KR’2000), Breckenridge, Colorado, USA, 483-493. Mizoguchi, R., & Bordeau, J. (2000). Using ontological engineering to overcome AIED problems. Journal of Artificial Intelligence in Education, 11, 107-121. Mohan, P., & Greer. J. (2003). Reusable Learning Objects: Current Status and Future Directions. In D. Lassner & C. McNaught (Eds.) Proceedings of ED-MEDIA 2003 World Conference on Educational Multimedia, Hypermedia and Telecommunication. Honolulu, Hawaii, USA. Noy, N. F., & Musen, M. (2000). PROMPT: Algorithm and Tool for Automated Ontology Merging and Alignment. In Proceedings of the 17th National Conference on Artificial Intelligence (AAAI’00), Austin, USA. Qin, J., & Finneran, C. (2002). Ontological representation for learning objects. In Proceedings of the Workshop on Documents Search Interface Design and Intelligent Access in Large-Scale Collections, JCDL'02, Portland, OR, July 18, 2002. Pedersen, T., Patwardhan, S., & Michelizzi J. (2004). WordNet: Similarity - Measuring the Relatedness of Concepts, In the Proceedings of the 19th National Conference on Artificial Intelligence (AAAI-04), San Jose, CA. Ramos, J. A. (2001). Mezcla automática de ontologías y catálogos electrónicos. Final Year Project. Facultad de Informática de la Universidad Politécnica de Madrid. Spain. Soboroff, I. (2005). IR Evaluation, retrieved http://www.csee.umbc.edu/~ian/irF02/lectures/09Evaluation.pdf.

December

2,

2005

from

Sampson, D., Karagiannidis C., & Kinshuk (2002). Personalized Learning: Educational, Technology and Standardization Perpective. Interactive Educational Multimedia, 4, 24-39. Stumme, G., & Maedche, A. (2001). FCA–Merge: A Bottom–Up Approach for Merging Ontologies. In IJCAI ’01 - Proceedings of the 17th International Joint Conference on Artificial Intelligence, Morgan Kaufmann. USA. Verbert, K., Gasevic, D., Jovanovic, J., & Duval, E. (2005). Ontology-based Learning Content Repurposing. WWW2005, Chiba, Japan, May 10-14, 2005. Vesanto, J., & Alhoniemi. E. (2000). Clustering of the Self-Organizing Map. IEEE Transactions on Neural Networks, 11 (3), 586-600.

42

Suhonen, J., & Sutinen, E. (2006). FODEM: developing digital learning environments in widely dispersed learning communities. Educational Technology & Society, 9 (3), 43-55.

FODEM: developing digital learning environments in widely dispersed learning communities Jarkko Suhonen and Erkki Sutinen Department of Computer Science, University of Joensuu, Finland, P.O.BOX 111, FI-80101 Joensuu, Finland [email protected] [email protected] ABSTRACT FODEM (FOrmative DEvelopment Method) is a design method for developing digital learning environments for widely dispersed learning communities. These are communities in which the geographical distribution and density of learners is low when compared to the kind of learning communities in which there is a high distribution and density of learners (such as those that exist in urban areas where courses are developed and taken by hundreds or thousands of learners who are simultaneously present in the area). Since only limited resources can be allocated for the design of a digital learning environment for widely dispersed learning communities, it is necessary to use what limited funds are available to obtain valid feedback from stakeholders and to utilize such feedback in an optimal way. In terms of the FODEM model, the design process consists of asynchronous development threads with three interrelated components: (1) needs analysis, (2) implementation, and (3) formative evaluation. In needs analysis, both theory and practice are used to define specifications for the environment. In implementation, fast prototyping in authentic learning settings is emphasized. Finally, formative evaluation is used to evaluate the use of the environment within the thread. FODEM has been applied to develop ViSCoS (Virtual Studies of Computer Science) online studies and LEAP (LEArning Process companion) digital learning too in the rural regions of Finland where the population is geographically widely dispersedly.

Keywords Design Methods, Digital Learning Environments, Online Learning, Formative Evaluation

Introduction Digital learning environments (DLEs) are technical solutions for supporting learning, teaching and studying activities (Suhonen, 2005). A digital learning environment can be educational software, a digital learning tool, an online study program or a learning resource. A digital learning environment may thus consist of a combination of different technical solutions. A DLE may thus be used as the basis for an e-learning program (Anohina, 2005). The development of effective DLEs is not an easy task. The challenge when developing DLEs for us to use technology ingeniously and creatively to solve problems and meet the needs that arise in various technological, educational and cultural contexts (Kähkönen et al., 2003). The best design methods are those that help designers to develop innovative and effective solutions by clearly depicting the most important procedures and aspects of the development process (Design-Based Research Collective, 2003). A widely dispersed learning community refers to a student population that is thinly distributed over a relatively large geographical region or throughout a long period of time. Since widely dispersed learning communities like this are restricted by cultural, geographical or temporal factors, they tend to be relatively small in number. A widely dispersed learning community might, for instance, consist of 30 students who live in an area with a radius of 200 kilometers and who are learning Java programming. Such characteristics have two consequences: firstly, a thinly distributed community needs outstandingly accessible DLEs because the students live too far away from one another to offer or receive assistance, and, secondly, not even the sum of fees collected from such a small number of students can finance excellent DLEs of the kind we are contemplating. Such difficulties have given rise to in poorly designed ad hoc DLEs. Table 1 presents typical differences between widely dispersed and dense learning communities. The United Kingdom Open University (UKOU) is an example of a dense learning community where the student numbers are reasonably high. For instance, the UKOU offers a full range of degrees and it has over 200,000 students. According to Castro et al. (2001) and Bork and Gunnarsdottir (2001), the UKOU has a full-scale preparation system for the courses. Several years and millions of euros are invested to make high quality learning products to be used over several years. The evaluations and course improvements are often conducted to the final version of a DLE. This paper explains how we applied a FODEM (FOrmative DEvelopment Method) to create effective DLEs for the Finnish context which is well known for its widely dispersed learning communities. Apart from meeting the needs of a widely dispersed learning community, FODEM has to prove itself as a practicable DLE design method. Whatever method is used, it has to be responsive to the diversity of learners’ needs and situations, to ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

43

whatever technologies are available, and to the cultural idiosyncrasies of the learning context (Soloway et al., 1996). The lack of unified theories for functional specifications of DLEs can make the design situation unpredictable (Moonen, 2002). Such uncertainty gives, for instance, rise to the need for a formative design process. This is especially applicable when the whole design situation is new, and the requirements and expectations of the developed environment can easily change during development (Fallman, 2003). Understanding the core problems and needs of learners is a crucial aspect of the development process.

Design costs per course Number of students Design team Development time Primary use of technology Production Feasible DLEs

Table 1. Features of widely dispersed and dense learning communities Sparse Dense Not more than US$10,000 100,000US$ or more Fewer than 100

From several hundred up to thousands 10-20 specialists

A few people with multi-disciplinary skills and several responsibilities 2-4 person months Information processing

1-2 years Information delivery

Tailor-made Multipurpose digital learning tools

Mass production Re-usable learning objects

In this paper we describe how we applied FODEM to two DLE development scenarios – ViSCoS and LEAP. ViSCoS is an online study program for first-year university-level courses in computer science. The ViSCoS curriculum consists of three main areas: the preliminaries of ICT, the basics of programming with Java, and an introduction to computer science (Sutinen & Torvinen, 2001). ViSCoS first began to be developed in 2000. Between 2000 and 2005, a total of 109 high school students completed the program. LEAP is a digital learning tool that was developed in response to challenges that arose during the ViSCoS Programming Project course (Suhonen & Sutinen, 2004). The main functions of LEAP are a digital learning portfolio and creative problem solving support (Suhonen, 2005).

Formative development method – FODEM Thread as a core structure The most basic concept in FODEM is that a thread represents an identified developmental theme within a DLE development process. Such a theme represents a particularly important area or need that has to be addressed if a learning process and its outcomes are to be improved by a DLE. An example of a thread is the design process of a crucial component in a system of DLEs – such as a visualization tool in a programming course. We use the term thread rather than stage or phase because FODEM development process consists of several parallel and simultaneous asynchronous threads. The thread structure differentiates the FODEM method from, say, many general software design methods which proceeds linearly, stage by stage (Whitten et al., 2004). Threads in a development process can be asynchronous or they can progress independently of one another. Thread structures are used because they represent the most important aspects of the development process. In spite of this, threads do not necessarily reflect all aspects of a development process. Because resources for design and implementation of a widely dispersed learning community context are limited, it is better to focus on the most important features (which are threads) and undertake quality work with them rather than create a flat learning environment which in which one cannot penetrate too deeply into any given area because it would be at the cost of other areas. The developers of FODEM were inspired in this regard by jagged study zones which are a cross between open learning environments and intelligent tutoring systems (Gerdt et al., 2002). A thread has three interdependent, dialectic components: needs analysis (NA), implementation (I) and formative evaluation (FE), and it can include several instances of such components. Each thread has a certain development theme which aligns the goals of a thread and relates the components to one another. The line (left thread in Figure 1) shows an indefinite interaction among components. One of the main principles of FODEM is to gradually develop the DLE on the basis of the feedback received from the stakeholders (students, teachers, administrative staff and designers alike). It is important to note that more threads can be added into the process on the basis of feedback received. In the beginning, there might only be one thread and a need to implement the 44

first version of a digital learning environment as quickly as possible. As a formative design method FODEM emphasizes research-orientation throughout the entire lifecycle of a DLE. Various thread types can also be identified. In a cycle of development thread type, the components within a thread have an iterative dependency. With a cycle of development thread, one can, for instance, represent how a certain development theme progresses clearly in linear or spiral form. This means that typically a cycle of development thread includes several instances of components. Figure 1 illustrates how two threads might work together in a development process. The thread on the left illustrates a cycle of a development thread.

First designs of courses NA

I

Single course NA

Focus on a special case FE

I

FE

Figure 1. Two threads in a development process FODEM components In a needs analysis, designers identify the needs and pedagogical objectives of the DLE and the consequent requirements of a design context. A needs analysis includes the definition of the main concepts, roles and desired goals of the environment in terms of the thread’s theme. In the early stages of development, the most urgent tasks are given the highest priority. The definition of pedagogical objectives can be grounded on both theory and practice. Learning theories, for example, can be a useful resource for suggesting various approaches and ideas to the designer. But design solutions may also be the product of human innovation and ingenuity, particularly in circumstances where no relevant theories exist. The requirements of the design context may also be constructed from extrapolating from experiences of successful practical developments in similar situations in the past. Needs analysis may also include an analysis of formative evaluations in other threads.

Tasks

Methods

Outcomes

Challenges

Table 2. Summary of the FODEM components NA I Identify the design solutions and Implement design solutions as main concepts (Kerne, 2002). quickly as possible so that experiments with learners can take place as soon as possible (Soloway et al., 1996). Analysis of contextual factors, Fast prototyping, sketching, learning theories, and practical experimenting (Bahn & experiences. Literature reviews. Nauman, 1997). Evaluation of experiences from other threads. Pedagogical and technical design An environment that is usable principles and solutions in authentic learning settings To incorporate and combine the design ideas from different origins in an effective and meaningful way (Abdelraheem, 2003)

To expose the environment to users at the right moment.

FE Identify viable features by evaluating.

Use of environmental and experiential analysis, content analysis. Information about the functions of the environments in future development To go beyond the structure of the design (Kommers, 2004)

The implementation component is used to implement the design solutions identified in needs analysis. One of the applied implementation methods is fast prototyping. This is used to test design ideas at an early stage for information about which design solutions are viable and which are unsuccessful. This can be achieved through 45

constructive feedback elicited from stakeholders. The developed environment must be developed to a level of maturity where its core functionalities are operating satisfactorily. While usability need not be fully refined, the attention of learners may be constantly distracted to secondary issues if there are too many usability problems. In the worst case scenario, learners might not even realize the main functions of the environment. If learners are exposed too late to the development process, major changes to the core functionality of the environment will be costly and complex to implement. Furthermore, learner participation in the implementation process (through making suggestions and proposing developmental ideas) also benefits the learners themselves. The third component is formative evaluation by means of which users’ experiences of the developed environment are evaluated. In FODEM, multiple data sources and pluralistic research methods ensure the production of a comprehensive, rich and layered overall picture. Activity logs, databases, and browsing histories can be used to evaluate learners’ usage of the environment. As part of FE, developers can also interpret users’ personal opinions, experiences and perceptions of the environment by using (most typically) interviews and questionnaires for to elicit data. Designers need insight, sensitivity and creative empathy to identify those often implicit or hidden viewpoints, attitudes and emotions that users have but sometimes cannot or will not reveal. An evaluation can also focus on obtaining practical design ideas from users themselves. Research results can indicate new problems or design ideas for future development. Table 2 summarizes the tasks, methods, outcomes and challenges of FODEM components. Dependencies: interdependent structure of threads Dependencies in FODEM are used to represent the interdependent structure of threads, the interaction among components and the critical relationships among components in different threads. A dependency is represented pictorially by an arrow showing the direction of the interaction. A dependency between threads and single components can be one-directional or bi-directional (a bi-directional dependency represents a mutual dependence between two components or threads). The name of a dependency shows the nature and meaning of the interaction. A thread dependency shows that a whole thread is initiated by another thread. Figure 2 illustrates a possible FODEM scenario. The dependency between Threads 1 and 2 illustrates how Thread 1 created a need for the re-design of the environment’s technical architecture (a thread dependency). The arrow from Thread 3 to Thread 2 illustrates that the implementation component in Thread 2 is dependent on the needs analysis component in Thread 3.

Figure 2. A possible FODEM scenario In principle, many layers of threads can exist. When a development process is extensive, a thread may be divided into a set of sub-threads. Should a sub-thread become too large, it may be reconstituted into a separate development process. The dependencies in multi-layered threads exist among thread layers and sister subthreads. Figure 3 visualizes an example of a multi-layered thread structure. Methods that are similar to FODEM FODEM has been inspired by general software development and educational technology design models. The System Development Life Cycle (SDLC) and Context Design (CD) originate from the software development tradition while Instructional Design (ID) and Design Research (DR) are used specifically in educational technology. Common to all of these approaches are specification (S), implementation (I) and evaluation (E) 46

phases. The specification phase is used to determine the needs, requirements and expectations for the solutions. In the implementation phases, the solution is implemented. Finally, the evaluation phase is used to analyze implementation. Table 3 shows how the different ideas (marked in bold) from similar methods relate to FODEM.

Figure 3. Multi-layered thread structure Table 3. Characteristics of FODEM’s sister methods in the specification (S), implementation (I), and evaluation (E) phases SDLC CD ID DR S De-contextualized Learner-centered User needs Practical knowledge requirements design I Iterative through SDLC methods Sketching for Embedded in use prototyping, sequential presenting the design versions, structured ideas to users approaches E Summative, phased Stories of experience, Based on theoretical Experimenting, actual observations development reflection, formative models of human behavior evaluation SDLC models are often based on structured approaches (Whitten et al., 2004). The development of any software is thought to take place in a series of phases (Löwgren, 1995). In traditional methods, evaluation is often conducted on the final product. In newer SDLC models, the development process follows an iterative, spiral model in which design ideas are tested in the early stages of development. In CD models, the emphasis is on problems that relate to particular user concerns such as, for example, how to recognize the needs of users (Löwgren, 1995). The focus in ID models falls more on the pedagogical side of the development process (Moallem, 2001). Contemporary ID models, for example, stress learner-centered approaches, reflection and formative development (McKenna & Laycock, 2004; Moonen, 2002). DR includes a series of approaches that issue in technological designs and practices for learning and teaching (Barab & Squire, 2004). The emphasis there is on testing systems in authentic settings. The development of an environment is blended with a strong theoretical emphasis on human behavior. If one compares it with similar methods, the uniqueness of FODEM consists of the way in which it models the parallel, but interdependent and asynchronous units of the DLE development – units that we call threads.

FODEM in action FODEM in ViSCoS development ViSCoS (Virtual Studies of Computer Science) is an online study program offered by the Department of Computer Science, University of Joensuu (ViSCoS, 2005). ViSCoS students study first-year university-level 47

Computer Science courses by means of the Web. The curriculum consists of three main parts: the preliminaries of Information and Communication Technology (ICT), basics of programming with Java, and an introduction to Computer Science (Torvinen, 2004). The development of ViSCoS began in 2000, and the first students commenced studies that same year. One hundred and nine (109) students had completed the program by the end of 2004. While ViSCoS studies were initially offered only to high school students between the ages of 16 and 18 who lived in the rural areas of Eastern Finland, subsequent collaboration between the Department of Computer Science and the Continuing Education Centre at the University of Joensuu made it possible for this course to be taken by anyone in Finland. Table 4 summarizes the content of ViSCoS courses. Table 4. ViSCoS courses Course Introduction to Information and Communication Technology (ICT) and Computing Programming I Programming II Hardware, Computer Architecture and Operating Systems Programming Project Introduction to the Ethics of Computing Design of Algorithms Research Fields in Computer Science

Content Computer hardware components, computer programs and operating systems, data communications and local area networks, network services and digital communication, controlling the risks of information technology. Practical skills needed for using word processing and spreadsheet applications, basics of Unix, introduction to HTML. Algorithmic thinking. Basic structures of programming with Java. Introduction to object-oriented programming (objects, classes, inheritance) An overview of architecture of computers, parsers, system software, and databases Software design, implementation, testing and documenting General principles of the ethics of computing. Introduction to basic issues in computer science; algorithms, computation, and data structures. Introduction to a selection of research fields in Computer Science

There are five threads in ViSCoS development: first designs, loop of improvements, new technical solutions to support learners, ViSCoS Mobile, and English version. We implemented and ran the first versions of the courses in the first designs thread in 2000 and 2001. The design of ViSCoS is based on the CANDLE scheme (Torvinen, 2004), the primary goal of which is to get courses up and running as quickly as possible. We went out of our way to rework these initial courses so that they reflected the needs and interests of the kind of young high school students who would take the course. Once they had been modified and adjusted, the course examples and assignments were more topical and reflective of the everyday concerns and interests of this target group. While these first courses were running, we collected feedback from all the stakeholders. Our main research efforts were dedicated to identifying and understanding the reasons why students encountered difficulties in the ViSCoS program (Meisalo et al., 2003). In pursuit of our aim of achieving a clear understanding of how students experienced the courses, we made use of questionnaires, interviews, log files, analyses of examination and submitted exercises and assignments, and analyses of student and tutor feedback to elicit the required information. First evaluations indicated that it was the programming courses that caused students the greatest difficulties since the highest drop-out rates had occurred there. The Programming Project course in particular was evidently a difficult one for students. In the second thread – loop of improvements – we addressed these problems by improving the programming courses in whatever was we could (Torvinen, 2004). We made changes, for example, on the basis of feedback from Thread 1 to the course structures and learning materials. One such change required us to merge two courses into one. After careful consideration of the student feedback we received, we then also modified the curriculum so that students could give more of their attention to the more demanding topics and so that they would have more time for Programming I (we increased the duration of the course from 10 to 12 weeks). We also created optional exercises, new examples and assignments, as well as interactive animations, to illuminate what we by then knew were the most difficult student topics (arrays, loops, and methods). Several studies within the loop of improvements also investigated the reasons why students were electing to drop out of the ViSCoS program. (This loop of improvements thread has been active since 2001, and research efforts to investigate the drop-out phenomenon are still continuing.) Our aim in the third thread, new technical solutions to support learners, was to develop and introduce new technical solutions that would enhance ViSCoS studies. We introduced the first of these new solutions in 2002. 48

Four sub-threads can be identified: the LEAP digital learning tool, the ethical argumentation tool called Ethicsar, the Jeliot program visualization tool, and data mining techniques. LEAP is a tool for helping students to manage their programming projects in the Programming Project course. Details of LEAP development are presented in the next section. Ethicsar is a web-based tool for argumentation and evaluation of ethical issues and questions (Jetsu et al., 2004). Jeliot permits novice programmers to visualize their own Java code (Moreno et al., 2004). The development of data mining techniques focused specifically on processing data related to the course assignments. Our ultimate aim here was to create intelligent support for learners by means of instructional intervention (Hämäläinen et al., 2006). Dependency between ViSCoS and the technological solutions is bidirectional (Figure 4). The introduction of these improved technologies has meant that students now receive better and more comprehensive support. The two most recent threads are ViSCoS Mobile and the English version. ViSCoS Mobile was created so that students would be able to use a mobile device to study their ViSCoS courses (Laine et al., 2005). Because this is a relatively new concept, only the first procedures in the needs analysis component are currently being implemented. An English version thread has also been created so that the ViSCoS courses can be offered to English-speaking students. The first English-medium learning materials were implemented in the courses Introduction to ICT and Computing, Programming I, Introduction to Ethics of Computing and Research Fields of Computer Science in September and December of 2005. Table 5 summarizes how the ViSCoS program was developed, while Figure 4 depicts the process graphically. Table 5. ViSCoS development (Th = Thread) NA I Prioritization of the tasks, Running the first identifying the contextual versions of the courses factors of the high school students

Th 1

Theme First designs

2

Loop of improvements

Identification of the most difficult aspects of the courses

3

New technical solutions created to support learners ViSCoS Mobile

Needs identified while running the first versions of the course. Practical experiences. Content adaptation of course materials. Programming with a mobile device. Student support activities in a mobile device (eg. Mobile Blog)

–

–

English version

Analysis of the current Finnish versions

Implementation of the English version. Improvements included in the Finnish version.

–

4

5

Improvement of the program on the basis of feedback received from learners Development of Ethicsar, Jeliot, LEAP, and data mining

FE Feedback obtained from all stakeholders through direct questions, interviews, content analysis. Drop-out phenomenon, questions, interviews, analysis of exercises The evaluation of the implemented solutions from many different points of view

FODEM in the development of LEAP Figure 4 shows ViSCoS development at a higher level of abstraction. Details may be examined by zooming into a particular thread. The LEAP digital learning tool development, for instance, can be divided into a set of subthreads. LEAP application has two main functions. These are digital learning portfolio and creative problemsolving support (Suhonen & Sutinen, 2004). Three sub-threads can be identified: the first prototype, re-design of the technical implementation, and a mobile adaptation extension. We based the first prototype in Thread 1 on the needs that we had identified in the ViSCoS Programming Project course. Some kind of tutorial mechanism was needed to help students to manage their projects in a creative way. This tool was also inspired by digital learning portfolio, creative problem-solving support and provocative agent concepts (Suhonen, 2005). But we did not restrict the implementation of LEAP to the Programming Project course. Our purpose was rather to implement a generic gadget that could be used in several different settings. The formative evaluation component in the first thread included two studies on use of the tool in authentic learning settings. In the first study, we used the tool in the Problem Solving contact teaching course at the 49

University of Joensuu (we used the digital learning portfolio functionality in this study). In the second study, we used the tool in the ViSCoS Programming Project course. This study allowed us to test the LEAP tool in its original design context (we used both the digital learning portfolio and creative problem-solving support functionalities of the tool). In both studies we used a dual evaluation scheme. These two parts dealt with the analysis of how students were using the tool (use analysis) and the analysis of students’ opinions about the tool (experience analysis). We investigated usage by analyzing the content of students’ contributions with LEAP, and elicited learners’ reflective impressions and opinions from interviews. The first designs

Loop of improvements

NA

NA

ogramming Focus on pr FE

I

FE

Programmin

e ng ha ex c ata t, d me n ve pro

NA

NA

NA

Im

Etchic s

NA

I

Data mining

NA

In I

FE

English version

tio ra g te

n,

em ov r p im

ts en

FE

FE

LEAP pta tio n

I

Jeliot

da

Ethicsar

FE

nt a

I

NA

I

nte

FE

Co

I

rse

C

ew ,n ts l su g e r e an h h rc xc ea a e s re at re s, d a p ea om id

g Project Cou

of Com puting

Cours

e

I

FE

ViSCoS Mobile

Figure 4. Development of ViSCoS In the second thread, re-design of the technical architecture, we modified LEAP in terms of the findings of the two studies within the first thread. In that thread we identified problems by means of the usability of the tool. Our main finding was that LEAP was complicated to use and that it required simplification. We added no new features within the second thread, but merely refined existing features. We also re-implemented the technical architecture of the tool because of requirements posed by the third thread. We then conducted a third study within the second thread in the ViSCoS Programming Project course, and evaluated it with an evaluation scheme that was similar to the approach that we had used in our earlier studies. Our main finding was that LEAP should be customized for the context of the course since students were of the opinion that the tool was not relevant (Suhonen, 2005). Thread 3, a mobile adaptation extension, is still in the concept design phase (Kinshuk et al., 2004). Although we had decided that we should begin to implement the mobile adaptation after the core functions of LEAP had become sufficiently mature, the needs analysis for the technical implementation of the mobile adaptation extension affected the implementation in the second thread. The mobile adaptation extension will certainly influence the ViSCoS Mobile thread at the higher level. Figure 5 visualizes the development of LEAP. It shows the interaction among the three threads. Table 6 shows the details of the LEAP development process. 50

Th 1

Table 6. Summary of the LEAP development NA I Pedagogical and technical design; First designs digital learning portfolio and creative problem-solving support

Theme First prototype

2

Re-design of the technical architecture

Ideas and knowledge from the first thread. Requirements from the mobile adaptation extension thread.

XML-based implementation

3

Mobile adaptation extension

Concept design of mobile adaptation extension

–

The first prototype

Re-design of the technical architecture

NA

NA

FE Two studies; Programming Project and Problem Solving courses A study in the ViSCoS Programming Project course. Comparison with previous studies. –

Mobile adaptation extension Requirements and needs NA

Experiences I

FE1

I

FE

I

FE

FE2

Comparison, changes to the evaluation scheme

Figure 5. Visualization of the LEAP development

Woven Stories-based environment to support FODEM The multithreaded structure of FODEM allows us to specify a corresponding visual environment to support the design process. The first sketch for a FODEM design environment prototype has been implemented. The prototype is built on a client-server web-based application called Woven Stories (WS) (Gerdt et al., 2001; Nuutinen et al., 2004). The prototype can be used to manage the threads, components and dependencies in a development process. A designer can add, remove and edit threads to represent the main aspects of the development process. Information can be added to each component in a thread. Although the current version of the prototype allows the inclusion of web links to the components, users cannot add documents to the environment. Dependencies can be created between threads and components. The client-server implementation enables a number of users to work with the same design process from remote locations. The prototype also includes a chat facility which allows designers to interact while they work with the environment. Figure 6 shows the FODEM design environment prototype. The next challenge in the design environment development was to improve the preliminary prototype that would enable more meaningful support for the designers. The first new feature would have to be the addition of arbitrary documents to components in threads. Designers could, for example, add requirements documents, articles and review results to a needs analysis component. The environment would also need to include basic functionalities for document management. The design environment would also need a zooming functionality that would help designers to grasp the development process at various levels. The environment should also automatically construct different visualizations of the development process by, for example, presenting the process in a time-based view, or by separating the most important threads (as defined by the designers) from other threads. A comprehensive design environment would also include built-in procedures that would facilitate working with components. A formative evaluation component, for instance, could include a list of possible evaluation methods, and the designers could get some information about their applicability in different situations. This kind 51

of service would help designers to decide which evaluation methods would be most suitable for a given development phase (Reeves & Hedberg, 2003). Tacit knowledge about different aspects of the development process could also be stored in the environment. Finally, such a web-based environment would help designers to work collaboratively with one another, to exchange ideas and information about problems in different threads, and ultimately to undertake collaborative management of the development process.

Figure 6. FODEM design environment The possibilities that FODEM opens up for a novel DLE design tool (such as the one described above) is yet another indication of the effectiveness and feasibility of the FODEM approach. The inherently parallel structure of FODEM allows the designers to manage a complex but need-based process for the development of effective DLEs.

Conclusions In this paper we introduced the FODEM method for developing digital learning environments in the context of widely dispersed learning communities. We also used ViSCoS and LEAP development cases in a Finnish educational context to show how the method works. The ViSCoS program has now been running for five years and has been proven to be sustainable. ViSCoS provides a flexible solution for studying first-year computer science courses over the net. But the development of LEAP is still in its early stages. Three prototyping experiences have in the interim revealed both some positive and negative features in LEAP. The two cases have shown how FODEM can be used to develop different types of digital learning environments. FODEM provides tools to conceptualize the most important aspects of a development process with thread, component and dependency structures (Suhonen, 2005). FODEM includes tools that can capture the dynamics of a development process and that can be used to model various representations of the FODEM development process from different perspectives. FODEM also supports the integration of different development processes: a thread within a development process can be a part of another development process. An important aspect of FODEM is its emphasis on the utilization of all feedback from the users of the developed digital learning environment. Case studies, interviews, user observations and contextual analysis methods are among the appropriate evaluation techniques that developers utilize gradually to improve the environment.

52

The unique features of FODEM are evident from three features of the development of DLEs for widely dispersed learning communities. Firstly, development costs should be low. Both ViSCoS and LEAP have been developed with reasonably low investments (approximately US$10,000 per course). Secondly, needs analysis and research orientation in FODEM can be used to create contextualized and appropriate DLEs. When we developed ViSCoS, for example, we used a number of methods to fit the course arrangements and digital learning materials to the students’ needs. Finally, the development process should follow a simple, yet structured, design method. FODEM models the development process with parallel, interrelated threads. The formative evaluation component ensures that the development process is both rigorous and focused. One of FODEM’s best -known characteristics is how well it adapts to the context of everyday school life and youth culture. Far too frequently, development methods (and hence the DLEs) were far too heavily loaded or rich in technical features. This might account for the fact that ICT has in the past been widely underutilized in several educational contexts. FODEM – in contrast to the proliferating approach of comprehensive design methodologies – emphasizes simplicity. At the same time, FODEM’s inherently parallel approach which represents and reflects the dynamic nature of most educational contexts, gives due weight to the complexity of learning and teaching needs. Unlike rapid prototyping, FODEM does not reduce the simultaneous needs that are identified in real school contexts into a sequential design process. FODEM therefore copes better with the uncertainty that is characteristic of real life, a common expectation of ICT in other areas of society as well. Because FODEM treats the individual needs of heterogeneous student communities with appropriate seriousness, it is an affordable alternative to design information-delivery-oriented DLEs for dense communities which take into account individual needs through adaptive technologies or re-usable learning objects.

Acknowledgements We are grateful to Teemu Laine for implementing the FODEM design environment prototype on the basis of the Woven Stories application. We also thank Roger Loveday for his work on revising the text.

References Abdelraheem, A. Y. (2003). Computerized learning environments: Problems, design challenges and future promises. The Journal of Interactive Online Learning, 2 (2), 1-9. Anohina, A. (2005). Analysis of the terminology used in the field of virtual learning. Journal of Educational Technology & Society, 8 (3), 91-102. Bahn, D. L., & Naumann, J. D. (1997). Evolution versus construction: Distinguishing user review of software prototypes from conventional review of software specifications. In Proceedings of the ACM Conference on Computer Personnel Research (SIGCPR’97), New York: ACM Press, 240-245. Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13 (1), 1-14. Bork, A., & Gunnarsdottir, S. (2001). Tutorial Distance Learning: Rebuilding Our Educational System, New York: Kluwer Academic/Plenum Publishers. Castro, M., López-Rey, A., Pérez-Molina, C.M., Colmenar, A., de Mora, C., Yeves, J., Carpio, J. Peire, J., Daniel, J.S. (2001). Examples of distance learning project in the European Community. IEEE Transactions on Education, 44 (4), 406-410. Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32 (1), 5-8. Fallman, D. (2003). Design-oriented human-computer interaction. In Proceedings of the Conference on Human Factors in Computing Systems (CHI 2003), New York: ACM Press, 225-232. Gerdt, P., Kommers, P., Looi, C.-K., & Sutinen, E. (2001). Woven Stories as a cognitive tool. Lecture Notes in Computer Science, 2117, 233-247. 53

Gerdt, P., Kurhila, J., Meisalo, V., Suhonen, J., & Sutinen, E. (2002). Rambling through the wilds: A concept of Jagged Study Zones. In Proceedings of the World Conference on Educational Multimedia, Hypermedia & Telecommunications (ED-MEDIA2002), Norfolk: AACE, 596-601. Hämäläinen, W., Laine, T. H., & Sutinen, E. (2006, forthcoming). Data mining in personalizing distance education courses. In Romero, C. & Ventura, S. (Ed.), Data Mining in E-learning. WitPress. Jetsu, I., Meisalo, V., Myller, N., & Sutinen, E. (2004). Ethical argumentation with Ethicsar. In Proceedings of the IADIS International Conference on Web Based Communities (WBC2004), Lisbon, Portugal: IADIS, 255-261. Kerne, A. (2002). Concept-context-design: A creative model for the development of interactivity. In Proceedings of the 4th Conference on Creativity & Cognition (C&C’02), New York: ACM Press, 192-199. Kinshuk, Suhonen, J., & Sutinen, E. (2004). Mobile adaptation extension of a digital learning tool. In Proceedings of the International Conference on Computers in Education (ICCE2004), Altona & Melbourne, Australia: Common Ground Publishing and RMIT University, 1581-1590. Kommers, P. (2004). Concepts in the world of the mind. In Kommers, P. A. M. (Ed.), Conceptual Support Systems for Learning - Imagining the Unknown, Amsterdam, The Netherlands: IOS Press, 3-30. Kähkönen, E., Bonnano, P., Duveskog, M., Faggiano, E., Kampf, C., Lahtinen, S.-P., Nuutinen, M., Polsani, P., Rossano, V., & Tedre, M. (2003). Approaches and methodologies for a culturally contextualized educational technology. In Kähkönen, E. (Ed.), Proceedings of the 2nd International Conference on Educational Technology in Cultural Context, Joensuu, Finland: University of Joensuu, 5-10. Laine, T., Myller, N., & Suhonen, J. (2005). ViSCoS Mobile: Learning Computer Science on the Road. Accepted for publication. In Proceedings of the 5th Annual Baltic Sea Conference on Computer Science Education, November 17-20, Koli, Finland. Löwgren, J. (1995). Applying design methodology to software development. In Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods & Techniques, New York: ACM Press, 87-94. McKenna, P., & Laycock, B. (2004). Constructivist or instructivist: Pedagogical concepts practically applied to a computer learning environment. In Proceedings of the 9th Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE2004), New York: ACM Press, 166-170. Meisalo, V., Sutinen, E., & Torvinen, S. (2003). How to improve a virtual programming course. In Proceedings of the 32nd ASEE/IEEE Frontiers in Education Conference (FIE2002), November 6-9, Boston, MA. USA. Moallem, M. (2001). Applying constructivist and objectivist learning theories in the design of a web-based course: Implications for practice. Educational Technology & Society, 4 (3), 113-125. Moonen, J. (2002). Design methodology. In Adelsberger, A. A., Collins, B. & Pawlowski, V. (Eds.), Handbook on Information Technologies for Education and Training, Springer-Verlag, 153-180. Moreno, A., Myller, N., Sutinen, E., & Ben-Ari, M. (2004). Visualizing programs with Jeliot 3. In Proceedings of the International Working Conference on Advanced Visual Interfaces, New York: ACM Press, 373-376. Nuutinen, J., Sutinen, E., Liinamaa, K., & Vanharanta, H. (2004). Woven Stories as a tool for corporate strategy planning. In Proceedings of the IADIS International Conference on Web Based Communities (WBC2004), Lisbon, Portugal: IADIS, 438-441. Reeves, T. C., & Hedberg, J. G. (2003). Interactive Learning Systems Evaluation, Engelwood Cliffs, New Jersey: Educational Technology Publications. Soloway, E., Jackson S. L., Klein, J., Quintana, C., Reed, J., Spitulnik, J., Stratford, S. J., Struder, S., Eng, J., & Scala, N. (1996). Learning theory in practice: Case studies of learner-centered design. In Proceedings of the Computer Human-Interaction Conference (CHI'96), New York: ACM Press, 189-196.

54

Suhonen, J. (2005). A Formative Development Method for Digital Learning Environments in Sparse Learning Communities, Doctoral Dissertation, Department of Computer Science, University of Joensuu, Finland, Retrieved April 6, 2006 from http://joypub.joensuu.fi/publications/dissertations/suhonen_learning/suhonen.pdf. Suhonen, J., & Sutinen, E. (2004). Formative development model in action - case of a digital learning tool. In Proceedings of the IEEE International Conference on Advanced Learning Technologies (ICALT 2004), Los Alamitos, CA: IEEE Computer Society Press, 720-722. Sutinen, E., & Torvinen, S. (2001). The Candle scheme for creating an on-line computer science program – experiences and vision. Informatics in Education, 2, 93-102. Torvinen, S. (2004). Aspects of the Evaluation and Improvement Process in an Online Programming Course – Case: the ViSCoS Program, Licentiate Thesis. Department of Computer Science, University of Joensuu, Finland, Retrieved April 6, 2006 from ftp://cs.joensuu.fi/pub/PhLic/2004_PhLic_Torvinen_Sirpa.pdf. ViSCoS (2005). Virtual Studies http://cs.joensuu.fi/viscos/?lang=eng.

of

Computer

Science,

Retrieved

April

6,

2006,

from

Whitten, J. L., Bentley, L. D., & Dittman, K. C. (2004). Systems Analysis and Design Methods (6th Ed.), McGraw-Hill.

55

Sierra, J. L., Fernández-Valmayor, A., Guinea, M., & Hernanz, H. (2006). From Research Resources to Learning Objects: Process Model and Virtualization Experiences. Educational Technology & Society, 9 (3), 56-68.

From Research Resources to Learning Objects: Process Model and Virtualization Experiences José Luis Sierra Dpto. Sistemas Informáticos y Programación. Fac. Informática. Universidad Complutense, Madrid, Spain [email protected]

Alfredo Fernández-Valmayor Dpto. Sistemas Informáticos y Programación. Fac. Informática. Universidad Complutense, Madrid, Spain [email protected]

Mercedes Guinea Dpto. Historia de América II. Universidad Complutense de Madrid. 28040, Madrid, Spain [email protected]

Héctor Hernanz Telefónica I+D S.A. C/ Emilio Vargas 6. 28043, Madrid, Spain [email protected] ABSTRACT Typically, most research and academic institutions own and archive a great amount of objects and research related resources that have been produced, used and maintained over long periods of time by different types of domain experts (e.g. lecturers and researchers). Although the potential educational value of these resources is very high, this potential may largely be underused due to severe accessibility and manipulability constraints. The virtualization of these resources, i.e. their representation as reusable digital learning objects that can be integrated in an e-learning environment, would allow the full exploitation of all their educational potential. In this paper we describe the process model that we have followed during the virtualization of the objects and research resources owned by two academic museums at the Complutense University of Madrid (Spain). In the context of this model we also summarize the main aspects of these experiences in virtualization.

Keywords Repositories of learning objects, Authoring of domain–specific learning objects, Virtual museums, Virtual campus

Introduction A research center usually owns and archives a great amount of objects and research related resources whose pedagogical value is unquestionable. Unfortunately, in many cases the scarcity and the value of this material also hinder its use for educational purposes. Two paradigmatic examples are the museums and the archives owned and maintained by many academic and research institutions. The transformation of all these materials into reusable digital learning objects (LOs) (Koper, 2003; Polsani, 2003) that can be integrated and used in an e-learning environment is, in our opinion, a key step to attaining the full exploitation of their educational value. We have confirmed this fact during our experiences with the virtualization of two academic museums at the Complutense University of Madrid (Spain): the Antonio Ballesteros museum, an academic museum of archaeology and ethnology maintained by the Department of American History II, and the José García Santesmases Computing museum, an academic museum maintained at the School of Computer Science. In this paper we present and illustrate the process model used in the two aforementioned virtualization experiences. Our virtualization process model establishes a set of guidelines for the construction of repositories of digital LOs from pre-existing research resources in specialized domains like the two mentioned above. This model, which is based on our previous experiences with a document-oriented approach to the development of content-intensive (e.g. educational, hypermedia and knowledge-based) applications, makes it easy for the virtualization of these resources to be carried out by the same experts that use, and in many cases have produced, them. In addition, this virtualization task should suppose a minimum overload in the habitual work of these experts. For this reason, the approach involves a community of developers supporting experts in the task of virtualization. Experts and developers collaborate in the definition of an adequate LO model specific to the domain. This model lets developers build a domain-specific application for the authoring and deployment of ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

56

these LOs. Since this application is especially adapted to the bodies of expertise and to the skills of the experts, the task of virtualization can be dramatically facilitated. The rest of this paper is organized as follows. We begin by describing the virtualization process model: we present the different products and activities involved in the process, we show the sequencing of these activities, and we also outline the main participants and their responsibilities. Next we describe our virtualization experiences in the domain of the two academic museums aforementioned: we illustrate how the different activities described in the process model take form in each scenario, and we concentrate on the results of the virtualization. We finish the paper with some conclusions and lines of future work. A previous version of the work described in this paper can be found in (Sierra et al. 2005b).

The Virtualization Process Model The collaboration between specialists in different knowledge areas, in order to be effective, must be adequately ruled. In our opinion, the rules used must emerge from the working experience in each specific domain instead of being adapted from aprioristic universal principles; consequently they must be refined and developed to accommodate the ongoing needs of experts and developers in each domain. The production and maintenance of reusable learning material from pre-existing research resources in specialized domains (e.g. the academic museums) is not an exception. The virtualization process model presented in this section is in accordance with these pragmatic considerations, since it has emerged from our practical experiences at the Complutense University. During these experiences we have recognized the typical scenarios contemplated by our previously mentioned document-oriented approach to the development of content-intensive applications (Sierra et al. 2004; Sierra et al. 2005a), and thereby we have adopted a strategy in structuring our virtualization process model similar to the strategy promoted in the document-oriented approach. Hence we introduce three different views, as established in Figure 1: ¾ In the products and activities view we include the activities in the approach along with the products produced and consumed by these activities. ¾ In the sequencing view we show how the activities considered are sequenced. Note that in this context the term sequencing does not mean the sequencing of the learning activities as it is usually understood in the e-learning domain, but the sequencing of the activities followed by the experts and the computer science technicians when they collaborate in the production and maintenance of the learning materials. Therefore, this view has nothing to do with any e-learning specification. It only reflects a pre-established characteristic of a process model. ¾ In the participants and responsibilities view we outline the participants in the activities along with their responsibilities in these activities. In this section we examine the process model from these three different perspectives. Products and activities The products and activities contemplated in the virtualization process model are displayed in Figure 1a. As shown in this Figure, the model introduces three different activities (Domain Analysis, Operationalization and Virtualization) and it produces three different kinds of products (a domain-specific LO model, a domain-specific authoring and deployment tool and the LO repository itself). Also notice how the model supposes the existence of a huge body of pre-existing research resources. Next we describe these aspects. The goal of the Domain Analysis activity is the formulation of an LO model that makes the educational features of the research resources created and manipulated by the experts explicit. This model must be able to integrate all the specific features needed by the experts in a knowledge area to describe and to manipulate the objects and the research goals in that area. Therefore the model for a domain must closely mirror the nature of the actual resources in the domain (e.g. the LO model in the domain of archeology can include resources and features different from the LO model used in natural history, because experts in each domain are interested in different characteristics of the object and have different research objectives). The capability of the model to include all the domain-specific characteristics of the object is critical in order to increase its acceptance and usability by the domain experts. In addition, the model should be conceptually independent of existing LO technologies. While these technologies are very valuable from a developer’s point of view, they must not condition domain experts unnecessarily during the characterization process of the LOs in their domains of expertise. On the contrary, this 57

activity could be better based on techniques used in software engineering (Arango, 1989) and knowledge engineering (Studer et al., 1999) for the construction of domain models.

(a) Products and activities

(b) Sequencing

Research resources Evolutionary iteration Domain Analysis

Domain Analysis

Virtualization [Changes or extensions in the LO model required]

Corrective iteration

Operationalization

LO Repository LO Model Operationalization

LO Authoring and Deployment Tool

Virtualization

[Virtualization completed]

(c) Participants and responsibilities Describe the domain Domain Analysis Formulate the LO model

Domain experts

Virtualize

Assess Operationalization

Build the tool

[Changes or extensions in tool required]

Virtualization

Detect needs for changes/extensions

Developers

Figure 1. The three views of the virtualization process model During the Operationalization activity a suitable authoring and deployment tool is constructed. This activity is driven by the LO model formulated during the Domain Analysis activity. Therefore, the resulting tool will also be domain-specific. This activity can take advantage of existing e-learning technologies. Hence, recommendations and standards like those proposed by IMS Content Packaging (IMS CP, 2004), IMS Learning Design (IMS LD, 2003; Koper & Tatersall, 2005) and ADL Shareable Content Object Reference Model (SCORM) (ADL, 2003) can be adopted in order to promote the interoperability of the resulting tool and the shareability and reusability of the LOs produced. These technologies must be considered implementation mechanisms providing additional functionalities to the tool and as such their potential complexity must be hidden from the domain experts. Thus while IMS CP can be easily adopted to add import and export facilities for LOs to the tool under development, IMS LD and SCORM are notably more complex in nature. Therefore the authoring/deployment tool, being domain-specific (i.e. being oriented to let experts author and deploy LOs in a specific and well-established domain), will only provide navigation facilities through the contents of the learning material organized accordingly to these general-purpose information models. In addition, it can provide an application programming interface to connect with standard editors and players for these recommendations. For this purpose, suitable mappings between the generic and the domain-specific model supported by the domainspecific tool must be defined. These mappings can be actually incorporated to the tool using pluggable facilities of the application programming interface. When mapping a general representation of a LO to a domain-specific one, some features can be loosen, although the information items that enables these features can be preserved in the domain-specific representation as hidden resources in order to make the revival of the original material possible when exported. On the other hand, domain-specific LOs can be represented in general-purpose formats in order to allow its use in external general-purpose authoring and playing environments - e.g. visualization systems for SCORM 2004 as those described in (Roberts et al. 2004). 58

Finally, during the Virtualization activity, the repository of LOs is populated with the virtualizations of the research objects and resources. This virtualization is carried out using the tool produced during the Operationalization activity. Nevertheless if import and export standard facilities for LOs are added to the tool, LOs can be exported and edited with external tools and after re-imported to the repository. Sequencing of the activities The diagram in Figure 1b shows the sequencing of the activities in the virtualization process model. Instead of proceeding sequentially, performing an exhaustive domain analysis, followed by exhaustive operationalization and virtualization, the three activities are interleaved in time. According to this iterative - incremental conception, the LO model and the associated LO authoring and deployment tool are refined whenever new domain knowledge is acquired during virtualization. Our process model introduces two types of iterations in the construction of the repositories, which are highlighted in Figure 1b. On one hand there are corrective iterations, which are related to the process of updating and fine-tuning the LO authoring and deployment tool to accommodate it to the needs of the experts (e.g. by introducing an enhanced interaction style in its user interface). On the other hand, the model also contemplates evolutionary iterations, which are related to the evolution of the LO model to capture new research or educational features of the virtualized resources (e.g. by considering new kinds of attributes for the LOs). Both types of iterations can be started during the Virtualization activity in response to the specific needs manifested by the domain experts. During our experiences with the approach we have realized that continuous maintenance and evolution of the LO model and their associated tools are mandatory to better accommodate them to the desires and changing expressive needs of the experts. This obligation supposes a heavy interaction between the experts and the developers of the applications, which can decrease overall productivity. To manage this interaction we are studying the application of the specific techniques proposed by our document-oriented approach for dealing with the intrinsic evolutionary nature of content-intensive applications (Sierra et al. 2005c). Participants and their responsibilities in the activities Domain experts and developers are the two main participants involved on the construction of LO repositories, as mentioned before. The different responsibilities that they have in the model’s three activities are depicted in Figure 1c. Next we detail these responsibilities. During the Domain Analysis activity, the main role of developers is to formulate the LO model for the objects and resources managed by the domain experts. In turn, domain experts must describe these resources to the developers, how they are used and how they are interrelated, letting developers perform an adequate conceptualization. The acceptability and usability of the resulting LO model will strongly depend on the participation of domain experts, because they know the actual resources, they can describe these resources to the developers, and they can help them in the elicitation of the possible educational uses of this material. During the Operationalization activity, the main responsibility is for the developers. They must construct the LO authoring and deployment tool. During its construction, they are driven by the LO model and they can also be assessed by the domain experts regarding different aspects not contemplated in the model (e.g. presentation and edition styles). Finally, during the virtualization activity, domain experts use the LO authoring and deployment tool to populate the LO repository. In this activity developers can react to the needs manifested by the experts and can start new corrective and/or evolutionary iterations when required.

Experiences in the Domain of the Academic Museums We have successfully applied the principles of virtualization explained in the previous section during the virtualization of two different museums at Complutense University of Madrid, as mentioned in the introduction: the museum of archaeological and ethnographical material maintained at the Department of American History II (the Antonio Ballesteros museum) and the museum of computing at the School of Computer Science (the José 59

García Santesmases Computing museum). Using these experiences we illustrate the analysis performed in these domains and we introduce the LO model formulated. Next we briefly outline the operationalization and the architecture of the authoring and deployment tools produced. Finally we detail our working experiences during virtualization. Domain analysis: virtual objects Academic museums contain collections of real objects that can be directly chosen as the most suitable candidates for conversion into LOs. In these scenarios it is natural to distinguish between such real objects and their virtual representations. These virtual representations will be called virtual objects (VOs) because they come from the virtualization of real objects initially with educational purposes (Figure 2a). The VO model was formerly proposed in (Fernández-Valmayor et al. 2003) in relation with the archaeology and ethnology museum but it has also been used in the computing museum, although with different specific features (Navarro et al. 2005). (b)

(a) Virtual object Real object Metadata

VO resource

Data Virtualization

…….

…

…

…

Foreign resource

Resources

Figure 2. (a) Real and virtual objects; (b) Virtual objects can be related using foreign and reference resources. In Figure 2a the structure of a VO is outlined. That way a VO is characterized by a set of data, a set of metadata, and a set of resources: ¾ The data in a VO represent all the features of the real object that are considered useful for its scientific study. Examples of this kind of data are the dimensions and the result of the chemical analysis of a piece of pottery of the archaeology museum, or the model and the components of a computer of the computing museum. ¾ The metadata are the information used to describe and classify the VO from a pedagogical point of view. Examples of metadata are the name of the VO’s author, its version number or its classification. The different features covered by metadata are chosen from existing exhaustive metadata schemas like the IEEE LTSC Learning Object Metadata (LOM) (LOM, 2002). ¾ The resources are all the other informational items associated with the VO. These are further classified in own, foreign and VO resources. The own resources of a VO are the multimedia archives resulting from the virtualization of the real objects (e.g. a set of photographs of a pottery vessel or a video showing the operation of a computer). The foreign resources are references to resources belonging to another VO but related to the first one (e.g. documents describing different aspects of the culture that manufactured the pottery vessel, or those related to the research and design process of a computer model). Finally, VO resources are references to other VOs in the repository that are related in some way to the first one (e.g. a VO for another piece discovered at the same excavation as the vessel, or a set of VOs for the different components of the computer). Foreign and VO resources allow the establishment of basic relationships between different VOs (Figure 2b). Indeed this mechanism can be used to build new VOs based on existing ones. These resulting composite VOs may, or may not, correspond to real objects in the museum. In this latter case they usually represent new constructed knowledge and/or new educational facilitators that arise during the virtualization process (e.g. thematic guided tours based on the objects owned by the museum about a cultural or technical subject). Although the real objects of both museums are very different in nature, the VO model has proved to be flexible enough to deal with the domain-specific data in each knowledge area. Indeed: 60

¾ ¾ ¾

Although the set of data needed to describe a computer prototype is very different from the set of data needed to describe a cultural artifact, in both cases researchers can choose the set of attribute-value pairs needed to describe the set of features in which they are interested. In the same way, resources associated with the VO can gather all the digital archives that result from the virtualization process of a computer prototype or from the virtualization process of an archaeological object. Moreover, through foreign and VO resources, the relation of a computer prototype with its components and with the research work, documents and tests that preceded its construction can be expressed in a similar manner to the relation between a pottery vessel and the other artifacts that were dug at the same archaeological site. In addition, when dealing with cultural artifacts, foreign resources will mainly be research documents describing different aspects of the culture that produced them or, alternatively, the VO representing all the field work involved in a specific archaeological project. Similar relationships also exist between computers and digital devices and their manufacturing processes.

It is important to point out that although VOs and their associated resources can resemble SCOs and assets in SCORM still there are important differences between them. In SCORM, SCOs cannot refer to assets in other SCOs neither other SCOs themselves, while the possibility of these relations is a main feature for VOs. Conceptually, a VO gathers all the information relevant to a physical or conceptual object. Its associated resources can be as simple as SCOs assets, but also can be complex structures describing a learning process and the workflow of the learning activities in which this VO can be implicated. Operationalization: web-based authoring and deployment tools for virtual objects The VO model has led us to develop simple web-based authoring and deployment tools for the two aforementioned museums (Figure 3). Basically these tools enable authors to create VOs, upload their resources and to establish their relations with other VOs and/or their resources. Users can navigate the repository of VOs and their associated resources, thus the complexity of the navigation depends of the navigational structure of the resource. The tool for the museum of archaeology (Figure 3a) is called Chasqui. The word chasqui means messenger in Quechua, the language spoken in the Inca Empire. Chasqui has gone through a strong evolution, as described in (Navarro et al. 2005). In the initial version of Chasqui (Chasqui Web Site, 2005), VOs were directly mapped onto its database representations. While the resulting application enabled users to author learning objects using domain-specific authoring tools, we found serious difficulties regarding portability and interoperability with other systems and authoring tools. For this purpose we have adopted a more sophisticated architecture, based on well-known interoperability standards. The resulting version can be visited in its testing site (Chasqui2 Web Site, 2005), and it is scheduled to be in production in the first quarter of 2006. The architecture of this new version of Chasqui has also been reused in the development of the tool for the computing museum (Figure 3b). This tool is called MIGS (MIGS Web Site, 2005), the acronym for the name of the museum in Spanish. (a)

(b)

Figure 3. Snapshots for (a) Chasqui; (b) MIGS The final architecture proposed during Operationalization is depicted in Figure 4. According to this architecture, the repository of VOs continues to be supported by a relational database. The tools include pre-established web interfaces tailored to the museums being virtualized. In addition, these tools also include programmatic interfaces accessible via web services (Cerami, 2002). Web services facilitate the interoperation with other 61

repositories, enable different accessing mechanisms (e.g. mobile devices) and permit the use of external tools with alternative and more powerful interaction styles (e.g. IMS LD and SCORM editors and players). As suggested in Figure 4, this architecture is entirely implemented using open source technologies. Repository (MySQL)

Web Interface (PHP)

Web Service (AXIS)

Web Client Other repositories

Mobile Devices

Alternative authoring / deployment tools

Figure 4. Architecture of the Web Based Authoring and Deployment Tools To enable interoperability, the tools incorporate import and export facilities of VOs in accordance with the IMS CP specification. This way, VOs can be packaged according to this specification and can be exported to other IMS-aware Learning Management Systems able to conduct a more elaborated learning process. Direct importation is also possible between repositories associated with museums sharing a common VO model. More complex importation processes can also be automatically achieved by connecting the appropriate adapters to the web service interface. Indeed, this interface is the point where mappings between general-purpose representation formats (e.g. SCORM or IMS LD compliant learning materials) and the VO model can be readily incorporated. By incorporating to it the appropriate importation and exportation mappings, many current and future learning application profiles could be readily supported, maybe with slight evolutions of the current VO model. Virtualization The tools described in the previous section are being used in the virtualization of the two aforementioned museums. While the virtualization of the computing museum is in its initial stages (115 VOs included on December 2005), the virtualization of the archaeology museum is in an advanced state. Indeed, Chasqui has been in production since the year 2003 and has therefore had enough time to prove the usefulness of the iterative – incremental conception of the aforementioned virtualization process. Currently, the resulting virtual museum contains more than 1500 VOs and the virtualization process continues to be active. Beyond its capabilities for creating simple collections of VOs, the Chasqui tool, being in a more advanced virtualization state than MIGS, has proven very valuable in several research and pedagogically related activities oriented to students with very different backgrounds. In particular, as it will be detailed in the next sections, Chasqui has evolved from its original purpose to be a very valuable tool for supporting the active learning of the research process by junior students and of its refinement by senior and PhD. students. In addition, Chasqui has also proven very valuable as a basic tool for the just-in-time publishing of research resources. Next we summarize some of these experiences. Virtualization and dissemination of pre-existing materials The primary use of Chasqui and MIGS was to let students and the general public learn about the objects found or collected during an archaeological or ethnological field season, or to learn about the computing devices built at the university by pioneering professors. For this purpose, in the initial virtualization iterations the repositories of both museums were populated with such elementary VOs. The publishing of this kind of stored and archived resources makes tools like Chasqui and MIGS very valuable artefacts for disseminating pioneering work and the patrimony of the academic museums among the general public. This way, by exhibiting many pieces of hardware, like those shown in Figure 5a (MIGS VO number 2), 62

as VOs, the computing museum has gained considerable popularity among students. The museum itself is discovered in Internet, and instructors and students can use MIGS for documentation prior to visiting the real museum at the Computer Science School. As another recent example, 23 pieces in the archives of the museum of archaeology were located using Chasqui and selected for the exposition Pueblos Amazónicos: Un Viaje a otras Estéticas y Cosmovisiones (Amazon Cultures: A Journey to other Aesthetics and Cosmovisions) organized for the general public by the Museum of Science of Castilla-La Mancha (Cuenca, Spain, 1-30 Junio 2005). (a)

(b)

Figure 5. (a) a VO in MIGS; (b) Unpublished pottery vessel neck from Proyecto Esmeraldas that was rerecovered during the virtualization process Another interesting experience, this time related to Chasqui, has been its use for reviving unpublished archaeological material for research and pedagogical purposes (Figure 5b). This is indeed the case with several Archaeological and Ethnological projects already finished at the Department of American History II: Chinchero, Incapirca and Esmeraldas. Work Assignments for Undergraduate Students In Chasqui the research materials collected at different archaeological sites have been used to design work assignments for undergraduate students. The flexibility of the VO model lets teachers conceive these assignments as a new composite VOs. In Figure 6a we show some snapshots corresponding to one of these assignments for an undergraduate introductory course on the Andean archaeology area (Chasqui VO number 1183). As a deployment tool, Chasqui can be combined with typical Learning Management Systems (LMS) as has been the case with these work assignments that have been made accessible to students using the virtual campus of the Complutense University, currently supported by the WebCT platform (Rehberg et al. 2004). This lets us take advantage of the communication and learning management features of a typical LMS (e.g. discussion forums and management student facilities) in conjunction with the other features of Chasqui. Involving Advanced Undergraduate and Graduate Students in the Virtualization Process We have also used Chasqui to involve advanced undergraduate and graduate students in the construction of new composite VOs, therefore promoting active learning among these students. Again the flexibility of the VO model enables students to construct new VOs by referencing pre-existing foreign resources and also pre-existing VOs. In addition, students can also collaborate in the elaboration of new original resources. The snapshots shown in Figure 6b correspond to a VO produced by a group of graduate students enrolled in a course about Aztec Culture (Chasqui VO number 1683). In addition to the pedagogical benefits of their active involvement, the work performed by the students is made available to other users and it can be used as support material in future editions of the course. As with the undergraduate-level experiences, we have realized the advantages of integrating the domain-specific virtualization activity with the generic facilities provided by a 63

general-purpose LMS, like WebCT. In this case the groups of students can share a server file to interchange the digital materials needed for the virtualization, as well as a private newsgroup and a personal e-mail for communication purposes.

(b)

(a)

Figure 6. Snapshots of (a) a work assignment for undergraduate students, (b) a VO produced by graduate students

(a)

(b)

Sonidos de América: un recorrido por los instrumentos musicales de la Colecciones de Arqueología y Etnografía Partiendo de América Central Bajando por Sudamérica ... En la Colecciones de Arqueología y Etnografía de América del Departamento de Historia de América II, Universidad Complutense de Madrid, se puede consultar una limitada pero interesante cantidad de instrumentos musicales procedentes de distintas culturas ... ...

(c)

Figure 7. (a) An archeological marked document; (b) presentation generated from (a); (c) VO resource with the presentation in (b). 64

Involving PhD. Students in the Virtualization Process We have used Chasqui to implement a new pedagogical strategy in a PhD. course on New Information Technologies in Andean Archaeology. Students enrolled in this course are integrated as active members in our research group. This is an interdisciplinary group formed by both archaeologists and computer scientists. One of the main research interests of the Computer Science branch of the group is the formulation of domain-specific descriptive markup languages (Goldfarb, 1981; Coombs et al. 1987) to structure documents of different knowledge areas for multiple purposes. Our interest in the use of makup languages is directly related with the aforementioned document-oriented approach for the development of content-intensive applications. Therefore, we propose that PhD. students define their own markup languages to structure the documents that they produce with different purposes: research, dissemination or education. The languages created by the students (with the help of their teacher) are conceived as descriptive markup languages defined using XML (XML, 2004). A typical workflow of the process is as follows: ¾ The teacher assigns research papers about different subjects to each student. ¾ Then the students must synthesise the main ideas in the papers they have studied and, in a following session, discuss their syntheses with the rest of the group. ¾ When the students have enough knowledge about the subject, they must create composite VOs, gathering and structuring as much information as possible regarding this subject. Among these resources, they must include documents describing the results of their research. These documents, which are the most common in the domain, can be classified by: (i) their final purpose (site reports, essay, review, research or dissemination papers), (ii) their primary sources (artefacts, monuments, site plans or collections), or (iii) by the targeted audience (public or academic). ¾ In the next stage, each of the documents elaborated is analyzed to identify its structural elements, its hierarchical organization and the labels and the attributes to be used to make all the information that they convey explicit. ¾ The documents are marked up and a first attempt to abstract the type of these documents as XML document grammars is carried out with the help of Chasqui’s developer community. Note that these document grammars define new domain-specific descriptive markup languages used for authoring purposes. Therefore they do not should be confused with extensions to the XML bindings for the information models proposed by the different e-learning specifications. These document grammars can be also used and refined by subsequent groups of students. Therefore, these document grammars are subjected to a continuous evolution. In order to manage this evolution, developers can use suitable XML schema technologies (Lee & Chu, 2000) as well as the techniques for the incremental definition of domain-specific descriptive markup languages described in (Sierra et al. 2005c). ¾ Finally, the documental resources associated with the composite VOs already created are automatically generated from the marked documents without need, for the students, of further manual processing. For this purpose, they also get support from the developer community. This community produces suitable processors for the domain-specific descriptive markup languages defined by the students. For this purpose they can use standard XML processing technologies (Birbeck et al. 2001) or the more advanced solutions oriented to facilitate the incremental construction of such processors described in (Sierra et al. 2005c). In Figure 7a we show part of a document marked with a domain-specific markup language developed by a group of PhD. students during the course’s edition of 2004-2005. Note again that this document is used only during authoring. Indeed, the resource finally integrated in the VO is the result of transforming it to HTML - in this particular case an XSLT style sheet (XSLT, 2004) was used to carry out the transformation. Therefore, the markup used here is domain-specific and has nothing to do with the tags used in the XML bindings for the different information models proposed by the e-learning community (e.g. markup for representing IMS CP manifests). In Figure 7b we show the resource (an HTML page) generated from this document. In Figure 7c we show some snapshots of the Chasqui VO where the resource is finally integrated (Chasqui VO 1430). Just-In-Time Diffusion of the Research Results Domain-specific authoring tools like Chasqui allow, for researchers, the continuous and just-in-time update of the VOs. The only requirement is an internet connection, since the usability of the tool makes the presence of developers unnecessary. That way, research results and field work reports can be published as they are obtained, letting the interested researchers and students access these results without waiting for their diffusion over more conventional publishing channels (e.g. specialized conferences and journals).

65

In Figure 8 we show some snapshots of a VO (Chasqui VO number 1743) related to the field work and the preliminary reports for the Project Manabí Central, which is being carried out by an international research consortium at Chirije and San Jacinto de Japoto, on the Ecuadorian Coast (Bouchard, 2004). Another related use of Chasqui is the organization as VOs of the content and development of other research activities, like symposiums (see Chasqui VO number 1431 in the Chasqui web site).

Figure 8. VO with the preliminary reports of the Project Manabí Central

Conclusions and Future Work The virtualization process model described in this paper lets domain experts create repositories of LOs in a very dynamic way. For this purpose domain-specific LO models are formulated and supporting authoring and deployment tools based on these models are developed. These tools are used by the experts to produce the repositories. We have realized that the approach is very valuable for exploiting the educational potential of otherwise underused and/or access-limited research materials. We have also realized that the approach allows the creation of new knowledge as reusable composite LOs for many different pedagogical and research purposes. But perhaps the most relevant and in some way unexpected result of our work, from an e-learning perspective, has been how teachers and students are using the system. This way, they are finding that Chasqui is a very useful tool for supporting active and collaborative learning, involving both student-student and teacher-student relations as has been shown in the virtualization experiences described in this paper. Currently we are finishing the first stage in the virtualization of the computing museum, and we are also starting several virtualization experiences regarding the virtual campus of the Complutense University of Madrid. We are also working on another evolution of the VO concept, by extending VOs with scripts documenting the sequencing of their resources. These scripts will be implemented by using the IMS Learning Design Specification (IMS LD, 2003; Koper & Tatersall, 2005). As future work we are planning to undertake the virtualization of other museums at this university in order to refine the concept of VO. We are also planning to further use our document-oriented approach (Sierra et al. 2004; Sierra et al. 2005a) to improve the maintenance of LO domain-specific authoring tools by enabling their full documental description.

Acknowledgements This work is supported by the Spanish Committees of Science and Technology (TIC2002-04067-C03-02, TIN2004-08367-C02-02 and TIN2005-08788-C04-01) and of Industry, Tourism and Commerce (PROFIT, FIT-350100-2004-217). We also want thank Antonio Navarro for his participation in early versions of this work.

66

References ADL (2003). Shareable Content Object Reference Model SCORM, Retrieved April 7, 2006, from, http://www.adlnet.org/scorm. Arango, G. (1989). Domain-Analysis: From Art Form to Engineering Discipline. ACM SIGSOFT Notes, 14 (3), 152-159. Birbeck, M., Kay, M., Livingstone, S., Mohr, S. F., Pinnock, J., Loesgen, B., Livingston, S., Martin, D., Ozu, N., Seabourne, M., & Baliles, D. (2001). Professional XML, (2nd Ed.). Birminghan,UK: WROX Press. Bouchard, J. F. (2004). Project Manabí Central. Rapport d'Activité Année 2004, Paris, France: Division des Sciences Sociales de l'Archéologie del Ministére des Affaires Etrangéres. Cerami, E. (2002). Web Services Essentials, Sebastopol, CA, USA: O’Reilly. Chasqui Web Site (2005). Chasqui http://macgalatea.sip.ucm.es/chasqui.html.

website,

Retrieved

April

7,

2006,

from,

Chasqui2 Web Site (2005). Chasqui website, http://macgalatea.sip.ucm.es/web3/principal/principal.html.

Retrieved,

April

7,

2006,

from,

Coombs, J., Renear, A., & DeRose S. (1987). Markup Systems and the Future of Scholarly Text Processing. Communications of the ACM, 30 (11), 933-947. Fernández-Valmayor, A., Guinea, M., Jiménez, M., Navarro, A., & Sarasa, A. (2003). Virtual Objects: An Approach to Building Learning Objects in Archeology. In Llamas-Nistal, M., Fernández-Iglesias, J., & AnidoRifon, L. E (Eds.), Computers and Education: Towards a Lifelong Society, Dordrecht, The Netherlands: Kluwer Academic Publishers, 191-202. Goldfarb, C. F. (1981). A Generalized Approach to Document Markup. ACM SIGPLAN Notices, 16 (6), 68-73. LOM, (2002). IEEE Standard for Learning Object Metadata, IEEE Standard 1484.12.1-2002. IMS LD (2003). IMS Learning http://www.imsglobal.org/learningdesign.

Design

1.0,

Retrieved

April

7,

2006,

from,

IMS CP (2004). IMS Content Packaging Information Model Version 1.1.4 Final Specification, Retrieved April 7, 2006, from, http://www.imsglobal.org/content/packaging. Koper, R. (2003). Combining re-usable learning resources and services to pedagogical purposeful units of learning. In Littlejohn, A. (Ed.), Reusing Online Resources: A Sustainable Approach to eLearning, London, UK: Kogan Page, 46-59. Koper, R., & Tatersall, C. (2005). Learning Design, a handbook on modelling and delivering networked education and training, Heidelberg: Springer-Verlag. Lee, D., & Chu, W. W. (2000) Comparative Analysis of Six XML Schema Languages. ACM SIGMOD Record, 29 (3), 76-87. MIGS (2005). MIGS website, Retrieved http://macgalatea.sip.ucm.es/migs/publico/control.php.

April

7,

2006,

from,

Navarro, A., Sierra, J. L., Fernández-Valmayor, A., & Hernanz, H. (2005). From Chasqui to Chasqui 2: An Evolution in the Conceptualization of Virtual Objects. Journal of Universal Computer Science, 11 (9), 15181529. Polsani, P. R. (2003). Use and Abuse of Reusable Learning Objects. Journal of Digital Information, 3 (4), retrieved June 30, 2006, from, http://jodi.tamu.edu/Articles/v03/i04/Polsani/. 67

Rehberg, S., Ferguson, D., McQuillan, J., Eneman, S., & Stanton, L. (2004). The Ultimate WebCT Handbook, a Practical and Pedagogical Guide to WebCT 4.x. Clarkston, GA, USA: Ultimate Handbooks. Roberts, E., Freeman, M. W., Wiley, D., & Sampson, D (Eds.). (2005). Special Issue on SCORM 2004 Sequencing & Navigation. Learning Technology Newletter, 7 (1), retrieved June 30, 2006, from, http://lttf.ieee.org/learn_tech/issues/january2005/index.html. Sierra, J. L., Fernández-Valmayor, A., Fernández-Manjón, B. & Navarro, A. (2004). ADDS: A DocumentOriented Approach for Application Development. Journal of Universal Computer Science, 10 (9), 1302-1324. Sierra, J. L., Fernández-Manjón, B., Fernández-Valmayor, A. & Navarro, A. (2005a). Document-Oriented Construction of Content-Intensive Applications. International Journal of Software Engineering and Knowledge Engineering, 15 (6), 975-993. Sierra, J. L., Fernández-Valmayor, A., Guinea, M., Hernanz, H. & Navarro, A. (2005b). Building Repositories of Learning Objects in Specialized Domains: The Chasqui Approach. In Goodyear, P., Sampson, D. G., Yang, D.J.T., Kinshuk, Okamoto, T., Hartley, R., & Chen, N.-S. (Eds.), In Proceedings of the 5th IEEE International Conference on Advanced Learning Technologies ICALT’05, Los Alamitos, CA, USA: IEEE Society Press, 225229. Sierra, J. L., Navarro, A., Fernández-Manjón, B., & Fernández-Valmayor, A. (2005c). Incremental Definition and Operationalization of Domain-Specific Markup Languages in ADDS. ACM SIGPLAN Notices, 40 (12), 2837. Studer, R., Fensel, D., Decker, S., & Benjamins, V. R. (1999). Knowledge Engineering: Survey and Future Directions. Lecture Notes in Artificial Intelligence, 1570, 1-23. XSLT (2004). XSL Transformations http://www.w3.org/TR/xslt.

(XSLT)

Version

1.0,

Retrieved

April

7,

2006,

from,

XML, (2004). Extensible Markup Language (XML) (3rd Ed.), Retrieved April 7, 2006, from, http://www.w3.org/TR/REC-xml/.

68

Chen, C.-M., Hong, C.-M., Chen, S.-Y. & Liu, C.-Y. (2006). Mining Formative Evaluation Rules Using Web-based Learning Portfolios for Web-based Learning Systems. Educational Technology & Society, 9 (3), 69-87.

Mining Formative Evaluation Rules Using Web-based Learning Portfolios for Web-based Learning Systems Chih-Ming Chen Institute of Learning Tech, National Hualien Univ of Education, Hualien 970, Taiwan, R.O.C. [email protected]

Chin-Ming Hong Institute of Applied Electronics Technology, National Taiwan Normal University, Taipei 106, Taiwan, R.O.C. [email protected]

Shyuan-Yi Chen Institute of Industrial Education, National Taiwan Normal University, Taipei 106, Taiwan, R.O.C. [email protected]

Chao-Yu Liu Institute of Learning Technology, National Hualien University of Education, Hualien 970, Taiwan, R.O.C. [email protected] ABSTRACT Learning performance assessment aims to evaluate what knowledge learners have acquired from teaching activities. Objective technical measures of learning performance are difficult to develop, but are extremely important for both teachers and learners. Learning performance assessment using learning portfolios or web server log data is becoming an essential research issue in web-based learning, owing to the rapid growth of e-learning systems and real application in teaching scenes. The traditional summative evaluation by performing examinations or feedback forms is usually employed to evaluate the learning performance for both the traditional classroom learning and the web-based learning. However, summative evaluation only considers final learning outcomes without considering learning processes of learners. This study presents a learning performance assessment scheme by combining four computational intelligence theories, i.e., the proposed refined K-means algorithm, the neuro-fuzzy classifier, the proposed feature reduction scheme, and fuzzy inference, to identify the learning performance assessment rules using the web-based learning portfolios of an individual learner. Experimental results indicate that the evaluation results of the proposed scheme are very close to those of summative assessment results of grade levels. In other words, this scheme can help teachers to assess individual learners precisely utilizing only the learning portfolios in a web-based learning environment. Additionally, teachers can devote themselves to teaching and designing courseware since they save a lot of time in evaluating learning. This idea can be beneficially applied to immediately examine the learning progress of learners, and to perform interactively control learning for elearning systems. More significantly, teachers could understand the factors influencing learning performance in a web-based learning environment according to the obtained interpretable learning performance assessment rules.

Keywords: Learning Performance Assessment, Web-based Learning, Web-based Learning Portfolio, Data Mining

Introduction Gagnés’ research on the internal process of learning has indicated that the complete learning process should assess learning performance (Gagnés, 1997). Learning performance evaluation instruments can generally be classified as summative assessment and formative assessment (Torrance & Pryor, 1998; Campbell et al, 2000). While summative evaluation is generally performed after finishing an instruction unit or class, formative assessment emphasizes the learning process. Teachers can assess the overall learning performance using summative assessments. Conversely, the use of formative assessments helps teachers obtain feedback about how well students are learning and particular difficulties they might be having. The standard summative assessment scheme at course level is to perform either an examination or an assessment form. However, these methods cannot collect all learning process information. Since the formative assessment strongly emphasizes integrating “what people learned” with “how people learned”, the formative assessment occurs when teachers feed information back to students in ways that enable the student to learn better, or when students can engage in a self-reflective process. Therefore, while assessing learning performance using the portfolio has become increasingly popular, technology is facilitating its evolution and management (Torrance & Pryor, 1998; ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

69

Campbell et al, 2000; Lankes, 1995). Moreover, learning performance assessment based on the learning portfolio is a very useful concept for some fields of learning assessment, such as psychological testing, knowing how subjects take the test can be more important than their answers. Traditional portfolio assessment relies on man-made data collection and a writing-centered learning process. Difficulties in data storage, search and management after long-term implementation have become problematic in developing and implementing portfolio assessment (Chang, 2002). In contrast, a web-based learning portfolio can be collected, stored and managed automatically by computers when learners interact with an e-learning platform. Therefore, learning performance assessment using a web-based learning portfolio has received significant attention recently (Lankes, 1995; Rahkila & Karjalainen, 1999; Wu & Leung, 2002). Rasmussen et al. (Rasmussen et al. 1997) suggested that Internet-based instruction could also allow student progress to be evaluated through participation in group discussions and portfolio development. Lankes (Lankes, 1995) stated that implementing computer-based student assessment portfolios is an innovative educational innovation owing to its ability not only to offer an authentic demonstration of accomplishments, but also enable students to take responsibility for their completed tasks. Rahkila (Rahkila & Karjalainen, 1999) also proposed using the user profiles with log data for the learning performance assessment, finding that it can be applied to interactively control learning effectively. Rahkila emphasized that the learning portfolio assessment is supported by the cognitive–constructive theory of learning (Rahkila & Karjalainen, 1999; Bruner, 1996). Zaiane (Zaiane & Luo, 2001) proposed applying web access logs and advanced data mining techniques to extract useful patterns that can help educators evaluate on-line course activities. Wang et al. (Wang et al. 2003) found that learning behavior information, commonly referred to as a learning portfolio, can help teachers understand why a learner obtained a high or low grade. Carrieira (Carrira & Crato, 2004) proposed monitoring users’ reading behaviors to infer their interest in particular articles. However, developing a precise learning performance assessment scheme using web-based learning portfolio is a challenging task for web-based learning systems. Data mining, or knowledge discovery, attempts to obtain valuable knowledge from data stored in large repositories. Data mining has been considered as an appropriate method of knowledge discovery to excavate the implicit information (Dunham, 2002). This study presents a data mining approach that integrates four computational intelligence schemes, i.e., the refined K-means clustering algorithm, the neuro-fuzzy classifier (Nauck & Kruse, 1999), the proposed feature reduction scheme, and the fuzzy inference (Lin & Lee, 1996) , to evaluate on-line learning behavior and learning performance. The four computational intelligence schemes are employed to logically determine fuzzy membership functions for the neuro-fuzzy classifier, discover the fuzzy rules relating to the learning performance assessment, reduce the feature dimension of the discovered fuzzy rules, and infer the learning performance using the discovered fuzzy rules based on the learning portfolio of an individual learner, respectively. The proposed learning performance assessment scheme was implemented on the personalized e-learning system, and its effectiveness was demonstrated in a real-world teaching scenario. Experimental results show that the proposed learning assessment scheme can correctly measure learners’ learning performance according to their learning portfolios as well as help teachers save a lot of time for the learning evaluation during the learning process. Significantly, the learning evaluation results were applied to help teachers examine the learning progress of learners and interactively control learning for the personalized e-learning system. The remainder of this study is organized as follow. Section 2 describes the problem description based on the gathered learning portfolios for the proposed learning performance assessment scheme. Section 3 explains the proposed learning performance assessment scheme. Section 4 presents the detailed experiments. Conclusions are drawn in Section 5.

Problem Description Personalized E-learning System (PELS) with Learning Performance Assessment Mechanism Based on Learning Portfolios The personalized e-learning system (PELS) based on the Item Response Theory, which includes an off-line courseware modeling process, four intelligent agents and four databases, is presented in our previous study for adaptive courseware recommendation (Chen,, Lee & Chen, 2005), (Chen, Liu & Chang, 2006). However, the PELS mainly focuses on performing adaptive learning based on the difficulty parameters of courseware and learner ability of individual learner, the learning performance assessment is lacked feature in this system. In this paper, the functionalities of the PELS system are extended to include the learning performance assessment agent 70

in order to perform the evaluation of learning performance using the gathered learning portfolios of individual learners. The extended system architecture is shown as Figure 1.

Courseware Modeling Process

User Account Database

Learning Interface Agent

XML-based Courseware database

Test Agent

Courseware Management Agent

Teacher Account Database

Courseware Construction Module

Testing Items Database

Learner

User Profile Database

Learning Performance Assessment Agent

Courseware Recommendation Agent

Personalized Learning Module

Figure 1. The system architecture

Figure 2. The learning interface for learner 71

Figure 2 shows the entire layout of the learning interface on PELS. In the left frame, system shows the course categories, course units and the list of all courseware in the courseware database using a hierarchical tree topology structure. While a learner clicks a courseware for learning, the content of selected courseware will be exhibited in the upper-right window. Besides, the feedback interface is arranged in the bottom-right window. The proposed system can get learner’s feedback response from the interface of feedback agent through learner replies one randomly testing question related to the conveyed learning content. The answer of testing question helps system to get the learner’s comprehension percentage for the recommended courseware. System passes the feedback response to the courseware recommendation agent to infer the learner’s ability using the item response theory (Baker & Frank, 1992) detailed in our previous study (Chen,, Lee & Chen, 2005; Chen, Liu & Chang, 2006). After a learner presses the button of “submit”, this system will reveal a list of the recommended courseware based on his current ability. After the learner selects the next courseware according to the suggestion of the courseware recommendation agent for further learning, the learner can continue to learn the selected courseware. The PELS will continue to run the learning cycle until the evaluated learner ability satisfies the stop criterion. Next, the learning procedure will enter the final testing stage to perform a summative assessment through replying 25 randomly selected testing questions. The results of final testing will be used to verify the evaluation quality of the proposed learning performance assessment approach in the analysis of experimental results. Besides, the learner can also log in the discussion board on PELS to publish the learning questions of the individual self or contribute helpful ideas for the learned course unit. The PELS will automatically gather the useful learning portfolios of individual learners for the learning performance assessment during learning processes. Considered Learning Portfolio in the User Profile Database The IMS Global Learning Consortium defined that ePortfolios are collections of personal information about a learner that represent accomplishments, goals, experiences, and other personalized records that a learner can present to schools, employers, or other entities (http://support.imsglobal.org/ePortfolio/). The ePortfolio specification developed by IMS Global Learning Consortium enables the use of the Internet for personalized, competency-based learning and assessment, enhanced career planning, and improved employment opportunities. In the meanwhile, the proposed ePortfolio specification supports many kinds of portfolios, including assessment ePortfolios, presentation ePortfolios, learning ePortfolios, personal development ePortfolios, and working ePortfolios. The IMS Global Learning Consortium also summarized that ePortfolio can contain personal information about the owners, achievements, interests and values, test and examination results, and so on. Next, the learning portfolio information collected by PELS for the proposed learning performance assessment approach is presented based on the IMS ePortfolio specification. This learning portfolio information was saved into the user profile database according to the learner’s interaction with PELS. Hence, the learning performance was graded according to their PELS learning portfolios. The eight gathered learning factors are described in detail as follows: 1.

Reading rate of course materials (RR) After a learner has logged onto the PELS system, the PELS automatically counts and accumulates the reading amount of the learned course materials for each learner. The learning parameter is progressively increased for each section learning process. Thus, the reading rate of course materials is defined as the rate of studying course materials in a course unit, and the notation RR is employed to represent the learning factor. Table 1 presents an example of calculating the reading rate for both the learners A and B in the learning course unit.

A

Table 1. A calculating example for the learning factor of reading rate The The total number number of of course materials Learning path of course materials studying in the learned course course unit materials 1Ö5Ö8Ö10Ö11Ö12Ö13Ö14Ö15Ö16Ö17 11 11

1

B

1Ö5Ö4Ö6Ö7Ö8Ö9Ö10Ö(9)Ö(10)Ö11

0.82

Learner

9

11

Reading rate (RR)

( ) indicates the repeated course material during the learning process

72

2.

Total accumulated reading time of all learned course materials (RT) The total accumulated reading time of each learner is calculated by summing up the reading time of all learned course materials on the PELS system, and the notation RT is used to represent the learning factor. Table 2 presents an example of calculating the total accumulated reading time for both the learners A and B in the learning course unit.

Learner

Table 2. A calculating example for the learning factor of total accumulated reading time Total accumulated Learning path of course materials reading time(RT) 1(25)Ö5(32)Ö8(33)Ö10(30)Ö11(36)Ö12(43)Ö13(47)Ö14(43)

A

420(secs)

Ö15(45)Ö16(40)Ö17(46) 1(42)Ö5(56)Ö4(50)Ö6(62)Ö7(67)Ö8(72)Ö9(79)Ö10(93) B Ö9(70)Ö10(81)Ö11(118) ( ) indicates the reading time for the learned course material 3.

790(secs)

Learner ability evaluated by PELS (LA) After studying the recommended courseware, the PELS (Chen, Lee & Chen, 2005; Chen, Liu & Chang, 2006) can dynamically estimate a learner’s ability according to the item response theory (Baker & Frank, 1992) by collecting the replied responses of the learner to the randomly selected testing questions in the learned course unit. The value denotes the learner’s ability in the learned course unit measured by the PELS system during the learning process, and the range of learner’s ability are limited from -3 (i.e. lowest ability) to +3 (i.e. highest ability). As a learner logs in this system, if the user account database does not have any history records in the selected course unit for this learner, then his initial ability will be regarded as 0. That is, the system assumes beginner’s ability is moderate level. As a learner clicked the recommended courseware for learning, his/her ability in this course unit will be re-evaluated according to his/her feedback responses and the corresponding difficulty parameter of the learned courseware. The notation LA is used to represent the learning factor of learner ability in this paper. In the PELS, the quadrature form proposed by Bock and Mislevy (Baker & Frank, 1992) was employed to approximately estimate learner’s ability as follows: q

ˆθ =

∑θ

k

L( u1,u 2 ,..., u n | θ k ) A( θ k )

(1)

k

q

∑ L( u ,u 1

2

,..., u n | θ k ) A( θ k )

k

where θˆ denotes the learner’s ability of estimation, L(u1, u2 , L , un | θ k ) is the value of likelihood function at a level below their ability level θ k and learner’s responses are u1 , u 2 ,..., u n , θ k is the k th split value of ability in the standard normal distribution, and A(θ k ) represents the quadrature weight at a level below their ability level θ k . In Eq. (1), the likelihood function L(u1, u2 , L , un | θ k ) can be further described as follows: n

L(u1, u2 ,L, un | θ k ) =

∏ P (θ ) j

k

uj

1−u j

Q j (θ k )

(2)

j =1

P j (θ k ) =

e

D (θ k −b j )

where

1+ e

understand the j

th

D (θ k −b j )

,

Q j (θ k ) = 1 − Pj (θ k ) Pj (θ ) , denotes the probability that learners can

courseware at a level below their ability level θ k , Q j (θ k ) represents the probability 73

that learners cannot understand the j th courseware at a level below their ability level θ k , b j is the difficulty parameter of the j

th

courseware, and D is a constant 1.702, and U j is the understanding or not

understanding answer obtained from learner feedback to the j th courseware, i.e. if the answer is understanding then U j = 1 ; otherwise, U j = 0 .

4.

Correct response rate of randomly selecting testing questions (CR) After a learner has studied the recommended course material, the PELS tests the learner on his understanding by randomly selecting a relevant question from the testing item database. The rate of correct responses to test questions helps determine the learner’s degree of understanding for all learned courseware, and the notation CR is used to represent the learning factor in this paper. For example, if a learner gave seven correct responses and three incorrect responses for ten randomly selecting testing questions from the testing item database, then the correct response rate is 0.7.

5.

Posted amount of articles on the forum board (PN) The PELS system can automatically count the total number of posted articles and answered questions on the forum board for each learner during the learning process. The value usually shows a learner level of participation in group discussion, and the notation PN is used to represent the learning factor in this paper. For example, if a learner posted three articles and answered five questions on the forum board, then the posted amount of articles on the forum board is eight pieces.

6.

Accumulated score on the forum board (AS) The PELS system gives various scores for the various interactive behaviors among learners on the forum board. In this study, the PELS will automatically increment a learner’s score of one point when he/she logs onto the forum board, and accumulate a learner’s score of two points if he/she posts an article on the forum board. The accumulated score on the forum board represents the level of useful feedback given by a learner, and the notation AS is used to represent the learning factor in this paper. For example, if a learner logged onto the forum board two times and posted three articles, then the accumulated score on the forum board will be eight points.

7.

Effort level of studying course Materials (EL) Since each course material in the PELS system is assigned a required minimum reading time by course experts based on the courseware content, the effort level is defined as the actual reading time compared with the required minimum reading time for the learned courseware, and the notation EL is used to represent the learning factor in this paper. Suppose the actual reading time of learners A and B in the learned course unit is 420 seconds and 790 seconds, respectively, and listed as Table 2. Table 3 presents an example of calculating the effort level for the learners A and B in the learned course unit. In this study, the range of the effort level is limited from 0 to 1. Restated, the value of the effort level will be assigned as 1 if the calculated value is over 1.

Learner A B

Table 3. A calculating example for the learning factor of effort level Learning path of course materials 1(30)Ö5(60)Ö8(75)Ö10(60)Ö11(75)Ö12(75)Ö13(60)Ö14(60)Ö15(75) Ö16(75)Ö17(90) 1(30)Ö5(60)Ö4(45)Ö6(60)Ö7(60)Ö8(75)Ö9(60)Ö10(60)Ö9(60) Ö10(60)Ö11(75)

Effort level (EL)

420 / 735 ≈ 0.57 790/585 → 1

( ) indicates the required minimum reading time determined by course experts for the learned course material 74

8.

Final test grade (G) This study measures the final test grade through the summative assessment scheme of fixed-length testing examination after the entire learning process is completed, and the notation G is used to represent the learning factor in this paper.

Mining Learning Performance Assessment Rules Based on Learning Portfolios Flowchart of Mining Learning Performance Assessment Rules Figure 3 shows the entire flowchart of the proposed learning performance assessment scheme. The proposed refined K-means algorithm is initially employed to logically determine the fuzzy membership functions of each learning factor based on real data distribution of learning portfolios for mining learning performance assessment rules by the neuro-fuzzy classifier (Nauck & Kruse, 1999). After the fuzzy rules for learning performance assessment are identified by the neuro-fuzzy classifier, the feature reduction scheme is employed to simplify the linguistic terms of the discovered learning performance assessment rules under keeping a satisfied accuracy rate of learning performance assessment. This procedure aims to obtain simplified learning performance assessment rules for promoting the inferring efficiency in web-based learning systems as well as discover the main learning factors that affect the learning outcomes. Finally, the fuzzy inference is employed to grade the learning performance by the simplified fuzzy rules for individual learners. The following sections give details for the proposed learning performance assessment scheme for individual learners.

STA R

T

Refined K-means Algorithm Learner

Learning behavior

Clustering

The gathered learning portfolios

1 0.5

C1

Learning Performance

Fuzzy Inferring Refined fuzzy rules

Feature Reduction Algorithm Primal fuzzy

Neuro-Fuzzy Classifier

rules

C2

C3

Membership functions determined

Teacher

Figure 3. The flowchart of learning performance assessment Mining Learning Performance Rules Using Web-based Learning Portfolios

Determining Fuzzy Membership Functions by the Refined K-means Algorithm for the Neuro-Fuzzy Classifier This study employed the neuro-fuzzy classifier to discover the fuzzy knowledge rules from the learning portfolios for learning performance assessment. To identify the fuzzy knowledge rules by the neuron-fuzzy classifier, the membership functions used in the neuro-fuzzy classifier must be logically determined according to the data distribution of learning portfolios in advance. In this work, the proposed refined K-means clustering algorithm improved from the original K-means clustering algorithm (Xu & Wunsch, 2005) was employed to determine the centers of the triangle fuzzy membership functions automatically according to the data distribution 75

of each learning factor in learning portfolios for the neuro-fuzzy classifier herein. To obtain appropriate and interpretable learning performance assessment rules by the neuro-fuzzy classifier, this study sets the number of clusters in the refined K-means clustering algorithm as three based on the requirements of learning performance assessment in real teaching scenario. In other words, each considered learning factor contains three linguistic terms, i.e. low, moderate, and high, to describe a fuzzy rule. After that, the membership functions of the triangle fuzzy sets were automatically determined according to three cluster centers of each learning factor. Suppose the centers of three linguistic terms determined by the refined K-means clustering algorithm are respectively c1 ,

c 2 , and c3 , the membership functions for the linguistic terms “low”, “moderate”, and “high” can be formulated as follows: i.

The membership function for the linguistic term “low”

⎧ 1 ⎪⎪ c − x μ Low ( x) = ⎨ 2 ⎪ c 2 − c1 ⎩⎪ 0

if

x ≤ c1

if

c1 < x < c 2

if

x ≥ c2

(3)

ii. The membership function for the linguistic term “moderate”

⎧ 0 ⎪ x − c1 ⎪ ⎪⎪ c 2 − c1 μ Moderate ( x) = ⎨ 1 ⎪ c3 − x ⎪ ⎪ c3 − c 2 ⎪⎩ 0

if

x ≤ c1

if

c1 < x < c 2

if

x = c2

if

c 2 < x < c3

if

x ≥ c3

if

x ≤ c2

if

c 2 < x < c3

if

x ≥ c3

(4)

iii. The membership function for the linguistic term “high”

⎧ 0 ⎪⎪ x − c 2 μ High ( x) = ⎨ ⎪ c3 − c 2 ⎪⎩ 1

(5)

Next, the difference of the proposed refined K-means clustering algorithm with the original K-means clustering algorithm is compared and explained herein. Suppose there are s patterns in the gathered learning portfolios, the original K-means clustering algorithm is employed to find the k centers for each learning factor, where k ≤ s , and the set of the k cluster centers are represented as follows: C s ∈ {c1 ,..., c l ,..., c r ,..., c k } (6) where C s is the set of the k cluster centers, c l is the cluster center with the nearest Euclidean distance to the left boundary, and c r is the cluster center with the nearest Euclidean distance to the right boundary. In the proposed refined K-means clustering algorithm, the cluster centers c l and c r determined by the original K-means clustering algorithm are tuned by the refined value unchangeable. The refined value is computed as follows:

δ=

δ

; moreover, the other cluster centers are

max(S) + min(S) 2

(7)

where max(S ) and min (S ) represent the maximum and minimum feature values of patterns in each learning factor, respectively. Therefore, the new cluster centers determined by the refined K-means clustering algorithm can be represented as follows: C s( new) ∈ {c1 ,..., c l − δ,..., c r + δ,..., c k } (8) where C s( new) is the new set of the k cluster centers. 76

Figure 4 gives an example to explain how to tune the membership functions by the refined value δ defined in the proposed refined K-means clustering algorithm. In Fig. 4, the circle and square notations with black and grey colors represent respectively the cluster centers determined by the K-means and refined K-means clustering algorithms. Moreover, the dotted and solid lines respectively represent the membership functions determined by the K-means and refined K-means clustering algorithms. Compared with the original K-means clustering algorithm, we find that the refined K-means clustering algorithm has benefits in terms of promoting classification ability for the boundary patterns between different classes and reducing the number of unknown patterns due to expanding the boundary range while employing the neuro-fuzzy classifier to discover the learning performance assessment rules based on learning portfolios. The later experimental results will confirm these benefits.

x2 ′ μ 22

μ 22

μ 21 ′ μ 21

δ

δ

x1 δ

δ

′ μ11 μ11

′ μ12 μ12

Figure 4. Tuning membership functions by the refined value δ , where the circle and square notations with black and grey colors respectively represent the cluster centers determined by the K-means clustering algorithm and refined K-means clustering algorithm, and the dotted and solid lines respectively represent the membership functions determined by the K-means and refined K-means clustering algorithms Neuro-Fuzzy Classifier for Mining Learning Performance Assessment Rules After the triangle fuzzy membership functions were determined by the refined K-means clustering algorithm, the neuro-fuzzy classifier proposed by Nauck (Nauck & Kruse, 1999) was employed to infer the fuzzy rules for the learning performance assessment based on learning portfolios. Figure 5 shows the learning architecture of the used neuro-fuzzy classifier which can be represented as four-layer feed-forward networks. The first layer is the input layer which consists of input neurons and each neuron corresponds to an input feature. The second layer is the fuzzification layer which consists of fuzzy set neurons and each neuron corresponds to a linguistic term, such as low, moderate, and high, and so on. The third layer is the rule layer which consists of rule neurons and each neuron corresponds to a fuzzy rule. The fourth layer is the class layer which consists of class neurons and each neuron corresponds to a class label. To explain the detailed operation procedures of the neuro-fuzzy classifier, the notations used in Fig. 5 are first explained as follows: x k is the membership function of the k

th

input feature, Rule j is the j

In addition, ar j denotes the activation strength of the j

th

th

k th input feature, μ kl denotes the l th

fuzzy rule, and Class i is the i

th

output class.

neuron in the rule layer, and aci denotes the

th

activation strength of the i neuron in the class layer. Moreover, the input layer only receives input features from the gathered learning portfolios and directly passes the input features to the fuzzification layer for computing the fired membership degrees. Each fuzzy set neuron in the fuzzification layer could connect to several rule neurons and each rule neuron only connects to one class neuron, but each class neuron could be simultaneously connected by different rule neurons since different antecedent parts of fuzzy rules could lead to the same consequent part.

77

Class1

ar1

ar2

μ11

μ1m

μ12

Class c

Class i

arr

arj

Rule r

Rule j

Rule 2

Rule 1

acc

aci

ac1

μkl

μ n1

Input 1

Input k

Input n

x1

xk

xn

μn 2

μnm

Figure. 5 Four-layer learning architecture of the employed neuro-fuzzy classifier In this study, the employed neuro-fuzzy classifier follows six steps to generate fuzzy rules for learning performance assessment, and summarized as follows: Step 1. Computing the activation strength of each fuzzy neuron in the rule layer by the minimum operator, and formulated as follows:

ar j = min {u k( j ) }

(9)

k =1, 2 ,..., n

where

ar j is the activation strength of the antecedent part of the j th rule neuron, u k( j ) represents the

j th rule fired by the k th input feature, which can be computed as = max{u k1 , u k 2 ,..., u km }, and m is the number of the defined linguistic terms of each input feature.

membership degree of the

u k( j )

Step 2. Computing the activation strength of each class neuron in the class layer by the maximum operator, and formulated as follows:

aci = max {arj( i ) }

(10)

j =1, 2 ,..., z i

where

aci is the activation strength of the consequent part of the i th class neuron, ar j

activation strength of the

i

th

strengths of the rule neurons represented as ar fired fuzzy rules of the

represents the

th

j rule which is the set member of the activation ∈ {ar1 , ar2 ,..., ar j ,..., arr }, and z i is the number of the

class neuron fired by the (i ) j

(i)

i th class neuron, and z i ≤ r .

Step 3. Computing the corresponding class register values by respectively calculating the summation of the activation strengths fired by all training patterns for each fuzzy rule, and formulated as follows: 78

y

t i ( j ) = ∑ aci

( j)

( X p ), where i = 1,2,...,c, j = 1,2,..., r

(11)

p =1

where

ti ( j ) is the class register value of the i th class for the j th rule, aci th

of the i class neuron fired by the patterns.

th

( j)

( X p ) is the activation strength

th

p training pattern for the j rule, and y is the total number of training

Step 4.Finding the class label with the largest class register value for each fuzzy rule, and formulated as follows: (12) τ( j ) = arg max{ t i ( j )} i =1,2 ,...,c

where τ(

j ) represents the class label with largest class register value for the j th rule.

Step 5. Evaluating the performance for each fuzzy rule, and formulated as follows:

p( j ) = tτ( j ) ( j ) −

∑t ( j )

i i =1,2 ,...,c ,i ≠ τ

(13)

p( j ) represents the performance of the j th fuzzy rule, τ( j ) is the class label with the largest class th register value for the j fuzzy rule.

where

Step 6. Collecting the fuzzy rules with the best performance in each class as the learning performance assessment rules.

Feature Reduction Scheme for Simplifying the Learning Performance Assessment Rules To generate greater efficient and even accurate fuzzy rules, the original fuzzy rules discovered by the neurofuzzy classifier can be refined by the proposed feature reduction scheme. In the feature reduction process, it is assumed that there are sufficient relevant features in the original feature set to discriminate clearly between categories, and that some irrelevant features can be eliminated to improve efficiency and even accuracy. Generally, elimination of redundancy will improve efficiency without losing accuracy, and elimination of noise will improve both efficiency and accuracy (Chakraborty & Pal, 2004). In particular, the antecedent parts of fuzzy rules discovered by the neuro-fuzzy classifier always contain all features so that the main relevant features related to the classification accuracy cannot be clearly revealed as well as the inference efficiency is descended. More significantly, the antecedent parts of the fuzzy rules discovered by the neuro-fuzzy classifier containing all features will lead to over strict fired conditions so that many patterns cannot be inferred, and called as unknown patterns herein. The number of unknown patterns will obviously reduce the classification accuracy rate. Moreover, the simplified fuzzy rules for learning performance assessment are more easily interpreted by teachers, thus helping teachers to tune their teaching strategies. Based on these reasons, this study proposes a feature reduction scheme to simplify the discovered fuzzy rules. To obtain the simplified fuzzy rules, the proposed feature reduction scheme only performs the feature reduction scheme to the fuzzy rules with the best performance in each class. After that, the feature reduction is performed based on the proposed performance evaluation method for all feature combinations of the antecedent parts containing in each discovered fuzzy rule. Figure 6 shows that the linguistic terms used in a fuzzy rule usually have overlapped intervals between two neighborhood linguistic terms, such as the intervals I 1 , I 2 , I 3 , and I 4 . In these overlapped intervals, a training pattern with feature value mapped in the corresponding overlapped interval will simultaneously fire the antecedent parts of two linguistic terms defined in a feature dimension and the fired strengths of two linguistic terms are different. For example, the membership degree of the linguistic term “Small” fired by a training pattern with feature value mapped in the interval I 1 is larger than the linguistic term “Moderate”. In contrast, the membership degree of the linguistic term “Small” fired by a training pattern with feature value mapped in the interval I 2 is smaller than the linguistic term “Moderate”. Based on the observation, an excellent feature combination in a discovered fuzzy rule should trigger the membership degree on the fired linguistic term as large as possible, but trigger the membership degree on the other non-fired linguistic terms as small as possible when the discovered fuzzy rule is fired by a training pattern. Herein, the fired linguistic term indicates the linguistic term with the largest membership degree fired by a training pattern, and the other linguistic terms are viewed as 79

the non-fired linguistic terms. This study names the difference between the membership degree of the fired linguistic term with the summation of the membership degrees of the non-fired linguistic terms as the match degree. To develop the proposed feature reduction scheme, the match degree is first defined as follows:

u lj (i ) = u ljλ (i ) − where u j (i ) represents the match degree of the l

pattern,

n

∑ u (i ) l jk

(14)

k =1,k ≠ λ

j th feature of the l th fuzzy rule fired by the i th training

l

u jk ( i ) is the membership degree of the k th linguistic term of the j th feature of the l th fuzzy rule

i th training pattern, n represents the number of the defined linguistic terms in each fuzzy rule, λ th indicates the linguistic term with the largest membership degree fired by the i training pattern. fired by the

After the match degree is obtained, the performance of the discovered fuzzy rule with the considered feature combination can be measured as follows: ϕ

vql = l

where vq represents the performance of the of training patterns,

ϕ

fuzzy rule fired by the

t

∑∑ u (i ) l j

j =1 i =1

(15)

ϕ

l th fuzzy rule under the q th feature combination, t is the number

is the number of features,

l

u j ( i ) is the membership degree of the j th feature of the l th

i th training pattern.

Moreover, the total number of feature combinations for a discovered fuzzy rule can be computed as follows: p

g=∑ j =1

where

p! j ! ( p − j )!

(16)

g is the total number of feature combinations for a discovered fuzzy rule, p is the total number of j is the number of the considered features.

feature dimensions,

l

Figure 6. The overlapped regions between various linguistic terms, where u jk ( i ) is the membership degree of the

k th linguistic term of the j th feature of the l th fuzzy rule fired by the i th training pattern, and x lj ( i ) is the feature value of the

i th training pattern corresponding to the j th feature of the l th fuzzy rule

Fuzzy Inference by the Discovered Fuzzy Rules for the Learning Performance Assessment This section explains how to infer the learning performance according to the discovered fuzzy rules by fuzzy inference. The discovered fuzzy production rules are formed by IF-THEN rules represented as follows: IF

X 1 = A1

and

X 2 = A2

THEN

Y=B 80

where

X i and Y denote linguistic variables, and Ai and B represent linguistic terms.

A defuzzification strategy aims to convert the outcome of fuzzy inference into a crisp class label. In this study, the maximum operator (Lin & Lee, 1996) is employed as the defuzzification scheme, to infer the crisp class label from the respective fired degrees of membership functions. Restated, the result of the learning performance assessment is assigned as the class label with the largest membership degree fired by a training pattern among the discovered fuzzy rules.

Experiments The personalized e-learning system (PELS) was published on the web site http://192.192.6.86/irt4 to provide personalized e-learning services and assess the learning performance of individual learners by web-based learning portfolios. To verify the evaluation quality of the discovered fuzzy rules for the learning performance assessment, some third-year students of Jee-May Elementary School (http://www.jmes.tpc.edu.tw/), who had majored in the “Fractions” course unit in elementary school mathematics, were invited to test this system. The experimental results are described as follows. The Format of Learning Portfolio Gathered by PELS The “Fractions” unit currently includes a total of 17 course materials with various difficulty levels, each conveying similar concepts. The system was used by 583 learners from 18 different classes of Taipei County Jee-May Elementary School (http://www.jmes.tpc.edu.tw/). Among the eight gathered learning factors, the final testing score G was obtained through the summative assessment scheme of fixed-length testing examination after the entire learning process is completed. Table 4 shows the format of the partial learning portfolios in the user profile database gathered by PELS.

Table 4. The format of the learning portfolio in the user profile database Learning G Factor Learning (0~100) Record T1 83.24

RR (0~1)

RT (sec)

LA (-3~+3)

CR (0~1)

PN (piece)

AS (point)

EL (0~1)

0.630

2212

1.68

0.815

0

41

1.00

T2

74.17

0.600

1413

0.26

0.533

0

42

1.00

︴

︴

︴

︴

︴

︴

︴

︴

︴

T583

84.81

0.436

3747

1.36

0.846

1

27

0.95

Evaluating Accuracy Rate of Learning Performance Assessment

Evaluation Method To measure the accuracy rate of learning performance assessment for the proposed method, the score level method was used to evaluate the accuracy rate of the predicted learning performance. In this evaluation method, each learner is assessed according to one of three score levels based on the mapping membership degrees of learning factor G. For example, the score level of a learner with a final test score of 86.16 was set to moderate level if the linguistic term of moderate level has the largest mapping membership degree among all linguistic terms of the learning factor G.

Experimental Results In our experiments, half the learning portfolios among 583 learners selected randomly were used as training data and the remaining learning portfolios were used as testing data. Table 5 illustrates the number of learning records in each grade level. To obtain simple and interpretable fuzzy rules for learning performance assessment, the number of cluster centers in the K-means algorithm is set to 3. That is, each learning factor contains three 81

various linguistic terms to describe a fuzzy rule, and named as “low”, “moderate”, and “high” herein. Table 6 shows the determined centers of the linguistic terms for eight learning factors by the K-means algorithm. To obtain fuzzy rules for learning performance assessment more accurately, the proposed refined K-means algorithm was employed to tune the centers of the linguistic terms listed in Table 6. Tables 7 and 8 display the revised amount of cluster centers for each learning factor and the revised centers of the linguistic terms of eight learning factors by the refined K-means algorithm, respectively. Figure 7 reveals the membership functions of triangle fuzzy sets for eight learning factors determined by the revised centers of the linguistic terms in the refined K-means algorithm. Basically, the left and right widths of triangle membership functions of the linguistic terms “moderate” for each learning factor are determined by the differences between the self-center with the left and right neighborhood centers, respectively. Moreover, the left and right widths of triangle membership functions of the linguistic terms “low” and “high” for each learning factor are determined by the difference between the self-center with the left or right neighborhood center as well as the boundary of each learning factor. Table 5. The number of learning records in each grade level Training data set Testing data set Whole data set Data set Item Low Moderate High Low Moderate High Low Moderate High Number of 51 194 47 65 188 38 116 382 85 patterns Total number of 292 291 583 patterns

Table 6. The determined centers of the linguistic terms for eight learning factors by the K-means algorithm Learning factor Linguistic term The centers of the linguistic term “low” The centers of the linguistic term “moderate” The centers of the linguistic term “high”

G

RR

RT

LA

CR

PN

AS

EL

81.95

0.25

739.24

1.25

0.77

1.52

28.38

0.74

87.25

0.44

1585.02

1.58

0.87

2.61

53.88

1.05

90.44

0.63

3284.04

1.62

0.91

5.08

90.12

1.08

Table 7. The revised amount of cluster centers for each learning factor Learning factor G

RR

RT

LA

CR

PN

AS

EL

12.5

0.5

3154

1.15

0.5

13

175

1.87

Revised amount The revised amount

δ

Table 8. The revised centers of the linguistic terms of eight learning factors by the refined K-means algorithm Learning factor Linguistic term The revised center of the linguistic term “low” The revised center of the linguistic term “moderate” The revised center of the linguistic term “high”

G

RR

RT

LA

CR

PN

AS

EL

69.45

-0.26

-2414.76

0.10

0.27

-11.48

-146.62

-1.13

87.25

0.44

1585.02

1.58

0.87

2.61

53.88

1.05

102.94

1.13

6438.04

2.77

1.41

18.80

256.12

2.95

Next, the neuro-fuzzy neural network with the refined K-means algorithm for logically determining membership functions was employed to discover the formative evaluation rules based on the learning portfolios of the training data set. There are totally 24 fuzzy rules discovered by the employed neuro-fuzzy neural network in our 82

experiment. Among all discovered fuzzy rules, there are 10, 9, and 5 rules discovered for assessing the low, moderate, and high grade levels, respectively. Table 9 shows three fuzzy rules with the best performance for assessing three different grade levels based on gathered learning portfolios. In our experiment, ten independent runs were performed to yield an average performance for the training and testing data sets. Table 10 displays the averaged accuracy rate of learning performance assessment evaluated by the discovered rules listed in Table 9. The averaged accuracy rates of learning performance assessment for the training and testing data sets are 74.3% and 71.1%, respectively. G

RR

RT

1

1

1

0.5

0.5

0.5

0

69.45

87.25 LA

0

102.94

-0.26

0.44 CR

1.13

0 -2414.76

1

1

1

0.5

0.5

0.5

0

0.1

1.58 AS

0

2.77

1

1

0.5

0.5

0 -146.62

53.88

0

256.12

0.27

-1.13

0.87 EL

1.05

0

1.41

1585.02 PN

-11.48

6438.04

2.61

18.8

2.95

Figure 7. The membership functions of learning factors determined by the refined K-means algorithm Table 9. The learning performance assessment fuzzy rules discovered by the neuro-fuzzy network with the refined K-means algorithm for determining fuzzy membership functions Consequent Learning factor Antecedent part part The discovered rule The rule 1 The rule 2 The rule 3

RR

RT

LA

CR

PN

AS

EL

G

M M H

M M M

L M M

L M M

M M M

M M M

M M M

Low Moderate High

Table 10. The accuracy rate of learning performance assessment evaluated by the discovered rules listed in Table 9 Data set Training data set Testing data set Item Number of patterns Number of the discovered rules Number of the used fuzzy sets in the discovered rules Number of patterns with correct learning performance assessment Number of patterns with incorrect learning performance assessment

292 3 10

291 3 10

217

207

66

73 83

Number of patterns with unknown learning performance assessment Averaged accuracy rate of learning performance assessment

9

11

0.743

0.711

Table 11. All combinations of seven learning factors in the antecedent parts of the discovered fuzzy rules evaluated in various training cycles The combined learning Training cycle factors 1~7 x ~x 1

7

8

x1 , x2

9

x1 , x3

10

x1 , x4

M

M

125

x1 , x3 , x4 , x5 , x6 , x7

126

x2 , x3 , x4 , x5 , x6 , x7

127

x1 , x2 , x3 , x4 , x5 , x6 , x7

90

80 70

60

Performance

50

40 30

20 10

0 -10

0

20

40

60 80 Training Procedure of Rule 3

100

120

140

Figure 8. The performance evaluation plot of the discovered rule 1 for assessing the low score level Although the used neuro-fuzzy classifier with the refined K-means algorithm can discover as few fuzzy rules as possible to assess the learning performance of an individual learner accurately, the obtained fuzzy rules always contain all learning factors so that the main learning factors that affect the learning performance cannot be revealed clearly. To simplify the obtained fuzzy rules, the feature reduction scheme was employed to further reduce the fuzzy rules of learning performance assessment. In the proposed feature reduction scheme, the performance index defined in Eq. (17) for all combinations of learning factors of the antecedent parts of discovered rules will be used to find the main learning factors that affect the learning performance. Table 11 lists all combinations of seven learning factors contained in the antecedent parts of the discovered fuzzy rules, where

xi represents the i th considered learning factor, and their performances were respectively evaluated in

various training cycles. Figures 8, 9 and 10 show the performance evaluation plots of the discovered rules 1, 2 and 3 for assessing the low, moderate and high score levels, respectively. Table 12 shows the learning factor combinations with top six excellent performances for three discovered fuzzy rules. In particular, the learning factor combination with best performance index among three discovered fuzzy rules is x3 , x2 and x1 . The result indicates that the third learning factor of the discovered fuzzy rule 1, i.e. learner ability (LA), is the main learning factor for assessing the learners with low score level. Similarly, the results also indicate that the second and first learning factors are the main learning factors for the discovered fuzzy rules 2 and 3, respectively. Table 13 shows the simplified learning performance assessment rules after performing the proposed feature reduction scheme. Table 14 displays the accuracy rate of learning performance assessment by these simplified fuzzy rules. 84

Compared with the results of learning performance assessment listed in Table 10, we find that the accuracy rate of the learning performance assessment using the simplified fuzzy rules is only slightly poor than the discovered fuzzy rules without performing feature reduction scheme. However, the simplified fuzzy rules provide benefits to teachers in terms of easily understanding main learning factors that affect learning performance for learners with various score levels (Paiva & Dourado, 2004; Mikut, Jakel & Groll, 2005). Moreover, this idea can be beneficially applied to immediately examine the learning progress of learners, and to perform interactively control learning for web-based learning systems. In the meanwhile, the simplified fuzzy rules will obviously promote the inference efficiency of learning performance assessment for web-based learning systems (Paiva & Dourado, 2004; Mikut, Jakel & Groll, 2005). 20

15

Performance

10

5

0

-5

-10

0

20

40

60 80 Training Procedure of Rule 3

100

120

140

Figure 9. The performance evaluation plot of the discovered rule 2 for assessing the moderate score level 30

25

20

Performance

15

10

5

0

-5

-10

0

20

40

60 80 Training Procedure of Rule 3

100

120

140

Figure 10. The performance evaluation plot of the discovered rule 3 for assessing the high score level Table 12. The top six combined learning factors with excellent performance of learning performance assessment for three discovered rules The discovered rule 1 The discovered rule 2 The discovered rule 3 The The The Training combined Performance Training combined Performance Training combined Performance cycle index cycle index cycle index learning learning learning factor factor factor 3 19 14 44 9 21

x3 x3 , x4 x2 , x3 x2 , x3 , x4 x1 , x3 x3 , x6

82.71

2

62.98

15

49.75

4

47.58

30

47.57

8

46.59

49

x2 x2 , x4 x4 x1 , x2 , x4 x1 , x2 x2 , x4 , x6

16.78

1

16.29

8

15.79

10

15.00

30

14.61

12

14.34

32

x1 x1 , x2 x1 , x4 x1 , x2 , x4 x1 , x6 x1 , x2 , x7

27.50 22.14 21.65 20.02 18.98 18.25 85

Table 13. The discovered learning performance assessment rules after performing feature reduction Consequent Antecedent part Learning factor part The discovered rule The rule 1 The rule 2 The rule 3

RR

RT

LA

G

L

Low Moderate High

M H

Table 14. The accuracy rate of learning performance assessment by the simplified fuzzy rules listed in Table 13 Data set Item Training data set Testing data set Number of patterns Number of the discovered rules Number of the used fuzzy sets in the discovered rules Number of patterns with correct learning performance assessment Number of patterns with incorrect learning performance assessment Number of patterns with unknown learning performance assessment Accuracy rate of learning performance assessment

292 3

291 3

3

3

212

204

80

87

0

0

0.726

0.701

Conclusion This study presents an effective learning performance assessment approach which contains the neuro-fuzzy network with the refined K-means algorithm for logically determining membership functions and the proposed feature reduction scheme to discover simplified and small amount fuzzy rules for evaluating learning performance of individual learners based on the gathered learning portfolios. The proposed method can help teachers to perform precise formative assessments according to the web-based learning portfolios of individual learners in a web-based learning environment. The inferred learning performance can be applied as a reference guide for teachers and as learning feedback for learners. Such a feedback mechanism enables learners to understand their current learning status and make suitable learning adjustments. Additionally, teachers can determine the main factors affecting learning performance in a web-based learning environment according to the interpretable learning performance assessment rules obtained. These factors can be used by teachers to tune their teaching strategies. Meanwhile, teachers can devote themselves to teaching job, since they save significant time in performing learning evaluation.

Acknowledgment The authors would like to thank the National Science Council of the Republic of China, Taiwan for financially supporting this research under Contract No. NSC94-2520-S-026-002.

References Baker, B. B., Baker, F. B., & Frank B. B. (1996), Item Response Theory: Parameter Estimation Techniques, New York: Marcel Dekker. Bruner, J. S. (1996). Toward a Theory of Instruction, Cambridge, MA, USA: Harvard University Press. 86

Campbell, D., Nettles. D., & Melenyzer, B. (2000). Portfolio and Performance Assessment in Teacher Education, Boston: Allyn and Bacon. Carrira, R., & Crato, J. M. (2004). Evaluating Adaptive User Profiles for News Classification. International Conference on Intelligent User Interfaces, January 13-16, 2004, 206-212. Chakraborty, D., & Pal, N. R. (2004). A Neuro-Fuzzy Scheme for Simultaneous Feature Selection and Fuzzy Rule-Based Classification. IEEE Transactions on Neural Networks, 15, 110–123. Chang, C.-C. (2002). Building A Web-Based Learning Portfolio for Authentic Assessment. Proceedings of the International Conference on Computers in Education (Vol. 1), 129-133. Chen, C.-M., Lee, H.-M., & Chen, Y.-H. (2005). Personalized E-Learning System Using Item Response Theory. Computers & Education, 44 (3), 237-255. Chen, C.-M., Liu, C.-Y., & Chang, M.-H. (2006). Personalized Curriculum Sequencing Using Modified Item Response Theory for Web-based Instruction. Expert Systems with Applications, 30 (2), 378-396. Dunham, M. H. (2002). Data Mining: Introductory and Advanced Topics, Prentice Hall. Gagnés, R, (1997). The Conditions of Learning and Theory of Instruction, New York: Holt, Reinehart & Winston. Lankes, A. M. D. (1995). Electronic Portfolios: A New Idea in Assessment, (ERIC Digest EDO-IR-95-9), Available April 11, 2006 at http://searcheric.org/digests/ed390377.html. Lin, C.-T., & Lee, C. S. G. (1996). Neural Fuzzy Systems: A Neuro-Fuzzy Synergism to Intelligent Systems, Prentice-Hall. Nauck, D., & Kruse, R. (1999). Obtaining Interpretable Fuzzy Classification Rules from Medical Data. Artificial Intelligence in Medicine, 16, 149-169. Mikut, R., Jakel, J., & Groll, L. (2005). Interpretability Issues in Data-Based Learning of Fuzzy Systems. Fuzzy Sets and Systems, 150 (2), 179–197. Paiva, R. P., & Dourado, A. (2004). Interpretability and Learning in Neuro-Fuzzy Systems. Fuzzy Sets and Systems, 147 (1), 17-38. Rahkila, M., & Karjalainen, M. (1999). Evaluation of Learning in Computer based Education Using Log Systems. 29th ASEE/IEEE Frontiers in Education Conference, 1, 12A3/16-12A3/21. Rasmussen, K., Northrup, P., & Lee, R. (1997). Implementing Web-based Instruction. In Khan, B. H. (Ed.), Web-Based Instruction, Englewood Cliffs, NJ: Educational Technology, 341-346. Torrance, H., & Pryor, J. (1998). Investigating Formative Assessment: Teaching, Learning and Assessment in the Classroom, Buckingham: Open University Press. Wang, W., Weng, J.-F., Su, J.-M., & Tseng, S.-S. (2003). Learning Portfolio Analysis and Mining in SCORM Compliant Environment. The 34th ASEE/IEEE Frontiers in Education Conference, TC2-17-TC2-34. Wu, A. K. W., & Leung, C. H. (2002). Evaluating Learning Behavior of Web-Based Training using Web Log. In Proceedings of the International Conference on Computers in Education, 736-737. Xu, R., & Wunsch, D. II (2005). Survey of Clustering Algorithms. IEEE Transactions on Neural Networks, 16 (3), 645-678. Zaiane, O. R., & Luo, J. (2001). Towards Evaluating Learners' Behavior in a Web-based Distance Learning Environment. IEEE International Conference on Advanced Learning Technologies, 357-360.

87

Morimoto, Y., Ueno, M., Kikukawa, I., Yokoyama, S. & Miyadera, Y. (2006). Formal Method of Description Supporting Portfolio Assessment. Educational Technology & Society, 9 (3), 88-99.

Formal Method of Description Supporting Portfolio Assessment Yasuhiko Morimoto Fuji Tokoha University 325 Obuchi, Fuji, Shizuoka 417-0801 Japan [email protected]

Maomi Ueno Nagaoka University of Technology 1603-1 Kamitomioka, Nagaoka, Niigata 940-2188 Japan [email protected]

Isao Kikukawa Tokyo Polytechnic University 1583 Iiyama, Atsugi, Kanagawa 243-0297 Japan [email protected],

Setsuo Yokoyama and Youzou Miyadera Tokyo Gakugei University 4-1-1 Nukuikita, Koganei, Tokyo 184-8501 Japan [email protected] [email protected] ABSTRACT Teachers need to assess learner portfolios in the field of education. However, they need support in the process of designing and practicing what kind of portfolios are to be assessed. To solve the problem, a formal method of describing the relations between the lesson forms and portfolios that need to be collected and the relations between practices and these collected portfolios was developed. These relations are indispensable in portfolio assessment. A support system for these based on the formal method was also developed. As the formal method of description can precisely and consistently describe these relations, the system makes it possible to support the assessment of portfolios in the design and practice phases.

Keywords Portfolio assessment, Formal description, Support system, Assessment tool, Educational evaluation

Introduction Although traditional methods of evaluation based on tests have improved in the field of education, portfolio assessment has attracted attention as a more authentic means of evaluating learning directly. The use of Electronic Portfolios (Digital Portfolios), in which the portfolio is saved electronically, is spreading with computerized learning systems (Mochizuki et al., 2003). However, portfolio assessment has not been carried out adequately in the educational field because of two problems, and teachers and learners both need support (Morimoto et al., 2005). ¾ Problem 1: When teachers decide to use portfolios for lessons that they intend to assess, they do not know what kinds of portfolios need to be collected. In other words, they need support in the design phase of portfolio assessment. ¾ Problemt 2: Learners do not know which collected portfolios should be used for practices. For example, when a learner carries out his or her self-assessment, he or she does not know that he or she should carry out the self-assessment while checking which collected portfolios should be used. In other words, learners need support on which portfolios they need to assess in the practice phase. Support for portfolio assessment is carried out with tools that treat electronic portfolios (Chang, 2001; Chen et al., 2000; Fukunaga et al., 2001). However, these tools cannot support teachers in selecting which portfolios to assess. Moreover, teachers have to match the type of collected portfolio and the method of practice to the tools being used because these tools individually determine the types of collected portfolios and the practice method. Therefore, support of portfolio assessment in the lesson that the teacher intends to assess cannot be expected. The relation between practices and collected portfolios is not clear, and practices are not supported. These tools cannot solve Problems (1) or (2). The purpose of this study is to achieve support for portfolio assessment at the design and practice phases, and solve these problems at the same time. We therefore developed a formal method of describing the relations ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

88

among elements that are indispensable in assessing portfolios and a support system for these based on this method. As the formal method of description can precisely and consistently describe the relations between the lesson forms and portfolios that need to be collected and the relations between practices and collected portfolios, the system makes it possible to support the assessment of portfolios in the design and practice phases. First, we outline the support of portfolio assessment that this study is aimed at. Second, we discuss the relations between elements indispensable to the support of portfolio assessment. These relations become the framework for the formal description method that we developed. Third, we explain the formal description method. Finally, we discuss the portfolio assessment support system (PASS), which is based on this method.

Support of portfolio assessment in this study Requirements for solving problems To solve Problem (1) in the design phase, we have to extract and clarify the relations between lesson forms and portfolios that need to be collected, and support portfolio-assessment design based on these relations. Moreover, to solve Problem (2) in the practice phase, we have to extract and clarify the relations between practices and collected portfolios, and support user portfolio-assessment practices based on these relations. Here, a person who carries out practices is called a “user”. Namely, a user means a learner, a teacher, or others. To achieve these two phases of support, we need to faithfully express the extracted relations. If a formal method that precisely and consistently describes the extracted relations is developed, a system is possible that systematically supports portfolio assessment based on the formal method. Therefore, the requirements for solving Problems (1) and (2) are as follows. Requirement(a): Extract and clarify the relations between the lesson forms and portfolios that need to be collected, and develop a formal method of describing these relations (Corresponds to Problem (1)). Requirement(b): Extract and clarify the relations between a user’s practices and collected portfolios that a user needs to carry out practices, and develop a formal method of describing these relations (Corresponds to Problem (2)). Requirement(c): Develop a system based on the formal method of description developed in (a) and (b) (Corresponds to Problems (1) and (2)). Blueprints to support portfolio assessment It is possible to support teachers’ designs for assessing portfolios and users’ practices by formally describing the relations between the lesson forms and portfolios that need to be collected (PDS: Portfolio assessment Design Semantics), which satisfy Requirement (a), and the relations between practices and collected portfolios (PPS: Portfolio assessment Practice Semantics), which satisfy Requirement (b), and develop a system based on these relations (see Figures 1 and 2). Figure 1 outlines a model illustrating how a teacher uses lesson forms to check PDS to select portfolios and how he or she uses portfolios to check PDS to select the lesson forms in the design phase. When he or she specifies the lesson form that he or she intends to teach, the system checks PDS and presents the portfolios that need to be collected dynamically according to this form. Moreover, when a teacher specifies portfolios that users need to collect first, the system presents lesson forms conforming to portfolios according to PDS. Thus, both lesson forms and portfolios are supported using PDS, and teachers can design portfolio assessment more easily. Both Requirements (a) and (c) are satisfied, and a solution to Problem (1) is expected.

Figure 1. Model supporting teacher’s portfolio-assessment design 89

Figure 2 outlines a model illustrating how a user uses practices to check PPS to select portfolios and how a user uses portfolios to check PPS to select practice in the practice phase. When a user specifies the practice that the user wants to carry out, the system checks PPS and presents collected portfolios that the user needs for the practice. The user can therefore effectively practice by checking necessary collected portfolios. Moreover, when the user specifies a collected portfolio, the system dynamically supports practice that adjusts the portfolio according to PPS. Thus, both practice and portfolios are supported using PPS. Here, both Requirements (b) and (c) are satisfied, and a solution to Problem (2) is expected.

Figure 2. Model supporting user’s portfolio-assessment practices Advantages of formal description We want to describe the PDS and PPS by using a formal method. The advantages of formally describing PDS and PPS are as follows. ¾ The relations between elements indispensable to the support of portfolio assessment can precisely be described by removing contradictions and vagueness, ¾ It is possible for the system developer to develop a support system that works systematically according to semantics formally described by constructing the system, which can interpret the framework for the formal description method and work based on it, ¾ As a result, a portfolio assessment support system that provides adaptive support according to the lesson form or a user’s practices can be achieved, and ¾ Changing the semantics of the framework with formal description rules can easily change the support methods of the developed system.

Extracting relations to support portfolio assessment Outline In this section, we discuss our extraction of the relations between elements indispensable for supporting portfolio assessment, which became the framework for formal description that was developed in this study. PDS and PPS were extracted. It is necessary to extensively investigate actual portfolios to assess them. However, the ideas and methods for assessing portfolios are inconsistent and varied (Bruke, 1992; Hart, 1993; Barton & Collins, 1996; Puckett & Black, 1999; Shores & Grace, 1998). We therefore focused on assessing indispensable portfolios that teachers actually use. We analyzed 298 practical records such as lesson plans that had actually been used. We extracted and analyzed methods and ideas applied to actual practical portfolio-assessment records, which teachers had obtained by trial and error. This paper discusses support for portfolio assessment based on these extracted results. However, this might be insufficient for extraction. This is because there may be a method of assessing portfolios that has been derived from analytical targets. We used these extraction results to devise a formal method of description. Therefore, although the relations we extracted form the framework for supporting portfolio assessment, which were our study targets, the formal method of description that we developed and discuss in the next section needs to be changed based on the framework. Extracting PDS We clarified the relations between lesson forms and portfolios that needed to be collected (Table 1). These relations will be represented by “Portfolio assessment Design Semantics (PDS)”. The rows in Table 1 are for 90

lesson forms, and the columns are for collected portfolios. Here, “X” means that the portfolio corresponds to the lesson form that needs to be collected. For example, if a teacher wants to carry out portfolio-assessment design, he or she decides the lesson form and finds the “X” in the corresponding row. Portfolios with “X” need to be collected. Teachers can also identify corresponding lesson forms by specifying portfolios first. Therefore, portfolios that need to be collected according to lesson forms can be adequately identified by PDS. We found from the rows that lesson forms conforming to the assessment of portfolios changed under three conditions, i.e., the lesson style, the person who set tasks for the lesson, and the person who created rubrics. In extracting lesson styles, we paid attention to learners’ behaviors in a class, and classified lesson forms into eleven lesson styles. These were “Lecture”, where the lesson is given in the form of a lecture, “Question and Answer”, where learners answer a teacher’s questions, “Exercise”, where learners repeat exercises, “Skill acquisition”, where skills are acquired, “Expression”, where the body is used to express ideas, “Creation”, where something is created, “Discussion”, where there is a discussion, “Seminar”, where people in a group learn from one another, “Experiment or Observation”, where phenomena are verified through experiments or observations, “Experience”, where social and technical experience is used, and “Problem Solving”, where learners attempt problem solving by themselves. In extracting the “person who assigns tasks (Setting Tasks)”, we found that there were two kinds of cases where the teacher set them or the learner set them. In extracting “person who creates rubrics (Rubric Creation)”, we found there were three kinds of cases where the teacher created them, the learner created them, or the teacher and the learner collaboratively created them. Collected portfolios in the columns were classified separately in terms of portfolios concerning records of assessment (Assessment Portfolios), portfolios concerning learners’ work (Work Portfolios), portfolios concerning records of learning processes (Process Portfolios), and other portfolios. We classified them according to their original role in learning without getting caught up with the common name for the portfolio. Work Portfolios were classified into ten kinds. We classified Production into two types. The first was “Sample Based”, which was based on a sample, the second was “Original Based” which was the learner’s original. We also classified Reports and Notes into two types. We called one’s own assessments in the Assessment Portfolios with rubrics “Self-assessment”, and called looking back on one’s activities without rubrics “Reflection”. We also called mutual assessments with rubrics “Other-assessment”, and called advising one another freely without rubrics “Advice”. We classified Process Portfolios into three types. The first was “Learning-record” where the learner objectively describes his or her ongoing learning in a log, the second was “Anecdotal-record” where the teacher describes a learner’s spontaneous events and behaviors, and the third was “Conversation-record” which describes conversations. We also classified the Learning-record into two types. The first was “Premeditatedrecord”, which describes premeditation according to lesson plans, and the second was “Situation-record”, which describes situations. Other portfolios consisted of records of portfolio conferences (“Portfolio Conference”), learning plans (“Learning Plan”), and learning materials (“Learning Material”). Extracting PPS We extracted the relations between practices and collected portfolios (Table 2). The relations will be represented by “Portfolio assessment Practice Semantics (PPS)”. Here, “X” means that the portfolio corresponds to the practices that are necessary to carry it out. For example, when a learner carries out practices, he or she finds an “X” in the corresponding row. Portfolios with “X” are collected portfolios that are needed for carrying out practices. The learner can also identify corresponding practices by specifying portfolios first. Therefore, portfolios that are needed for practices can be adequately identified by PPS. We found ten kinds of practices that could be carried out during the process of assessing portfolios. These were “Browsing”, where collected portfolios were perused, “Self-assessment”, where one assesses one’s self, “Otherassessment”, where others are assessed, “Reflection”, where one reflects on his or her learning, “Advice”, where advice is given to someone, “Registration of Work Portfolios”, where work portfolios are registered, “Selection of Collected Portfolios”, where one’s collected portfolios are carefully selected, “Process Recording”, where process portfolios are recorded, “Rubric Creation”, where rubrics are created, and “Portfolio Conference”, where the state of the portfolio conference is recorded. We also found teachers, learners, and others (e.g., parents and specialists) were people who carried out practices.

91

92

Teacher

Learner

Learner

X

Teacher

Learner

Learner

X

Learner X

Teacher X

Learner X

Teacher X

Learner X

Teacher X

Learner X

Problem solving

Experience

Learner X

Teacher X

Learner X

Teacher X

Experiment Teacher X or observation Learner X

Seminar

Discussion

Creation

Expression

Teacher

Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner Teacher Collaboration Learner

X

X X

X X

X X

X X

X X X X

X

X

X X X X

X X X

X X X X

X X X X

X X

X X X X

X X

X

X X X

X X X X

X X X X

X X X X

X X

X X

X X X X

X X

X X

X X

X X

X

X

X X X

X X X X

X X X X

X X X X

Teacher X Collaboration X

Learner

X Teacher Teacher X Collaboration X Skill Learner acquisition

Exercise

Learner

Teacher X Collaboration X

Production

Report

Note

X X

X X X X

X X

X

X X

X X X X

X X X X

X

X X

X

X

X X

X X X X

X X

X

X X X

X X X X

X X X X

X X X X

X X

X X

X X

X

X

X X

X X X X

X X

X

X X

X X X X

X X X X

X

X X

X

X

X X

X X

X

X X

X X

X X

X X

X X

X

X

X X

X X X X

X X

X

X X X

X X X X

X X

X X

X X X X

X X

X

X X

X X X X

X X

X X

X X X X

X X

X X X

X X X X

X X

X X

X X

X

X

X X

X X

X X

X

X X

X X X X

X X

X

X X

X X X X

X X X X

X

X X

X

X X

X

X X

X

X

X X

X X X X

X

X X

X X

Collection Test Presentation

Work Portfolios

SelfOtherSample Original Lesson Other Reflection Advice Conclusion Impression assessment assessment Based Based Note Note

Assessment without rubric

Assessment Portfolios Asssessment with rubric

X

X

Rubric Creation Rubric

Teacher X Collaboration

Setting Tasks

X Teacher Question Teacher X Collaboration Learner and Answer

Lecture

Lesson style

Lesson form

Other Portfolios

X X

X X

X X

X X

X X

X

　 　 X X

X X

X X

X X

X X

X X

X

X

X X

X X X X

X

X

X X

X X X X

X X X X

X

X X

X

X X

X X

X X

X X X X

X X

X

X X

X X X X

X X

X

X X X

X X X X

X X X X

X X X X

X X

X X

X X

X

X

X X

X X X X

X X

X

X X X

X X X X

X X X X

X X X X

X X

X X

X X

X

X

X X

X X X X

X

X

X X

X

X X

X

X X

X

X X

X

X X

X X X X

X X

X

X X X

X X X X

X X X X

X X X X

X X

X X

X X

X

X

Learning-record Practical Conversation Anecdotal Portfolio Learning Learning ability Situation Premeditated -record -record Conference Plan Material -record -record

Process Portfolios

93

Portfolio Conference

Rubric Creation

Process Recording

Sellection of collected portfolios

Registration of work portfolios

Advice

Reflection

Other-assessment

Self-assessment

Browsing

Practices

Learner Teacher Others Learner Teacher Others Learner Teacher Others Learner Teacher Others Learner Teacher Others Learner Teacher Others Learner Teacher Others Learner Teacher Others Learner Teacher Others Learner Teacher Others

User Note

X X X

X

X

X X X X

X X X

X X

X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X

X X

X X X

X X X X

X X X

X X

X

X X

X X X

X X X X

X X X

X X

X

X X

X X X

X X X X

X X X

X X

X

X X

X X X

X X X X

X X X

X

X

X

X

X X X

X X X

X

X X X X

X X X

X X X

X X

X X X X

X X X

Learning-record Practical Conversation Anecdotal Portfolio Learning Learning ability Situation Premeditated -record -record Conference Plan Material -record -record

Other Portfolios

X X X

X X X

X

X X X X

Collection Test Presentation

Process Portfolios

X X

X X X

X X

X X X X

Report

Lesson Other Sample Original SelfOtherConclusion Impression Reflection Advice Note Note Based Based assessment assessment

Production

Work Portfolios

X X

X X

X

X X X

X X X

X X X X

Rubric

Assessment without rubric

Assessment Portfolios

Asssessment with rubric

Formal description method Development policy This section describes the development of the method for formally describing PDS and PPS. Formal methods of description are currently being developed and used within the syntax definitions of programming languages, the context of software engineering, and other applications. These are necessary and indispensable for achieving systematic processing by computers, and specialists can effectively obtain precise and consistent specifications for system models. The formal method of description we have aimed at in this study should be able to precisely and consistently describe relations between elements indispensable for supporting portfolio assessment. In other words, the method of formal description can describe the structure and content of the model supporting the design of teachers’ portfolio-assessment designs (Figure 1) and the model supporting users’ portfolio-assessment practices (Figure 2). Moreover, it is necessary to design description rules to change dynamically to provide further portfolio assessment support in the future. However, a formal method specialized for describing the relations between the elements has not been found. We have therefore aimed at developing an original method for describing the relations between the elements. The framework to describe PDS and PPS precisely and consistently can be acquired by further developing our formal method of description. Therefore, a solution to Requirement (c) can be expected. Definitions Here, we provide definitions for PDS and PPS as follows. Definition 1 PDS is the relation between the set of lesson forms (LF) and the set of collected portfolios (CP). Therefore:

LF CP.

Here, LF is:

LF = LT × PT × PR.

And, CP is the subset of all portfolios:

CP = WP ∪ PP ∪ AP ∪ OP.

LT is the set of lesson styles, PT is the set of persons who assign tasks, PR is the set of persons who create rubrics, WP is the set of work portfolios, PP is the set of process portfolios, AP is the set of assessment portfolios, and OP is the set of other portfolios. The relations between details of lesson forms and the kinds of collected portfolios are:

[lt , pt , pr ] [ wp, pp, ap, op] Here, lt ∈ LT , pt ∈ PT , pr ∈ PR, wp ⊆ WP, pp ⊆ PP, ap ⊆ AP, op ⊆ OP

//

Definition 2 PPS is the relation between the set of practices (UP) and the set of collected portfolios (CP). Therefore:

UP CP.

Here, UP is:

UP = PN × UU .

And, CP is the subset of all portfolios:

CP = WP ∪ PP ∪ AP ∪ OP.

PN is the set of practice names, UU is the set of users, WP is the set of work portfolios, PP is the set of process portfolios, AP is the set of assessment portfolios, and OP is the set of other portfolios. The relations between details of practices and the kinds of collected portfolios can be written as:

[ pn, uu ] [ wp, pp, ap, op ] Here, pn ∈ PN , uu ∈ UU , wp ⊆ WP, pp ⊆ PP, ap ⊆ AP, op ⊆ OP

//

Notation This section proposes a notation based on the definitions in the previous section. Tables 1 and 2 list the PDS and PPS we extracted. Therefore, the developed notation needs to satisfy the following two Requirements. 94

i.

To be able to freely add and delete items that comprise the table (i.e., lesson form, collected portfolios, and others), and rewrite the relations. ii. To be able to precisely and clearly describe which computer systems read the text written by the notation and work systematically while checking the correspondence to the description in the text. The notation consists of three parts: declaring symbols (Symbol)，showing the structure of relations (Structure), and describing relations (Relations). Figure 3 shows a concrete example, which describes part of the PDS in Table 1 with the notation.

#Symbol: Symbol LT=”Lesson type”; PT=”Person who sets tasks”; PR=”Person who creates rubrics”; AP=”Assessment portfolio”; WP=”Work portfolio”; PP= ”Process portfolio”; OP=”Other portfolio”; lec=”Lecture”; q&a=”Question and answer”; exe=”Exercise”; skl=”Skill acquisition”; cre=”Creation”; dis=”Discussion”; sem=”Seminar”; eob=”Experiment or observation”; exp=”Experience”; plb=”Problem solving”; tcr=”Teacher”; lnr=”Learner”; clb=”Collaboration by teacher and learner”; rbc=”Rubric”; sfa=”Self-assessment”; ota=”Other-assessment”; ref=”Reflection”; adv=”Advice”; sam=”Sample based”; org=”Original based”; con=”Conclusion”; imp=”Impression”; lnt=”Lesson note”; ont=”Other Note”; col=”Collection”; tst=”Test”; prt=”Presentation”; pra=”Practical ability”; sit=”Situation-record”; prd=”Premeditated-record”; cnv=”Conversation-record”; anc =”Anecdotal-record”; pcf=”Portfolio conference”; lpl=”Learning plan”; lmt=”Learning material”; #Sturcture: [LT,PT,PR] [AP,WP,PP,OP] LT={lec,q&a,exe,skl,cre,dis,sem,eob,exp,plb}; PT={tcr,lnr}; PR={tcr,lnr,clb}; AP={rbc,sfa,ota,ref,adv}; WP={sam,org,con,imp,lnt,ont,col,tst,prt,pra}; PP={sit,prd,cnv,anc}; OP={pcf,lpl,lmt};

Structure

#Relations: [lec,tcr,tcr] [q&a,tcr,tcr] [exe,tcr,tcr] [exe,tcr,clb] [skl,tcr,tcr] [skl,tcr,clb] [exp,tcr,tcr] [exp,tcr,clb] [exp,lnr,clb] [exp,lnr,lnr] [cre,tcr,tcr] [cre,tcr,clb] [cre,lnr,clb] [cre,lnr,lnr]

Relations



[{rbc,sfa,ref}, {lnt,tst}, {sit,anc}, {pcf,lmt}]; [{rbc,sfa,ref}, {lnt,ont,col,tst}, {sit,cnv,anc}, {pcf,lmt}]; [{rbc,sfa,ref}, {ont,tst}, {sit,anc}, {pcf,lmt}]; [{rbc,sfa,ref}, {ont,tst}, {sit,anc}, {pcf,lmt}]; [{rbc,sfa,ref}, {sam,lnt,col}, {sit,anc}, {pcf,lmt}]; [{rbc,sfa,ota,ref,adv}, {sam,org,lnt,col}, {sit,anc}, {pcf,lmt}]; [{rbc,sfa,ref}, {prt,pra}, {sit,anc}, {pcf,lmt}]; [{rbc,sfa,ota,ref,adv}, {col,prt,pra}, {sit,prd,anc}, {pcf,lpl,lmt}]; [{rbc,sfa,ota,ref,adv}, {col,prt,pra}, {prd,anc}, {pcf,lpl,lmt}]; [{rbc,sfa,ota,ref,adv}, {col,prt,pra}, {prd,anc}, {pcf,lpl,lmt}]; [{rbc,sfa,ref}, {sam,lnt,col}, {sit,anc}, {pcf,lmt}]; [{rbc,sfa,ota,ref,adv}, {sam,org,lnt,col}, {sit,prd,anc}, {pcf,lpl,lmt}]; [{rbc,sfa,ota,ref,adv}, {sam,org,col}, {prd,anc}, {pcf,lpl,lmt}]; [{rbc,sfa,ota,ref,adv}, {org,col}, {prd,anc}, {pcf,lpl,lmt}];

Figure 3. Description of PDS with notation For example, when a teacher designs the assessment of portfolios in lesson form that consist of “Lesson Style: Creation”, “Setting Tasks: Teacher”, and “Rubric Creation: Collaboration by Teacher and Learner”, [cre,tcr,clb][{rbc,sfa,ota,ref,adv},{sam,org,lnt,col},{sit,prd,anc},{pcf,lpl,lmt}], which is a rule of notation that specifies that the portfolios that need to be collected are “Rubric”, “Selfassessment”, “Other-assessment”, “Reflection”, “Advice”, “Production: Sample based”, “Production: Original based”, “Note: Lesson Note”, “Collection”, “Learning-record: Situation-record”, “Learning-record: Premeditated-record”, “Anecdotal-record”, “Portfolio Conference”, “Learning Plan”, and “Learning Material” in Table 1.

95

Although Figure 3 is a description corresponding to Table 1, the structure of the table can be changed and the relations can also be changed individually. In other words, even if relations are changed and added, the notation can still be expressed. PPS can be described like PDS. Figure 4 describes part of the PPS in Table 2 with the notation.

#Symbol: Symbol PN=”Practice name”; UU=”User”; AP=”Assessment portfolio”; WP=”Work portfolio”; PP=”Process portfolio”; OP=”Other portfolio”; bro=”Browsing”; prsa=”Self-assessment”; proa=”Other-assessment”; prrf=”Reflection”; prad=”Advice” rgt=”Registration of work portfolios”; slt=”Selection of collected portfolios”; prec=”Process Recording”; rcre=”Rubric Creation”; prcf=”Portfolio Conference”; tcr=”Teacher”; lnr=”Learner”; otr=”Other”; rbc=”Rubric”; sfa=”Self-assessment”; ota=”Other-assessment”; ref=”Reflection”; adv=”Advice”; sam=”Sample based”; org=”Original based”; con=”Conclusion”; imp=”Impression”; lnt=”Lesson note”; ont=”Other Note”; col=”Collection”; tst=”Test”; prt=”Presentation”; pra=”Practical ability”; sit=”Situation-record”; prd=”Premeditated-record”; cnv=”Conversation-record”; anc=”Anecdotal-record”; pcf=”Portfolio conference”; lpl=”Learning plan”; lmt=”Learning material”; #Sturcture: [PN,UU] [AP,WP,PP,OP] PN={bro,prsa,proa,prrf,prad,rgt,slt,prec,rcre,prcf}; UU={tcr,lnr,otr}; AP={rbc,sfa,ota,ref,adv}; WP={sam,org,con,imp,lnt,ont,col,tst,prt,pra}; PP={sit,prd,cnv,anc}; OP={pcf,lpl,lmt};

Structure

#Relations: Relations [bro,lnr][{rbc,sfa,ota,ref,adv},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv},{pcf,lpl,lmt }]; [bro,tcr][{rbc,sfa,ota,ref,adv},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv,anc},{pcf,lpl,lmt}]; [bro,otr][{rbc,sfa,ota,ref,adv},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv},{pcf,lpl,lmt}]; [prsa,lnr][{rbc,sfa,ota,ref,adv},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv},{lpl}]; [pros,tcr][{rbc,sfa,ota,ref,adv},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv,anc},{lpl}]; [pros,otr][{rbc,ota},{sam,org,com,imp,lnt,ont,col,tst,prt,pra}, {sit,prd,cnv},{}]; [prrf,lnr][{ref},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv},{}]; [prad,tcr][{adv},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv,anc},{}]; [prad,otr][{adv},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv},{}]; [slt,lnr][{},{sam,org,com,imp,lnt,ont,col,tst,prt,pra},{sit,prd,cnv},{}];

Figure 4. Description of PPS with notation As previously discussed, we developed a framework that formally describes PDS and PPS. As the framework can be used to describe the relations between the lesson forms and portfolios that need to be collected and the relations between a user’s practices and collected portfolios that a user needs to carry out practices, Requirements (a) and (b) are satisfied. The system can systematically analyze relations by reading text described with the notation. As a result, we can also expect a solution to Requirement (c).

Portfolio Assessment Support System based on formal method of description Outline of system This section discusses our development of the Portfolio Assessment Support System, which we called PASS, based on the formal method of description. Figure 5 shows the system configuration. It consists of four subsystems: semantics analysis, design support, practice support, and DB management. Furthermore, each consists of various modules. We used Perl as the development language for the system, and also XMLDB. 96

¾ ¾

¾

¾

Semantics analysis The subsystem consists of PDS and PPS analysis modules. Each module checks PDS or PPS semantics. PDS and PPS are prepared as a text file beforehand. Design support The subsystem supports teachers’ portfolio-assessment designs through cooperation with the designinterface and design-control modules. After receiving the teacher’s orders from the design-interface module, the design-control module requests a check of the relations from the semantics analysis subsystem and receives the results. The design-interface module generates dynamically adaptive interfaces according to the results and provides them to the teacher. Practice support The subsystem supports the user’s practice through cooperation with the practice-interface and practicecontrol modules. After receiving the user’s orders from the practice-interface module, the practicecontrol module requests a check of the relations from the semantics analysis subsystem and receives the results. The practice-interface module generates dynamically adaptive interfaces according to the results, and provides them to the user. DB management The subsystem consists of a DB control module and Portfolio DB, and reads, writes, updates, and stores portfolios.

Figure 5. System configuration Practical example In the design phase, the teacher decides the lesson style first to support his or her portfolio-assessment design (left screen of Figure 6). He or she then decides who sets tasks and who creates rubrics (right screen). The right screen in Figure 6 shows the interface that the system generated dynamically based on PDS according to the teacher’s decisions. This is the case of a lesson form that consists of “Lesson Style: Creation”, “Setting Tasks: Teacher” and “Rubric Creation: Collaboration by Teacher and Learner”. The system then determines necessary portfolios based on PDS according to the lesson form, generates the adaptive interface, and provides them to the teacher. Thus, the system supports portfolio-assessment designs that adjust to the teacher’s intended lesson. In the practice phase, when the learner clicks the “Self Assessment” button in the menu, the system checks PPS and provides him or her with collected portfolios that he or she needs for self-assessment (Figure 7). The learner can therefore self-assess him or herself efficiently with the collected portfolios.

97

Figure 6. Screen of teacher’s portfolio-assessment design

Figure7. Screen of user’s portfolio-assessment practice We developed a system that works systematically based on PDS and PPS, which satisfied Requirement (c). We also developed PASS, which satisfied Requirements (a), (b), and (c). Therefore, Problems (1) and (2) were solved.

Conclusion We extracted and clarified the relations between lesson forms and portfolios that needed to be collected, i.e., Portfolio assessment Design Semantics (PDS) and the relations between practices and collected portfolios, i.e., Portfolio assessment Practice Semantics (PPS). We also developed a formal method of describing all relations between PDS and PPS. Moreover, we developed a portfolio assessment support system (PASS) based on the formal description method, which can precisely and consistently describe PDS and PPS. The system makes it possible to assess portfolios according to the lesson forms and practices based on the formal description method. Reusing past lessons has made it possible to design a more efficient way of portfolio assessment by storing past lessons for which portfolios have been assessed as case studies based on the formal description framework. There are specifications related to assessment. They are QTI (2005) and ePortfolio (2005). QTI (2005) is a specification that enables question and tests to be exchanged, and ePortfolio (2005) is a specification that can 98

allow comparability of portfolio data across organizations and interoperability of applications with other systems. However, it is impossible to achieve support for portfolio assessment at the design and practice phases by only using these specifications. Therefore, we expect that the formal description method we developed will become useful as a standard for portfolio assessment, and consider that its integration with QTI (2005), ePortfolio (2005), and other e-learning standards (e.g., Learning Design, 2003; SCORM2004, 2004) will make it even more efficient. We intend to use the system in actual lessons and evaluate it in the near future.

References Barton, J., & Collins, A. (1996). Portfolio Assessment, New Jersey, USA: Dale Seymour Publications. Bruke, K. (1992). The Mindful School: How to Assess Authentic Learning Third Edition, Illinois, USA: IRI/SkyLight Training and Publishing Inc. Chang, C. (2001). Construction and Evaluation of a Web-Based Learning Portfolio System: An Electric Assessment Tool, Innovations in Education and Teaching International, 38 (2), 144-155. Chen, G., Liu, C., Ou, K., & Lin, M. (2000). Web Learning Portfolios: A Tool For Supporting Performance Awareness. Innovations in Education and Teaching International, 38 (1), 19-30. ePortfolio (2005). IMS ePortfolio Specification, IMS ePortfolio Best Practice Guide, IMS ePortfolio Binding, IMS Information Model, IMS Rubric Specification. Version 1.0 Final Specification, retrieved February 28, 2006 from http://www.imsglobal.org/ep/. Fukunaga, H., Nagase, H., & Shoji, K. (2001). Development of a Digital Portfolio System That Assists Reflection and Its Application to Learning, Japan Journal of Educational Technologies, 25 (Suppl.), 83-88. Hart, D. (1993). Authentic Assessment: A Handbook for Educators, New Jersey, USA: Dale Seymour. Learning Design (2003). IMS Learning Design. Information Model, Best Practice and Implementation Guide, XML Binding, Schemas. Version 1.0 Final Specification, retrieved February 28, 2006 from http://www.imsglobal.org/learningdesign/. Mochizuki, T., Kominato, K., Kitagawa, T., Nagaoka, K., & Kato, H. (2003). Trends and Application of Portfolio Assessment for e-Learning. Media and Education, 10, 25- 37. Morimoto, Y., Ueno, M., Takahashi, M., Yokoyama, S., & Miyadera, Y. (2005). Modeling Language for Supporting Portfolio Assessment. Proceedings of the 5th IEEE International Conference on Advanced Learning Technologies, Los Alamitos, CA: IEEE Computer Society, 608-612. Morimoto, Y., Ueno, M., Yonezawa, N., Yokoyama, S., & Miyadera, Y. (2004). Meta-Language for Portfolio Assessment, Proc. The 4th IEEE International Conference on Advanced Learning Technologies, Los Alamitos, CA: IEEE Computer Society, 46-50. QTI (2005). IMS Question and Test Interoperability Information Model, Overview, Information Model, XML Binding, Implementation Guide, Conformance Guide, Integration Guide, Meta-data and Usage Data, Migration Guide. Version 2.0 Final Specification, retrieved February 28, 2006 from http://www.imsglobal.org/question/index.html#version2.0. Puckett, M. B., & Black, J. K. (1999). Authentic Assessment of the Young Child: Celebrating Development and Learning (2nd Ed.), New Jersey, USA: Prentice Hall. SCORM2004 (2004). Sharable Content Object Reference Model: SCORM 2nd Edition, Overview, SCORM RunTime Environment Version 1.3.1, SCORM Content Aggregation Model Version 1.3.1, SCORM Sequencing and Navigation Version 1.3.1, Addendum Version 1.2, Advanced Distributed Learning, retrieved December 16, 2005 from http://www.adlnet.org/scorm/index.cfm. Shores, F. E., & Grace, C. (1998). The Portfolio Book: A Step-by-step Guide for Teachers, Maryland, USA: Gryphon House. 99

Von Brevern, H. & Synytsya, K. (2006). A Systemic Activity based Approach for holistic Learning & Training Systems. Educational Technology & Society, 9 (3), 100-111.

A Systemic Activity based Approach for Holistic Learning & Training Systems Hansjörg von Brevern and Kateryna Synytsya International Research and Training Center for Information Technologies and Systems, Kiev, Ukraine Tel. + 380.44.242-3285 [email protected] [email protected] ABSTRACT Corporate environments treat work activity related to processing information and the one of learning & professional training (L&T) separately, by keeping L&T on a low priority scale, and perceiving little dependency between both activities. Yet, our preliminary analysis of an organisation has revealed that both activities mutually affect each other. Wholes or parts of corporate information and L&T must simultaneously be available in “real time”. To manage technical systems and their content from either source, we postulate an approach that embeds employees, learners, artefacts, and the object of their activity into a systemic-structural system based on activity theory. Our suggested approach is applicable to resolve key issues of work activity, professional training, and the modelling of adaptive technical system behaviours; it converges human activity and engineering, and positions the latter appropriately within the super ordinate and encompassing human activity system.

Keywords Systemic-Structural Activity Theory, Task Analysis, Adaptive Learning & Training Systems

Introduction L&T as a way to ensure up-to-date cognition and skills as well as relevance to emerging organizational needs and goals have become a characteristic feature of our time, implemented through traditional processes or technology-based environments. However, the distribution of information technologies is leading the way to heavy dependences of business processes on information processing and knowledge management that a variety of software systems support. Despite contemplated connection and interaction among information and cognition processing activities, little attention has been paid so far to modelling a holistic view to computer-supported professional activities in learning organizations. (von Brevern & Synytsya, 2005) posited the need for a shift of paradigm in L&T that form one holistic activity using the Systemic-Structural Theory of Activity (SSTA) as the underlying methodology. This paper integrates the L&T into a corporate realm that continuously seeks to ameliorate organisational performance i.e., efficiency, effectiveness, and cost-saving. L&T is no longer an isolated unit within an organisational environment but is directly and actively interwoven with corporate interests and issues. Equally, changes in work processes not only have a mutual affect on work activity and L&T but also on the goals of these activities. These facets and more challenge the structure of activities and the type of roles that Information Systems (IS) and Learning Technology Systems (LTS) play within such dynamic environment. Up to the present, research in educational technology has not yet addressed the interrelationship between IS and LTS and has largely failed to model technical solutions under dispositional viewpoints that model responsive systems to human cognitive behaviour. Instead, today we have educational technology based on positional viewpoints; i.e., mere engineering practice prevails and, in doing so, ignores the dispositional aspect caused by human behaviours. We conjecture that the accord of processes involved is the prerequisite to answer the “why’s” and only once we grasp the “why’s”, we can determine the “what’s” to solve the “how’s”. The same applies to L&T as (Savery & Duffy, 1995, p. 31) argue: “We cannot talk about what is learned separately from how it is learned, as if a variety of experiences all lead to the same understanding. Rather, what we understand is a function of the content, of the context, the activity of the learner, and perhaps, most importantly, the goals of the learner”. In this connection, the paper discusses the need for unified consideration of both professional information processing and training and suggests a systemic approach to analyze needs for training and cognitive support that may be addressed by technical systems. The goal of this research is to provide a holistic view on human activities related to information processing, learning and training, and offer psychologically grounded mechanisms for their study. It is expected that this approach facilitates revealing generic patterns in work ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

100

acivities, bottlenecks caused by information overload, typical tasks and types of content engaged, and thus determining priorities, functions and content structure for technical support systems.

Context An informal analysis of call agent’s work processes of a major Swiss mobile phone company made us realise that L&T is part of a larger context that comprises work activity and unites organisational goals, requirements, principles, etc. Corporate L&T in a learning organization has to cover a wide range of needs, such as training new workforces, updating employees on new products and services offered to customers, training on new tools used in the company, raising professional skills and educational levels, providing corporate information for smooth coordination of activities, teaching so-called “soft skills”, etc. L&T is tightly connected to organisational information and corporate knowledge structures. The diversity of information about new, altered, or obsolete products, services, promotions, etc. needed in responding to telephone enquiries makes call agents heavily rely on IS and LTS. Thus, the job of call agents requires them to spatiotemporally process information while answering customers’ queries or betrothal in problem solving. In doing so, answering to enquiries and troubleshooting is a subject of processing information whose wholes or parts do not only reside in IS but also in LTS as illustrated in Figure 1.

Figure 1. Context Diagram of Processing Information and L&T Availability and accessibility of L&T content affect the efficiency of work activity, so both IS- and LTS-residing material should be made available in “real-time”. Both systems therefore need to respond to human stimuli and behaviours by returning meaningful and significant material to the call agent under a situated viewpoint. A significant response means that it has personal sense to the call agent at a particular point of time. Complexity and informativeness (Bedny & Meister, 1997) characterise the meaningfulness of a response; the social context of the work activity rather embeds than predetermines the essence of a meaningful response. However, L&T and information processing include not only “stimuli – response” but also complex cognitive activities of humans that have to be supported by the systems as shown in Figure 1. Moreover, the activities of processing information and L&T are no longer independent but represent two mutually dependent and equally important activities in the corporate domain. Both activities share one common end goal, which in our case is optimal customer satisfaction that contributes to the corporate mission. To visualise the domination of the availability of content and learning experience, contemplate the following: At some point of time a call agent may be preoccupied with the task of a new problem while having forgotten how 101

he or she resolved a correspondent sub task earlier. This means that L&T content including decomposition of problems and lessons learnt may need to remain available and accessible at later points of time, yet changes of work processes and activities may require updates between relationships and dependencies, instantaneous availability, and may always need to reflect state-of-the-art relationships between work and L&T activities. Thus, arranging a supportive environment is to ensure that employees are always au courant with what was learnt earlier, the latest lessons learnt in customer care-taking, the latest product information, promotion details, repair processes, etc. The numbers of calls answered within a certain time measures call agents’ efficiency. The problem-solving nature of their job requires them to not only have information available instantaneously but also that systems facilitate content adaptation by “what is important to them” at a given point of time. The continuously changing nature of information requires call agents to be trained. This process is the basis of an iterative and inseparable interrelationship between processing information and L&T and requires technical support instruments enabling creation, updates and focused access to specific content. Yet, we have seen another reality. IS and LTS are separate, hardly share any content, and in doing so, do not facilitate call agent’s work processes effectively and efficiently so that processes are under the line very expensive. On top of that, IS or LTS deliver large junks of not closely related information; the output is neither structured according to the current problem nor presented in the befitting format. Hence, our call agents have learnt the lesson to rely on their own personal notes, printouts of L&T material, or the like (Figure 1). The severe result of such irrelevant and too much information is cognitive overload. Moreover, when call agents continuously face such ill-structured situations like absence or non-accessibility of appropriate information the decision-making process involved in responding to customers’ enquiries is accompanied by emotionally motivated stress. This is also why, call agents cannot sustain in their job for more than three years. So, the more complicated and indeterminate the situation, the more mental mechanisms of imagination and image manipulation are included as discussed by (Bedny & Meister, 1997). Apparently from an engineering standpoint, we would need to conduct a gap analysis between the present and desired system states to depict what it takes to have technical systems adapt to human behaviour to return meaningful, significant, and situated responses. Despite efforts made in performance support systems (Gery, 1991), today’s technical systems still cannot yet respond satisfactorily to cognitive behaviours. New requirements for technical systems like IS, LTS, and their content processing arise based on human work activities whose evolving processes we may neither have known nor could predict earlier. So, we need a methodology that in the first place helps us to decompose and formalise human work activity that “… is a coherent system of internal mental processes and external behaviour and motivation that are combined and directed to achieve goals” (Bedny & Meister, 1997, p. 1). As the past decades of experience have proved, a mere engineering based methodology that ignores mental processes will not resolve the task. The pitfall of attempting to resolve human activity by conventional engineering is that it attempts to model systems on “stimulus – response” behaviours. This approach largely ignores cognitive mechanisms of humans and with them the embedded roles, context, tasks, and goals of technical systems. Instead, the answer of how to envision, understand, and model technical systems should be explored in human activity and its manifestation. From this follows that technical systems ought to present content according to how humans process it during work and L&T activities. Ergo, we need a methodology that helps us formally analyse and decompose work and L&T activities as an embedded scrutiny, while an engineering methodology that enables us to extract identifiable (technical) system events and responses to actions from the context of the higher-order activity system is equally indispensable, followed by conventional engineering methods to ultimately model technical solutions.

Methodology The activities of processing information and L&T become meaningful when their processes are coupled with situated actions that make part of super ordinate tasks. These are a subject of the goal of activity and are altogether coupled with individual internalisation processes. With this respect, we recognise that there is no onesize-fits-all process and with it, no one-size-fits-all technical system, because activity “A” differs from activity “B” in the same way as organisations and their behaviours vary. Yet, unless the individual systematically captures corporate learning experiences, adopts, and continuously transforms new experiences and lessons, they will be lost. Organisations are therefore interested in preserving learning experiences, cognition, and information and in improving efficiency and effectiveness of manpower. Hence, technological empowerment requires us to observe, understand, and measure human activities in the first place, which in return“… requires studying the structure of activity, which is a system of actions involved in task performance that are logically organised in time and space” (Bedny & Meister, 1997, p. 31). 102

In (von Brevern & Synytsya, 2005), we have presented the Systemic-Structural Theory of Activity (SSTA), which allows us to study cognitive and social impacts that can equally be experienced by call agents and customers as well as learners and instructors during work activity that includes the mediated use of technology. With the term systemic, “… one is not here speaking of man-machine systems, rather describing human activity and behaviour as a system, which, of course, is in dynamic interaction with machinery. A system is a set of interdependent elements that is organized and mobilized around a specific purpose or goal. System analyses entail extracting the relevant elements and their dynamic interactions. Systemic analyses not only differentiate elements in terms of their functionality, but also describe their systematic interrelationship and organization. Whether or not there is a system approach depends not so much on the qualities and specificity of the object under consideration, but rather on the perspective and methods of analysis” (Bedny et al., 2000, p. 187). (Bedny & Meister, 1997) describe SSTA as a pragmatic and systemic methodology that enables us to analyse human behaviours and processes. As we have addressed in (von Brevern & Synytsya, 2005, p. 746), an activity is a “goal directed system in which cognition, behaviour and motivation are integrated and organized by goals and the mechanisms of self-regulation” (Bedny et al., 2000). SSTA presents us with a model that recognizes where “at the root of the gnostic dynamic is the self-regulation process, which provides the mental transformation, evaluation, and correction of the situation” (Bedny & Karwowski, 2004). The goal-oriented activity builds an interrelated, complex, and dynamic system between subjects, subjects and the goal-oriented object of activity, and their mediating artefacts or tools. Figure 2 presents the triadic schemata of the call agent’s work activity concerned with processing information; Figure 3 presents the triadic schemata of the L&T activity. Both triadic activity schemas share the same-targeted end result and their mediating technical tools (IS and LTS) have different distinct contextual functionalities. While IS’ functionality is concerned with processing corporate information and providing appropriate content upon call agents’ requests during the processing information, LTS’ mediate between an instructor, a learner, peers, and is aimed at facilitating L&T. In this sense, the activity system of SSTA widens and extends our understanding of a mere engineering-based approach in view of analysing and modelling the roles of technical artefacts because it incorporates cognitive processes and models the context in which the dynamic activity system lives.

Figure 2. Triadic Schemata of the Processing Information Activity System

Figure 3. Triadic Schemata of the L&T Activity System 103

SSTA builds the necessary formal grounds that allow analysis of human work activity, goals, tasks, and actions according to which technical support may be arranged. Although it is outside the scope of this paper to discuss each formal method of SSTA in-depth they are informational or cognitive, parametric, morphological, and functional. “Informational or cognitive analysis studies the cognitive processes that are concerned with task performance; Parametric analysis is quantitative and qualitative and allows evaluation of the effectiveness of work activity in terms of quality, quantity, and reliability; Morphological analysis looks at describing the logical and constructive features of activity on the basis of actions and operations; Functional analysis looks at the function blocks” (von Brevern & Synytsya, 2005, p. 747). SSTA separates the goal-driven activity into objects of study i.e., activity, tasks and units of analysis such as actions, operations, and function blocks (Bedny & Harris, 2005). For a better comprehension, Figure 4 visually illustrates the relationship between the objects of study and the units of analysis in a simplified form. Once we have identified the objects of study, the units of analysis become important to analyse human algorithmic behaviours. A human algorithm is the representation of a “…logically organized system of actions through which an individual subject transforms some initial material in accordance with the required goal of the task. These algorithms are distinguished from similar devices such as flow charts by their use of actions as the basic units of analysis” (Bedny & Harris, 2005, p. 8). It is the composite of different actions that make an activity to a complicated dynamic structure. Therein, all elements of an activity are “… tightly interconnected and influence each other. From this, the requirement of using the systemic method of study follows. As can be seen, an activity is a process. However, this process consists of a sequence of a hierarchically organised structure of elements (actions and operations)” (Bedny & Meister, 1997, p. 48). So, for us to understand and pragmatically analyze a goal-oriented activity and its elements, we must first engage in task analysis as Figure 4 demonstrates.

Figure 4. The Objects of Study and Units of Analysis of an Activity System

Task Analysis The task holds an essentially central role in the activity system. In fact, our degree of understanding of the task and precision of task analysis determine if we model technical systems correctly or not. Under such critical aspects, we will need to know the principle task characteristics, context, definition, and influences for system modelling. According to (Leont'ev, 1978), tasks set by an individual for himself or herself “… in thought originate in the social conditions of his [or her] life”. (Leont'ev, 1978) enlarges our viewpoint of the integral embodiment of a task, its subordinate actions, the situation requiring goal-achievement, and conditions. He argues that “every purpose, even one like the “reaching of point N,” is objectively accomplished 104

in a certain objective situation. For the consciousness of the subject, the goal may appear in the abstraction of this situation, but his action cannot be abstracted from it. For this reason, in spite of its intentional aspect (what must be achieved), the action also has its operational aspect (how, by what means this can be achieved), which is determined not by the goal in itself but by the objective- object conditions of its achievement. In other words, the action being carried out is adequate to the task; the task then is a goal assigned in specific circumstances” (Leont'ev, 1978). Among various definitions, a task “ … is inherently a problem-solving endeavour” and “… a set of human actions that can contribute to specific functional objective and ultimately to the output of a goal of a system” (Bedny & Meister, 1997, p. 19). When it comes to determine of how to improve organisational performance, we will need to know the requirements and conditions of the activity. With this respect, (Bedny & Meister, 1997, p. 19) claim that “Anything that is presented to the operator or known by him or her is a condition of the task, the requirements of which include the finding of a solution (or what we need to prove)”. Therefore, it is indispensable to first conduct task analysis in any activity system. Accordingly, the kinds of tasks that tasks call agents face during their daily activity of processing information have become our primary foci of interest. Nevertheless, even an informal task analysis has become a major challenge because most tasks have never been formulated precisely. The following five examples from our findings reflect the importance, breadths and widths, impacts, and values contained in task understanding: 1.

2.

3.

4.

5.

The task to understand ill-structured problems. A call agent’s job requires a high degree of sensory perceptual and imagery responses to solve situational problems. This may subjectively dominate sensory tasks. A call agent’s goals are often imprecise due to ambiguous task definitions by customers, superiors, systems that leave space for goal formation. So, systems need to help on problem identification for which IS could provide alternative parameters of problem descriptions. Additionally, call agents need training to recognize and classify customer’s input and associate it with appropriate goals. The task to do a spatiotemporal search on specific information. Irrelevant, insignificant, or too much feedback by IS cause unfavourable conditions to the call agent. Contradictions between task requirement and unfavourable conditions cause interaction breakdowns within the activity system. Task decomposition provides detailed information on causes of breakdowns, which serves as a starting point to modify content and structure of support system responses to the agent. The task to switch between tasks. The pre task instruction to a call agent is to answer the incoming call. The latter regulates a call agent’s behaviour. When training sessions are embedded in the daily work, the agent should be able to change activities, and together with them goals, tasks, and actions. Call answering and learning activities are different in nature so that we need to investigate motives and actions between task switching to determine how IS or LTS could best facilitate task switching to stimulate the motivation for each activity, to help refresh the knowledge of the last visit, etc. The task to classify information. Information distributed by IS is not personalized so it has to be sorted and prioritized by each agent. The IS may have alerted the call agent about an upcoming summer promotion when the agent will eventually be on holidays. Simultaneously of the system alert, customers are complaining about a new model for which no training has been received. So, at that very point of time the information about the summer promotion is not appropriate to the call agent. Analysis of personal interpretations – meaning and sense – can be absolutely crucial as (Bedny & Karwowski, 2004) argue. Therefore, information needs to be spatial and temporally significant, which altogether impose critical system requirements. The task to master skills consciously. For example, a routinized agent may know how to repair even the latest model of a mobile phone of a certain brand. So, motor actions that consist of automatisation may be unconscious to the call agent. (Bedny & Meister, 1997) argue that motor actions of a higher order task can disintegrate. That means that if for any reason, the call agent suddenly consciously deliberates each hierarchical motion involved in the repairing process and becomes aware of each individual goal associated with each step, the process will slow down and regress. L&T therefore needs to shape a call agent’s accord of the conditions of the task and make the call agent master it.

Apprehension of the tasks of the activity of processing information is crucial and integral to L&T because both activities share the same end-goal or result as illustrated in Figure 2 and Figure 3. We also need to understand those tasks of the L&T activity because analogous activities mutually affect each other, albeit not discussed herein. For example, the requirement for modular content is equally important to the activities of processing information and L&T and so is the notion of different zones of proximal development (ZPD) (Giest & Lompscher, 2003) as in learning & teaching (von Brevern & Synytsya, 2005). Moreover, the accord of informal tasks is difficult when we do not have an elucidate picture of the object of study. However, this deficiency is not uncommon in organisational environments. Therefore, if we want to ameliorate organisational performance we need to know the precise foci on the objects of study. Its tasks are then the subject of task decomposition. 105

Task Decomposition Task decomposition is the next step in the study of the algorithmic description of the call agent’s activity. Following our preliminary cognitive analysis of the task, we classify actions. “The purpose of classifying actions is to present an activity as a structure with a systemic organisation. All actions are organised as a system because of the existence of a general goal of activity. Any changes in individual action can immediately influence other actions. A person continually evaluates his or her own activity from the viewpoint of the goal and conditions under which he or she performs the activity” (Bedny & Meister, 1997, p. 32). In our study, we have discovered indeterminate tasks in which the goal has been ill-defined and the method of implementation partially unknown or ambiguous. Now, let us contemplate an example and demonstrate of how to decompose a task into subtasks. We can use subtasks during the decomposition of activity and build a hierarchical order of the units of analysis (Fig. 4). “In the algorithmic description of task, a subtask is … composed of one to three homogeneous, tightly connected actions” (Bedny & Meister, 1997, p. 32). Now, suppose the following simple case: “If a customer’s promotion name is X, check in the IS the promotion procedure and inform the customer. If the promotion name is Y, transfer the call to department Z” (where X, Y = getPromotion (promotionName:character, promotionCode:integer):character{1 character}; Z = department():numeric{2 digits}). Table 1 presents the method of decomposition; i.e., an informal algorithmic description of the task of how a semi-routinized call agent solves the task, the hierarchical order of the units of analysis, duration of time, and a classification of actions. Albeit the semantic task description emanates to be simple, this task leaves an indeterminate situation that bestows space for options. (Bedny & Meister, 1997, p. 22) argue “When task instructions do not contain a sufficiently detailed description of how to achieve the goal and consequently the task requires conscious deliberation about how to accomplish that goal, the task becomes one of a problem solving.” Based on the task decomposition of Table 1, the following addresses some of our findings: 1.

Task dependency. Our task presumes only two promotions, which is unlike. Hence, our task requires dependency on another as (Bedny & Meister, 1997, p. 23) claim: “An aspect of complexity derived from engineering theory is the degree of dependency of one task and action on another”. When we change the task, it re-affects the algorithmic description. Dependency between actions. Changes of the algorithmic description of tasks affect the structure between actions. 2. Figure 5 illustrates a concept map of the structure of mental actions (based on Bedny & Meister, 1997) which is appended to illustrate combination of motor and mental actions in the activities of processing information and L&T. Notwithstanding, the object of study can have its main focus on mental actions, analysis of algorithmic behaviours cannot ignore either kinds of actions neither can we do so when we aim at improving organisational performance. 3. Taxonomy of standard activity and duration of time. Except for the structural effect, it is important to regard the factor time because any actions have their duration of time. Time is an important indicator to detect issues. With reference to effectiveness of skilful work activity (Bedny & Meister, 1997, p. 27) suggest to deploy a taxonomy. 4. System responses to mental and motor actions. Owned to the problem space of the task, task decomposition of Table 1, and the, at this stage unknown object of study create open options for both mental and motor actions. For example, the task does not include any validation subtask in view of misspelling, which includes motor and mental actions. For example, misspellings could be the cause of a simple typing error or fallacy by the customer or the call agent. Then it would be helpful if IS could facilitate resolving the case. Though no technical system is able to infere the cause or motive, we may wish to model technical systems, which “foresee” such kinds of events. Neither can we know such a need nor can we model such technical systems unless we first decompose the task. 5. Interaction breakdowns. “In systemic-structural terms, a breakdown can be defined as forced changes of subjects’ strategies of action caused by their evaluation of an unacceptable divergence between the actual results of actions and the conscious goals of those actions. Typically, breakdowns are characterized by subjects’ (temporary or permanent) abandonment of the task in hand, which … may lead them to reformulate the task-goal” (Harris, 2004, p. 4-5). Contradictions between task requirement and unfavourable conditions cause interaction breakdowns within the activity system. Although our task does not require the call agent to ask the customer for the name of promotion and the promotion code, the call agent has wisely done so in advance (Subtask 1) out of previous experience. The most severe interaction breakdown, however, is the requirement to inform the customer (see Table 1 “average time” of subtask 5) because stored information in IS is not modular. Apprehending interaction breakdowns and actions are extremely crucial when it comes to goal-formation, task-fulfilment, actions involved, and the efficiency of mediating tools. 106

One of the critical aspects of interaction breakdowns is the possible dilemma between the objective goal and the subjective goal of the task because of its affect onto operations: “One of the critical aspects of action is the formulation of a conscious goal and an associated capacity to regulate progress toward achievement of the goal voluntarily.” (Bedny et al., 2000, p. 174). Table 1. Decomposition of Activity during Task Performance

Figure 5. Concept Map of Mental Actions 107

Preliminary task decomposition builds the ground for the objects of study to depict desired system requirements. It provides guidelines for optimization of operator’s work and is suitable in the design of human-computer interactions, including communication mode, visualization features, and modularization of content. Herein, we have not formally demonstrated the stages and their fragmentation in space (motor and mental activity) and time as the formal methods of SSTA allow us to do so because such is outside the scope of this paper. Task decomposition reveals preciseness of the semantic task formulation. Exactitude and preciseness in task description are integral prerequisites to not only analyse or model the activity system but also in view of optimising technical artefacts and work activity itself.

Subject Domain After obtaining information from detailed analysis of call agent’s activity, we have a picture of current ways of operation and from that we can identify desired activities and certain requirements for processes and technical systems. At this point, findings related to cognitive activity should be incorporated into engineering description of the support system, so that we need an engineering method that is capable to extract events, entities, state transitions, and the like to construct technical systems. Thus, we turn to the description of subject domain, which is “… the part of the world that the messages received and sent by the system are about” (Wieringa, 2003, p. 16). The subject domain separates identifiable entities and events from non-identifiable ones (e.g., thinking actions, motives) and lets us recognise and organise identifiable responses and non-identifiable stimuli. In liaison with our previous discussion, we posit to classify messages and events according to the object of activity and the relationships between subjects and mediating tools. Thus, under the umbrella of this notion and discussed by (von Brevern, 2004), we can bring the higher-order human activity systems of Figure 2 and Figure 3 into the context of technical system engineering because of their shared end-goal or result. It is important to order the desired levels system functionality. We can do so by integrating functionality into e.g., a mission statement, function refinement trees, and service descriptions. In the same way that we have decomposed the task, we decompose a technical system into subject-orientation decomposition, functional decomposition, communication-oriented decomposition, and behaviour-oriented decomposition (Wieringa, 2003). This allows us to specify the desired system properties and functional requirements that truly originate from the activity system. So, technical systems acquire a dispositional stance with an activity system regardless whether they hold a mediating role or are the actual object of activity. Observation and analysis of operator’s activity provide information that may enhance learning and training in multiple ways: 1. Novice training program may be focused on most common tasks, most complex tasks and training priority and sequences in arranging training modules may be aligned with current company priorities and current state. 2. When a company uses blended learning, the learning objectives persuaded in face-to-face and technologysupported learning may be better balanced, and stable information and knowledge building, a framework of the activity, may be separated from dynamic information. 3. Training may be further individualized employing the concept of ZPD (Vygotsky, 1930) to maximize effect of online training by presenting learning content relevant to currently performed tasks and preparing for more complicated situations. 4. Presentation of material, both for learning purposes and information processing, is a subject of facilitating each core element of an action that is orientation, execution, and control. A number of principles, such as the split-attention, spatial contiguity, temporal contiguity, modality, redundancy, and coherence as delineated by (Mayer & Moreno, 2003) may be employed to derive rules for content delivery. 5. In online courses focused on education and learning as opposed to just-in-time training, organisational aspects contain generic and routine frameworks that are of general interest like course overview, description objectives, terms and conditions, etc. (Horton, 2000). These should be aligned with corporate norms, culture and vision. These and other aspects may be depicted in the description of the L&T processes that are to be implemented. The key issue at this stage is bridging cognition-grounded analysis with sound engineering synthesis leading to technological implementation. For this purpose Educational Modeling Language as suggested in IMS Learning Design specification (http://www.imsglobal.org/learningdesign/index.html) may be a valuable tool, as it is pedagogically neutral and equally appropriate to describe technology-based and traditional learning, both 108

individually and in groups. Moreover, through the concept of Support Activity it might be possible to describe information support services to connect their results directly to the L&T.

Technical Considerations A major requirement for technical support systems derived from the task analysis is provisioning of relevant information and avoidance of cognitive overload. This can be achieved by understanding what kind of task an agent is working on, the current situation in his/her problem solving, and providing content focused on the current needs. To be able to do so, the system should operate with modular content which can be adapted to the user needs, assembled from parts or selected according to request. So, the underlying base to enable adaptation to organised and well-structured information is decomposition of material. Practice has shown that the issue of granularity cannot be generalised but depends on individual and customised needs of use. The two aspects that fall into place here is technical efficiency and, when it comes to L&T, psychological and didactic orientation according to the method of L&T. Modular content may be re-arranged in a way to address particular needs of the user (learner) and specific “modules” – content blocks – may be reused across LTS and IS. However, the bottom line of the granularity is determined by our ability to describe and discern similar blocks. For this purpose metadata are designed that provide multi-facet description which facilitates indexing, management, search, and exchange of the described objects. Based on these descriptions, content structure and relations between its components may be revealed, conditions for their use may be set up, and various sequencing rules created. Moreover, a structured and well-organised presentation of content equally requires us to abduct generic, continuously recurring behaviours and rules. Alexander Patterns allow us to do so (Alexander, 1979). Taken from instructional design principles, norms and standards, generic and routine frameworks, logical dependencies between learning objects and the like we can derive generic rules and behaviours. At some point we could arrive at certain frames – abstract structures with didactical guidelines that are filled in with various specific information blocks (designed according to modularity principle) depending on the learning context. Resembling the method(s) of L&T (Gal'perin, 1969)’s work on the stages in the development of mental acts has revealed that learning includes various psychic stages in the process from “understanding” to “mastery”. (Gal'perin, 1969)’s reference points adverts to those informational parts of an activity that is required for an affirmative regulation of actions and for faultless performance. If the learner is capable to present the required reference points in the new material, it will permit affirmative execution of the entire task by moving from one point to another. Hypothetically, reference points could also form a type of hierarchical arrangement of parameters during resolving ill-structured problem descriptions in processing information and L&T activities. In a broader sense, reference points can be any type of organised and well-structured information as a whole or part that we need in the analysis of an objective task or condition. There are a number of engineering issues related to static and dynamic content management, however, we would like to stress importance of taking human user viewpoint when designing technical solutions. This step has already been done – or declared – in LTS. The next might be a holistic view on user activity in complex humancomputer systems comprising various functions, modes and services. Growth of processing speed and storage in technical systems does not automatically make them more convenient to the users thus support systems design require closer attention to their usability and usefulness for human users.

Conclusion Organisations are continuously striving to aim at ameliorating work efficiency, effectiveness, and saving costs. While corporate information is constantly growing, employees are challenged to manage it, retrieving adequate portions under stress and in a limited timeframe. Unfortunately, present technical systems, regardless of whether they are IS or LTS, are not sufficiently intelligent to support information management. They cannot effectively respond to human cognitive actions and return large junks of information that cause cognitive overload and stress, which ultimately leads to employees’ work dissatisfaction. While conventional system engineering methods cannot cope with these issues – not even when it comes to understanding requirements that originate from cognitive behaviours, it is obvious we need another methodology that tackles today’s dilemma at their very root; i.e. human activity and converges then with conventional system engineering methods. The very essence of this paper is that we change the conventional viewpoint that separates and isolates L&T from work activity. In the same manner, we diverge from common custom that models technical systems solely 109

on mere engineering principles. Instead, we subordinate technical systems to the psychological level of human activity. The theoretical methodology that allows us to do so is SSTA, which we have presented herein. Taken from the ambitious corporate end goal or result that we investigated under the roof of the activity system, we have discovered that the activities of processing information and the one of L&T are adjacent, which has made us realize that we must regard work activity and the one of L&T as one universe. Also, based on diverse corporate cultures and social contexts, work and L&T activities are flavoured with cultural and social facets. In such rich contexts, it is of no surprise that “one-size-fits-all kinds” of technical system have a difficult stance to survive because they are not adaptive to human cognitive behaviours. The dynamic nature of human activity requires systemic analysis to differentiate elements related to functionality, interrelationship, and organisation. Albeit it would be outside the scope of this paper to discuss and practically demonstrate the formal methods of SSTA, we have adverted to the foremast step to understand an activity; i.e. task analysis of the object of study. Task analysis is undoubtedly critical because the task of an activity is the placeholder of contextual requirements. From experience, an informal task analysis has proved difficult when the foci of the object of study are uncertain. Once tasks are known, task decomposition is powerful because it reveals the structure and validity of the (usually semantic) task description, which is relevant for efficiency and effectiveness, its first stages of human algorithmic behaviours, possible interaction breakdowns, its temporal aspects, and more. Except for numerous other findings, task analysis and decomposition demonstrated importance of content modularity. So, although SSTA requires high expertise, this approach provides most precise and accurate formal methods to analyze, measure, and optimise work activity even to the degree to depict motivational components and self-regulation. Insomuch, SSTA is applicable in a wide range of domains. Moreover, this approach can be smoothly converged with engineering methods, as shown in discussion of the subject domain and other technical issues concerning modularity of content. Our next research steps include formal analysis according to the methods of SSTA; e.g., morphological analysis, concise requirements identification abstracted from formal aspects of task analysis and algorithmic behaviours, modelling of the subject domain, and system decomposition. We are also looking for practical examples that may guide creation of implementation framework. It is our human processes and the methods for L&T that determine and impose the stringent system behaviours for we need IS and LTS that are adaptive in response to tasks and human derivative actions. In conclusion, we would like to transform (Bedny & Meister, 1997)‘s original statement into: “Formalization of procedures of description, their standardization, and the convergence of psychological methods with those developed in engineering and computer science, provide greater opportunities for the use of psychology in L&T technology”.

References Alexander, C. (1979). The Timeless Way of Building, New York: Oxford University Press. Bedny, G., & Karwowski, W. (2004). Meaning and sense in activity theory and their role in the study of human performance. International Journal of Ergonomics and Human Factors, 26 (2), 121-140. Bedny, G., & Meister, D. (1997). The Russian Theory of Activity: Current Applications to Design and Learning, Mahwah: Lawrence Erlbaum. Bedny, G. Z., & Harris, S. R. (2005). The Systemic-Structural Theory of Activity: Applications to the Study of Human Work. Mind, Culture and Activity, 12 (2), 1-19. Bedny, G. Z., Seglin, M. H., & Meister, D. (2000). Activity theory: history, research and application. Theoretical Issues in Ergonomics Science, 1 (2), 168-206. Gal'perin, P. Y. (1969). Stages in the Development of Mental Acts. In Cole, M. & Maltzman, I. (Eds.), A Handbook of Contemporary Soviet Psychology (1st Ed.), New York: Basic Books, 249-273. Gery, G. (1991). Electronic Performance Support Systems: How and Why to Remake the Workplace through the Strategic Application of Technology, Boston: Weingarten Publications. 110

Giest, H., & Lompscher, J. (2003). Formation of Learning Activity and Theoretical Thinking in Science Teaching. In Kozulin, A., Gindis, B., Ageyev, V. & Miller, S. M. (Eds.), Vyogotsky's Educational Theory in Cultural Context, Cambridge: Cambridge University Press, 267-288. Harris, S. R. (2004). Systemic-Structural Activity Analysis of HCI Video Data. In Bertelsen, O. W., Korpela, M. & Mursu, A. (Eds.), Proceedings of the ATIT2004: First International Workshop on Activity Theory Based Practical Methods for IT Design, Copenhagen, Denmark: Aarhus University Press, 48-63. Horton, W. (2000). Designing Web-Based Training, New York: John Wiley. Leont'ev, A. N. (1978). Activity, Consciousness, and Personality, Englewood Cliffs: Prentice Hall. Mayer, R. E., & Moreno, R. (2003). Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational Psychologist, 38 (1), 43-25. Savery, J. R., & Duffy, T. M. (1995). Problem Based Learning: An instructional model and its constructivist framework. Educational Technology, 35 (5), 31-38. von Brevern, H. (2004). Cognitive and Logical Rationales for e-Learning Objects. Journal of Educational Technology & Society, 7 (4), 2-25. von Brevern, H., & Synytsya, K. (2005). Systemic-Structural Theory of Activity: A Model for Holistic Learning Technology Systems. In Goodyear, P., Sampson, D. G., Yang, D. J.-T., Kinshuk, Okamoto, T., Hartley, R. & Chen, N.-S. (Eds.), Proceedings of the 5th IEEE International Conference on Advanced Learning Technologies (ICALT 2005), Los Alamitos, CA: IEEE Computer Society Press, 745-749. Vygotsky, L. S. (1930). Mind in Society (Blunden, A. & Schmolze, N., Trans.), Harvard University Press. Wieringa, R. J. (2003). Design Methods for Reactive Systems: Yourdon, Statemate, and the UML, San Francisco: Morgan Kaufmann.

111

Cutshall, R., Changchit, C., & Elwood, S. (2006). Campus Laptops: What Logistical and Technological Factors are Perceived Critical? Educational Technology & Society, 9 (3), 112-121.

Campus Laptops: What Logistical and Technological Factors are Perceived Critical? Robert Cutshall College of Business, Texas A&M University – Corpus Christi, TX 78412 USA Tel: +1 361-825-2665 Fax: +1 361-825-5609 [email protected]

Chuleeporn Changchit College of Business, Texas A&M University – Corpus Christi, TX 78412 USA Tel: +1 361-825-5832 Fax: +1 361-825-5609 [email protected]

Susan Elwood College of Education, Texas A&M University – Corpus Christi, TX 78412 USA Tel: +1 361-825-2407 Fax: +1 361-825-6076 [email protected] ABSTRACT This study examined university students’ perceptions about a required laptop program. For higher education, providing experiences with computer tools tends to be one of the prerequisites to professional success because employers value extensive experience with information technology. Several universities are initiating laptop programs where all students are required to purchase laptop computers. The success of these laptop programs rely heavily on the extent to which the laptop environment is accepted and wholeheartedly implemented by students and faculty. Defining the logistical and technological factors necessary to successfully implement a laptop program becomes a critical issue to the success of the program. By understanding what logistical and technological factors are critical to students, such a program can be made more useful to students as well as more beneficial to universities.

Keywords Portable computing initiative, Technology in higher education, Critical success factors, Laptop initiative

Introduction Students’ use of laptop computers is becoming more prevalent in today’s universities. Previous research has shown that laptop computers in the classroom can lead to positive educational outcomes (Finn & Inman, 2004; Gottfried & McFeely, 1998; Varvel & Thurston, 2002). This more ubiquitous use of technology has caused several universities to uncover and manage new perceptual issues in addition to some of the more familiar issues from the era of computers found only in university labs. University students’ views are paramount to newer perceptual issues regarding laptop initiatives. Weiser (1998) envisioned a third wave of ubiquitous computing where computer use is fully and seamlessly integrated into student life. According to the studies conducted by Finn and Inman (2004) and Lim (1999), the results reveal that the majority of University laptop program alumni agree that portable computers were beneficial during their college careers. They also agree that it is important to continue the laptop program for future students. Positive student-body responses encourage universities to continue their efforts in laptop program implementation. Such efforts to integrate technology, however, take much support and careful planning towards effective implementation. Vital to implementing a successful laptop initiative are examining successful initiatives and uncovering students’ perceptions of critical success factors towards a laptop initiative.

The Basis of Successful Initiatives Prior studies indicate that the success of computer use is largely dependent upon the attitudes of both instructors and students (Al-Khaldi & Al-Jabri, 1998; Liaw, 2002). A study reported that 77 percent of the variance of ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

112

intent to use information technology was attributed to users’ attitudes toward computers (Hebert & Benbasat, 1994; Liaw, 2002). No matter how capable the technology, its effective implementation depends upon users having positive attitudes towards the technology (Liaw, 2002). Experience with technology influences users’ attitudes toward computer technology (Rucker & Reynolds, 2002). Those who have had positive experiences with technology tend to develop positive attitudes towards it. Prior attitudes of educational technology use or lack thereof could impact present and future perceptions of technology (Steel & Hudson, 2001). An exemplary university that consistently proves students’ positive attitudes towards a laptop initiative is the University of Minnesota (UMC) – Crookston. UMC has the oldest successfully established university-wide laptop program (Lim, 1999). Annual evaluative studies have shown that UMC students were highly satisfied with the laptop initiative. This study concluded that the high percentage of satisfaction with computer training is likely to contribute toward a high usage of computers for learning.

Uncovering Students’ Perceived Critical Success Factors Campus-wide laptop initiative literature can be divided into two aspects. The first aspect is devoted to guidance and encouragement from campus-wide implementation to how curriculum and courses could be transformed (Brown, Burg, & Dominick, 1998; Brown, 2003; Brown & Petitto, 2003; Candiotti & Clarke, 1998; Kiaer, Mutchler, & Froyd, 1998, Spodark, 2003). The second aspect is composed of systematic studies documenting the effects of mandatory computing programs on student attitudes and learning as well as modes of teaching (Hall & Elliott, 2003; Kuo, 2004). Several prior studies focus on student experience with technology (Demb, Erickson & Hawkins-Wilding, 2004; Finn & Inman, 2004; Kuo, 2004; Mitra, 1998; Mitra & Steffensmeier, 2000; Platt & Bairnsfather, 2000; Varvel & Thurston, 2002). For instance, a study pointed out that convenience, hardware/ configuration choices, and cost were major issues for laptop initiatives (Demb et al., 2004). Demb et al. (2004) observed that a majority of students noted that the cost of a laptop computer was a very important factor. Nevertheless, a vast majority of students appreciated the convenience of the laptop, especially in terms of portability and size. Although a majority of students liked their laptops and used them heavily, their inability to make choices was very frustrating to them. Despite several studies on laptop initiatives, no prior study directly examined students’ perceptions of critical success factors necessary for the implementation of a laptop initiative. Obtaining a student body perceptual overview towards the critical success factors necessary for such a laptop program would assist actions towards positive attitudinal development during the implementation of the laptop program.

Methodology A direct survey was used to collect the data for this study. The survey questions were based on logistical and technological factors identified in previous studies by Demb et al. (2004), Luarn & Lin (2005), Moore & Benbasat (1991). Additional questions recommended by researchers and students were added to the survey. These questions were designed to gather data on students’ perceptions on the logistical and technological factors necessary to implement a laptop initiative, as well as their demographics. It should be noted that there are additional dimensions that influence the success of a laptop program. While pedagogical, social, and other dimensions are important to a laptop program’s success, the focus of this study will be on the logistical and technological dimensions. To check the clarity of these questions, three professors and three students were asked to read through the survey questions and provide feedback. Revisions to the survey were made based on the feedback received. The survey consisted of 55 items. Fifty-four of the survey items were used as five-point Likert scaled questions with end points ranging from “strongly disagree” to “strongly agree.” Survey items Q1 to Q28 collected demographic data such as age, gender, computer ownership, experience using computers, etc. Survey items Q29 to Q53 measured students’ perceptions on the logistical and technological factors necessary to implement a laptop initiative. Survey item Q54 measured students’ willingness to support a laptop initiative. Survey item Q55 was an open-ended question. This question was included on the survey to allow the respondents to list other factors they deemed important to the successful implementation of a campus-wide laptop program. 113

Data Collection Surveys were distributed to 272 undergraduate and graduate students enrolled in a mid-sized four-year university. The participants were given a 55-item survey and allowed class time to complete the survey. All participants were informed that participation in the study was voluntary and that all individual responses would be kept anonymous. The students were asked to rate each of the survey items on a Likert-scale from 1 to 5 with 1 being “strongly disagree” and 5 being “strongly agree.” Two hundred and sixty-nine surveys were returned. The average age of the respondents is 26.75 years old. Approximately 18.1 percent of the respondents agreed or strongly agreed with requiring all students to purchase a laptop computer for use in their education. Approximately 55.9 percent of the respondents disagreed or strongly disagreed with a laptop computer initiative. The remaining 26 percent of the respondents were neutral on a laptop computer initiative. Table 1 summarizes additional demographic characteristics of the respondents. Table 1: Demographic Characteristics Age (in years) Under 18 18-21 22-25 26-29 30-33 34-37 2 (0.7%) 105 (39%) 83 (30.9%) 29 (10.8%) 14 (5.2%) 12 (4.4%) Gender Female Male 166 (61.7) 102 (37.9%) Ethnicity African American Asian Caucasian Hispanic Native American 17 (6.3%) 15 (5.6%) 133 (49.4%) 98 (36.4%) 5 (1.9%) First Generation College Student Yes No 127 (47.2%) 139 (51.7%) Own a Computer Desktop Laptop 227 (84.4%) 109 (40.5%)

Over 37 24 (8.9%)

Analysis and Discussion All of the 272 surveys distributed were returned. Virtually all of the questions were answered on 269 of the surveys. The remaining three surveys were returned incomplete. The research data showed an odd-even reliability score of 0.901, suggesting internal consistency of the data. In addition, a Cronbach’s alpha score of 0.921 was calculated as a second measure of reliability. It should be noted that these high levels of reliability relate to the data resulting from the measurement, not the instrument itself.

Logistical and Technological Factors Perceived as Critical To isolate which logistical and technological factors were deemed as critical to the successful implementation of a laptop computer initiative, the mean responses to each question were calculated and examined. The survey items were on a Likert-scale ranging from 1 to 5. Figure 1 below shows the five factors rated as more critical with means ranging from 4.37 to 4.45. The students believe that the University must provide a wireless network for them to access information stored at various points on campus and to access the Internet. Wireless network access is viewed as the most important factor in the success of a laptop initiative. This factor had a mean score of 4.45 out of 5. Eighty-seven percent of the students agreed or strongly agreed with this factor. This finding is no surprise since one of the main benefits of laptop computers in education is the ‘learning anywhere, anytime’ ability. In addition, this finding follows the wireless networking trend that currently exists in the industry. Having sufficient power is also seen as a critical factor in the success of a laptop initiative. The mean student rating on this survey item was 4.43 out of 5. The majority of the students, 87.4 percent, rated the availability of power outlets in class as a necessity. The laptop batteries are easily drained when the laptop is under constant use. Thus this finding is definitely an issue that needs to be addressed to have a successful laptop initiative. 114

4.46

4.45 4.43

Response Mean

4.44 4.42

4.42 4.39

4.4

4.37

4.38 4.36 4.34 4.32 Provide a wireless network

Provide sufficient power outlets in the class

Provide Provide Provide onsite students with maintenance breakdown of access to all associated printers costs Factors

Figure 1. The logistical and technological factors perceived as critical In addition, students believe that access to printers is a critical factor in the success of a laptop initiative. This factor had a mean score of 4.42 out of 5 with 86.2 percent agreeing or strongly agreeing with this factor. This finding is consistent with observed student behavior in physical computer labs. Many students complete their assignments on computers off-campus and bring their work to the computer lab to print hard copies. Students also believe that it is critical for the University to provide onsite maintenance for the laptop computers. This is evidenced by the high mean score of 4.39 out of 5. The majority of the students, 85.9 percent, agreed or strongly agreed with the necessity of onsite maintenance. If the students are expected to use their laptops as a learning tool, they must be able to have their laptop quickly serviced when needed. Another factor rated as critical by students was the issue of providing them with a breakdown of all associated costs of owning a laptop computer. With the increasing cost of tuition and books, money is almost always an issue with students. Approximately 85.5 percent of the students agreed or strongly agreed with this item. The mean response for this issue was 4.37 out of 5.

Logistical and Technological Factors Perceived as Not so Critical To determine which logistical and technological factors were deemed as not so critical to the successful implementation of a laptop program, the mean responses to each question were calculated and examined. Figure 2 below shows the five factors rated as less important with the means ranging from 2.92 to 3.58. While students may see the benefits of using laptop computers in their education, they do not believe that it should be a requirement. This factor had a mean score of 2.92 out of 5 with only 36.4 percent agreeing or strongly agreeing with this factor. This factor is consistent with the cost factor that was deemed as a critical success factor for the students. Students also believe that it is not critical to exchange to a new laptop after two years. The mean score of this issue was 3.00. While only 33.1 percent agreed or strongly agreed with exchanging to a new laptop after two years, 46.9 percent agreed or strongly agreed with not requiring a two year exchange. This finding is consistent with the question asking about the need for instantly upgrade hardware. About 75 percent do not perceive that it is critical to instantly upgrade hardware of the laptop. In addition, students believe that it is not critical to require the purchase of a backup battery. This factor had a mean score of 3.15 out of 5 with only 42.8 percent agreeing or strongly agreeing with this factor. This finding is consistent with the observed critical factor of providing sufficient power outlets in class. With sufficient power outlets available, the need for a backup battery becomes a non-critical issue. 115

Response Mean

4 3.5 3 2.5 2 1.5 1 0.5 0

3.58

Provide physical storage space

3.51

Instantly upgrade hardware

3.15

3.00

2.92

Require all Require all Require all students to students to students to purchase a exchange purchase backup to a new laptop battery laptop after two years Factors

Figure 2. The factors perceived as not so critical The factor of upgrading the hard drive of the laptop was also a less critical issue. This factor had a mean score of 3.51 with only 50.5 percent agreeing or strongly agreeing with this factor. This is consistent with the findings regarding a required two year hardware update.

Difference Between Groups

o ye ar Ba s st c ku ud p en b ts at te m ry us Al tp lf ac ur ch ul In ty as st t e o an us t l e y D up la em pt gr on op ad st e ra so te ftw ho D ar w em e to on u st s e ra la te pt la op pt op be ne fit Ba s St s an ic tra da in rd in iz g ed M so ai f t nt w ar en e an ce su pp or t

Support Group Reject Group

Al l

Ex ch an ge

af te rt w

Response Mean

5 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0

Factors Figure 3. Differences between support and reject groups Students also perceived that the necessity of providing physical storage space, for the laptop computer when not in use, was a less critical issue. This factor had a mean score of 3.58 with 56.1 percent agreeing or strongly agreeing with this factor. This finding is consistent with the portability concept of the laptop computer. Laptop and notebook computers are smaller and lighter which make them easy to carry around when not in use. Hence, physical storage space on campus is not a necessity.

116

Support Group vs. Reject Group It is also interesting to examine whether there are any differences between students who favor the laptop program and those who are against the program in terms of their perceptions on each factor. The responses were therefore divided into two groups, those who favored the laptop initiative (support group) and those who did not support the initiative (reject group). The students who were uncertain on the laptop initiative were excluded. Then, t-tests were conducted to find out if there were significant differences between these two groups. Figure 3 below shows the logistical and technological factors exhibiting a significant difference between the two groups at a p-value < 0.01. There was a statistically significant difference between the two groups on ten of the factors (see Figure 3): (1) exchange the laptop for a new one after two years, (2) purchase a backup battery, (3) require all students to purchase a laptop, (4) encourage all faculty to integrate the laptop in their teaching, (5) instantly upgrade software installed on the laptop, (6) demonstrate to students how to use the laptop, (7) demonstrate the laptop’s benefits to the students, (8) provide basic training on laptop operation, (9) provide all students with standardized software, and (10) provide onsite maintenance support. The fact that the support group rated all factors higher than the reject group demonstrated that they tend to pay more attention to the details of program implementation. These findings suggest that the university may want to consider these logistical and technological factors before implementing the laptop program.

Conclusion Due to the ever increasing use of technology in primary and secondary education and the increasing demand, by industry, for more computer savvy graduates, the use of technology in higher education will continue to grow. Although many higher education institutions view the use of laptop computers as advantageous, in order to smooth the transition, the logistical and technological factors critical to a successful laptop program must be identified and addressed before implementing the requirement. This study identifies the logistical and technological factors that are important to students when requiring them to purchase a laptop computer. The results indicated that students, both those who support and those who do not support laptop initiatives, place a critical level of importance on the following factors: 1) having a wireless network in place, 2) having sufficient power outlets available in class, 3) having access to printers, 4) having onsite maintenance, and 5) having a breakdown of all associated costs. In addition, the two groups perceived that the following five factors were not so critical: 1) requiring all students to purchase a laptop, 2) requiring all students to exchange to a new laptop after two years, 3) requiring all students to purchase a backup battery, 4) instantly upgrading the hardware, and 5) providing physical storage space/locker for students to store the laptop when not in use. The results also revealed that ten logistical and technological factors were perceived differently between the groups who support and do not support the laptop program. These ten factors were: (1) exchange the laptop for a new one after two years, (2) purchase a backup battery, (3) require all students to purchase a laptop, (4) encourage all faculty to integrate the laptop in their teaching, (5) instantly upgrade software installed on the laptop, (6) demonstrate to students how to use the laptop, (7) demonstrate the laptop’s benefits to the students, (8) provide basic training on laptop operation, (9) provide all students with standardized software, and (10) provide onsite maintenance support. These findings suggest that the university may want to carefully consider these factors before implementing the laptop program. This study identified the logistical and technological factors perceived as critical by students regarding a laptop program. Determining such factors may allow educational institutions a base level awareness of students’ perceptions. This awareness could provide insights into what needs to be done towards an effective laptop program. These initial findings provide avenues for future research. To achieve a better understanding of all factors important to the laptop program, future research should also include the perceptions of faculty, administrators, and staff as well as those of students.

117

References Al-Khaldi, M. A., & Al-Jabri, I. M. (1998). The relationship of attitudes to computer utilization: new evidence from a developing nation. Computers in Human Behavior, 14 (1), 23-42. Brown, D. G. (2003). Ubiquitous computing: The universal use of computers on college campuses, Bolton, MA, USA: Anker Publishing Company. Brown, D. G., Burg, J. J., & Dominick, J. L. (1998). A strategic plan for ubiquitous laptop computing. Communications of the ACM, 41 (1), 26-35. Brown, D. G., & Petitto, K. R. (2003). The status of ubiquitous computing. Educause Review, 38 (3), 25-33. Candiotti, A., & Clarke, N. (1998). Combining universal access with faculty development and academic facilities. Communications of the ACM, 41 (1), 36-41. Demb, A., Erickson, D., & Hawkins-Wilding, S. (2004). The laptop alternative: Student reactions and strategic implications. Computers & Education, 43 (4), 383-401. Finn, S., & Inman, J. (2004). Digital unity and digital divide: Surveying alumni to study effects of a campus laptop initiative. Journal of Research on Technology in Education, 36 (3), 297-317. Gottfried, J., & McFeely, M. (1998). Learning all over the place: Integrating laptop computers into the classroom. Learning & Leading with Technology, 24 (4), 6-12. Hall, M., & Elliott, K. (2003). Diffusion of technology into the teaching process: Strategies to encourage faculty members to embrace the laptop environment. Journal of Education for Business, 78 (6), 301-307. Hebert, M., & Benbasat, I. (1994) Adopting information technology in hospitals: the relationship between attitudes/expectations and behavior. Hospital & Health Services Administration, 39 (3), 369-383. Kiaer, L., Mutchler, D., & Froyd, J. (1998). Laptop computers in an integrated first-year curriculum. Communications of the ACM, 41 (1), 45-49. Kuo, C. (2004). Perceptions of Faculty and Students on the Use of Wireless Laptops. Society for Information Technology and Teacher Education International Conference 2004, retrieved 13 April 2006 from http://dl.aace.org/14470. Liaw, S. (2002). An Internet survey for perceptions of computers and the World Wide Web: relationship, prediction, and difference. Computers in Human Behavior, 18 (1), 17-35. Lim, D. (1999). Ubiquitous mobile computing: UMC’s model and success. Educational Technology & Society, 2 (4), 125-129. Luarn, P., & Lin, H. H. (2005). Toward an understanding of the behavioral intention to use mobile banking. Computers in Human Behavior, 21 (6), 873-891. Mitra, A. (1998). Categories of computer use and their relationship with attitudes toward computers. Journal of Research on Computing in Education, 30 (3), 281-295. Mitra, A., & Steffensmeier, T. (2000). Changes in student attitudes and student computer use in a computerenriched environment. Journal of Research on Computing in Education, 32 (3), 417-433. Moore, G. C., & Benbasat, I. (1991). Development of instrument to measure the perceptions of adopting an information technology innovation. Information Systems Research, 2 (3), 192-222. Platt, M. W., & Bairnsfather, L. (2000). Compulsory computer purchase in a traditional medical school curriculum. Teaching and Learning in Medicine, 11 (4), 202-206.

118

Rucker, J., & Reynolds, S. (2002). Learners and technology: experience, attitudes, and expectations. In Remp, A. M. (Ed.), Technology, Methodology, and Business Education: 2002 Yearbook, Reston, VA, USA: National Business Education Association, 1-20. Spodark, E. (2003). Five obstacles to technology integration at a small liberal arts university. T.H.E. Journal, 30 (8), 14-24. Steel, J., & Hudson, A. (2001). Educational technology in learning and teaching: the perceptions and experiences of teaching staff. Innovations in Education and Teaching International, 38 (2), 103-111. Varvel Jr., V. E., & Thurston, C. (2002). Perceptions of a Wireless Network. Journal of Research on Technology in Education, 34 (4), 487-501. Weiser, M. (1998). The future of ubiquitous computing on campus. Communications of the ACM, 41 (1), 41-42.

119

Appendix 1: Critical Success Factor Survey Instrument CSF LAPTOP INITIATIVE QUESTIONNAIRE Please answer the following questions. Do you own a desktop computer?

1. yes

2. no

Do you own a laptop computer?

1. yes

2. no

How would you rate yourself with respect to your knowledge about computers? 1 (very poor) 2 3 4 5 6 7 (excellent) How long have you been using the Internet? 1. never 2. less than 1 year 3. 1-2 years 4. more than 2 years How often do you use the Internet per week? 1. never 2. one time 3. 2-3 times 4. more than 3 times How often did you use the desktop computer in high school per week? 1. never 2. one time 3. 2-3 times 4. more than 3 times How often did you use the laptop computer in high school per week? 1. never 2. one time 3. 2-3 times 4. more than 3 times How often do you use the desktop computer for classroom assignments per week? 1. never 2. one time 3. 2-3 times 4. more than 3 times How often do you use the laptop computer for classroom assignments per week? 1. never 2. one time 3. 2-3 times 4. more than 3 times How often do you use the desktop computer for leisure per week? 1. never 2. one time 3. 2-3 times 4. more than 3 times How often do you use the laptop computer for leisure per week? 1. never 2. one time 3. 2-3 times 4. more than 3 times Have you worked before?

1. yes

2. no

Did your previous work require a use of a desktop computer?

1. yes

2. no

Did your previous work require a use of a laptop computer?

1. yes

2. no

Does your current work require a use of a desktop computer?

1. yes

2. no

Does your current work require a use of a laptop computer?

1. yes

2. no

What is your current employment status?

1. full-time

2. part-time

3. unemployed

What is your expected cost per semester for the purchase of a laptop computer (including a service fee for onsite maintenance support)? 1. less than $100 2. $100-$150 3. $151-$200 4. $201-$250 5. $251-$300 6. $301-$350 7. $351$400 8. $401 up I prefer to pay ……….. 1. …. $350 per semester (including onsite maintenance support) and graduate with a TWO year old laptop computer. 2. …. $200 per semester (including onsite maintenance support) and graduate with a FOUR year old laptop computer. Your status:

1. student 2. faculty

3. staff 120

What is your classification?

1. Freshman

Your college: 1. Arts & Humanities

2. Business

Are you a first generation college student? Your gender:

1. male

Your age: 1. under 18 over 49

3. Education 4. Nursing

5.Graduate

5. Science & Technology

1. yes 2. no

2. female

2. 18-21

Your ethnicity: 1. African

2. Sophomore 3. Junior 4. Senior

3. 22-25

2. Anglo

4. 26-29 5. 30-33

9. 46-49

10.

4. children

5.

2. $20,000-$39,999 3. $40,000-$59,999 4. $60,000-$79,999

5.

3. Asian

4. Hispanic

6. 4-37 7. 38-41 8. 42-45 5. Native American

How do you support your education? (Mark all that apply) 1. self scholarship 6. financial aid 7. loan 8. employer 9. Other Your annual income: 1. under $20,000 $80,000 and over

How important do you think each of the following are ….…. 1--- not important 2 --- somewhat not important 3 --- neutral important

2. parents 3. spouse

4 --- somewhat important

5 ---

For the laptop initiative to be successful, the University must…. …. provide a wireless network 1 …. provide onsite maintenance support 1 …. provide a loaner computer while the laptop is in for service 1 …. provide sufficient power outlets in the class. 1 …. provide sufficient power outlets outside the class. 1 …. provide basic training to all students after they purchase the laptop 1 …. provide all students with a university e-mail account 1 …. provide a standardized package of software to all students 1 …. provide a help desk to answer basic laptop operating questions 1 …. provide a breakdown of all associated costs of owning a laptop 1 …. provide students a laptop lease option 1 …. provide students with network storage space 1 …. provide students with access to printers 1 …. provide update for virus protection 1 …. provide physical storage space/ locker for students to store the laptop when not in use 1 …. require all students to exchange to a new laptop after two years 1 …. not require all students to exchange to a new laptop after two years 1 …. require all students to purchase a backup battery 1 …. require all students to purchase a laptop 1 …. demonstrate the benefits of using the laptop before requiring them to purchase it 1 …. demonstrate how to fully utilize the laptop before requiring them to purchase it 1 …. encourage all professors to fully utilize the laptop in the class 1 …. instantly upgrade the hard drive of the laptop. 1 …. instantly upgrade the software installed in the laptop. 1 …. to be able to transfer the work on my laptop to the campus computer. 1 It is a good idea to require all students to purchase a laptop computer. 1

2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4

5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5

List any other items that you feel are necessary to successfully implement a campus-wide laptop computer program.

121

Cagiltay, N. E., Yildirim, S., & Aksu, M. (2006). Students’ Preferences on Web-Based Instruction: linear or non-linear. Educational Technology & Society, 9 (3), 122-136.

Students’ Preferences on Web-Based Instruction: linear or non-linear Nergiz Ercil Cagiltay Computer Engineering Department, Atilim University, Ankara, Turkey Tel: +90 312 586 83 59 Fax:+90 312 586 80 90 [email protected]

Soner Yildirim Department of Computer Education and Instructional Tech, Middle East Technical University, Ankara, Turkey Tel: +90 312 210 40 57 Fax: +90 312 210 11 05 [email protected]

Meral Aksu Department of Educational Sciences, Middle East Technical University, Ankara, Turkey Tel: +90 312 210 40 05 Fax: +90 312 210 11 05 [email protected] ABSTRACT This paper reports the findings of a study conducted on a foreign language course at a large mid-west university in the USA. In the study a web-based tool which supports both linear and non-linear learning environments was designed and developed for this course. The aim of this study was to find out students’ preferences pertaining to the learning environment and to address the factors affecting their preferences. The results of this study showed that the individual characteristics of the students affected their preferences on the learning path (linear or non-linear).

Keywords Linear instruction, Non-linear instruction, Web-based course, Individual differences

Introduction In traditional classroom settings, instructors present information by using a linear model. For example, a video may be shown from the beginning to the end or a textbook is covered from one chapter to the next. Generally, most of the early applications of modern technology were also based on structural and linear instruction through an electronic platform and mainly based on the delivery of course material (Roblyer & Edwards, 2000; Dalgarno, 2001, Simonson & Thompson, 1997). On the other hand, according to Howell, Williams and Lindsay (2003), instruction is becoming more personalized: learner-centered, non-linear and self-directed. Socialconstructivist pedagogical approaches have introduced different (active, learner-centered and communitycentered) models and pose strong arguments against the structured knowledge consumption approach (Koper & Oliver, 2004). As Silva stated (1999, p.1.), “The use of technology is important for Second Language courses, more important for Foreign Language courses and even more important in the curricula of the less-commonly taught foreign languages …”. Several studies found that computers have positive effects on teaching language. Stepp (2002) summarized some positive effective benefits of technology for foreign language learners. Research results of Computer-Assisted Instruction (CAI) showed a significant increase in students' scores in both reading comprehension and vocabulary and spelling (Stone, 1996; Kulik, 1994; SIIA, 2000) in the classrooms where computers are used. Students using computer software designed for developing spelling had significantly higher scores than the others (Stone, 1996; Anderson-Inman, 1990 cited in SIIA, 2000). Researchers have found that when students use word processors, they show a higher level of writing skills (SIIA, 2000). Hirata (2004) also shows that native English speakers who used the Japanese pronunciation training tool have improved their overall test scores significantly. However, there are some other studies showing that there were no significant differences between the classrooms using computer applications and those using traditional lecture-based courses (Wilson, 1996 cited in Gilbert & Han, 1999; Goldberg, 1997, Hokanson & Hooper 2000). For example, Cuban, Kirkpatrick, and Peck’s study (2001) show that investments in infrastructure and increased access to technology did not lead to increased integration, instead, most teachers remained “occasional” or “non-users” of classroom technology (p. 813). They state that limited time to learn and implement new technology was ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

122

considered a serious barrier as well as poorly implemented professional development and defects in the technology itself. In parallel to this finding, Hokanson and Hooper (2000) pointed out that the expanded use of computers in education continues despite research having failed to accrue definite benefits in learner’s performance. According to Gilbert and Han (1999), the main reason for finding no significant difference between the traditional education system and the system using technology is the instructional methods. Barker, Giller, Richards, Banerji, and Emery (1993) reported that early implementations of computer-assisted language learning (CALL) also had several limitations. They were mainly built on text-based instruction with very limited end-user interaction and participation. For some researchers, presenting the information in a linear form was not a problem when the information being presented is well structured and simple. Often, however, as the difficulty of the material increased so did the lack of structure. When the knowledge domain to be taught is complex and ill structured, the use of traditional linear instruction becomes ineffective (Spiro, Feltovich, Jacobson, Coulson, 1991). In that context Barker (1997, pp 5) states that: … for one reason or another, many academic organizations are now therefore exploring the possibilities of using these new technologies to support student-managed, self-study activities in a more extensive way than they have in the past. However research findings on learner control have proven contradictory. Some findings show learner controlled environments lead to higher performance, whereas others show no significant difference, or even that instructor or program controlled systems work better. These findings illustrate that although learners’ control over the learning environment is important to improve the learning process, there should be some factors affecting their learning and preferences in such an environment. The existing studies outline the trends in the evolution of CALL and the development from the perspective of pedagogy and language learning (Warschauer & Kern, 2000), however more research into CALL is needed (Chambers, 2001; Davies, 2001; Levy, 2001). CALL assists language learning and is intended to enhance the way in which a language is taught and learned (Decoo, 2003). Decoo (2003) summarizes some of the levels of language teaching methods such as the label method, program method, textbook method, teacher method and student method. Decoo (2003) conclude that CALL is used to strengthen and improve these existing methods. As Chan, and Kim reports (2004), there is a shift in the second language curricula from declarative knowledge or “what we know about” to procedural knowledge or “what we know how to do”. This causes a greater emphasis on learners’ learning process (Chan, & Kim, 2004). They believe that, appropriate use of suitably designed Internet-based materials can make a significant contribution towards facilitating autonomous learning (the ability to take charge of one’s own learning) (Chan, & Kim, 2004). Accordingly developers now try to design interfaces that give learners more autonomy (Lonfis & Vanparys, 2001). They also claim that any foreign language curriculum that aims to promote autonomy must focus on putting learners in control of their linguistic and learning process (Chan, & Kim, 2004). However there is very limited research which examines students’ preferences and performance in such a learner controlled learning environment. In this study, a web-based tool was designed and developed for an entry-level foreign language course, which supports both linear and non-linear learning paths. The aim of this study is to find out students’ preferences regarding learner controlled environments and address the factors affecting these preferences. The next section examines the factors affecting students’ preferences regarding learner controlled learning environments. Different instructional approaches such as direct instruction and indirect instruction are also discussed in this section. In the third section, the research method is discussed. The fourth section reports the results of this study while the final section presents the conclusions and discussions of the current study.

Background This section tries to investigate the factors affecting students’ preferences in a learner controlled environment. It also gives a brief explanation of direct and indirect teaching. Factors affecting students’ preferences on a learner controlled learning environment Individual Differences One prominent theory of individual differences is Howard Gardner’s Multiple Intelligences. Gardner suggests that all people have varying degrees of innate talents developed from a mixture of biological factors, evolution and culture (Gardner, 1983). Each intelligence represents an area of expertise with a specific body of knowledge, 123

as well as a way of approaching learning in any domain. Students may experience new ways of expression, helping them individually, to understand multiple perspectives. In parallel to Gardner’s theory (1983), studies on learning and information processing suggest that individuals perceive and process information differently (Hitendra, 1998). According to Gilbert and Han (1999), how much individuals learn is related with the educational experience geared toward their particular style of learning. Mcmanus (1997, p1) argues that: One of the great promises of computer based instruction is the idea that someday the instruction could be adapted to meet the specific needs and styles of individual learners, thereby enhancing their learning. In order for this to happen, educators need to know which instructional and presentation strategies, or combination thereof, is most effective for individuals with certain learning styles and differences, in a given learning environment. Hitendra found that, certain cognitive styles might suit certain types of test tasks (1998). Cho also found that individual learning styles and preferences are presumed to affect the moment-to-moment selection of options in non-linear learning environments (1995). For example, several studies showed that the field-independent (FI) learning style seems to facilitate understanding the structure intended by the designer of the instruction more than those with a field-dependent (FD) style (SIIA, 2000). Kelley and Stack found that learners having an external locus of control usually tend to perceive reinforcements from other people (Kelley & Stack, 2000). According to Kelley and Stack, people with an internal Locus of Control (LOC) seek more control over life’s circumstances, and like to have more personal responsibility for outcomes (Kelley & Stack, 2000). Age Additionally, there is some evidence to suggest that the age of the learner may be an important variable in learner control. Shin, Schallert and Savenye (1994) indicated that students (seven or eight years of age) who were given only limited access through a hierarchical hypertext structure answered more questions correctly in the post-test than students in the free access network hypertext structure. Hannafin (1984) concluded from a review of the relevant research that learner control compared with program control is likely to be most successful when learners are older. Gender Research studies also show that gender is an effective factor on learner control. For example Knizie et al. (1992) found that the use of a program controlled environment resulted in better post-test performance for male students. Similarly, Braswell and Brown (1992) show that in interactive video learning environments females had a better performance than the males. Student’s prior knowledge and familiarity with the material Research studies also show that a student’s prior knowledge and familiarity with the material to be learned and the subject domain to be learned are the factors affecting students’ success in a learner-controlled instructional environment. For example, students whose prior understanding of a topic is low should be provided with more structured information whereas students whose prior understanding of a topic is high can be given more control over the instructional system (Gay, 1986). According to the findings of Charney, students who are new to the subject domain and non-linear learning systems may sequence the information poorly or omit important information altogether (1987). Cho reports that learners make poor decisions about what to study and in what order (i.e., selecting links) because they have insufficient knowledge about the new content (1995). Therefore, students vary in their cognitive or learning styles and would benefit from teaching techniques that appeal to their individual styles (Brown & Liedholm, 2004). By providing several different instructional methods, the use of technology in education will significantly improve educational performance (Gilbert & Han, 1999) and web offers a rich environment for this purpose. Accordingly, as Internet technology is improved, Gardner’s theory has gained more popularity. The web offers a variety of instructional materials that could be incorporated into effective learning environments.

124

Direct Teaching (Teacher Instruction) Direct teaching or direct instruction is a systematic way of planning, communicating, and delivering in the classroom. This method provides the students with strong structure that helps them to concentrate on their academic task. The direct instruction approach assumes that all students learn at the same speed and at the same way. The role of the learners in direct instruction is to stay on the task and perform. In this context program controlled learning environments are considered to be “direct instruction”.

Indirect Teaching (Indirect Instruction) Indirect instruction is mainly student-centered. Indirect instruction seeks a high level of student involvement. It takes advantage of students’ interest and curiosity. It is flexible in that it frees students to explore diverse possibilities. Students often achieve a better understanding of the material and ideas under study and develop the ability to draw on these understandings. The current interoperability specifications have to be extended to include the multi-role interactions and the various pedagogical models that are needed to provide real support for learners and teachers in more advanced and newly developing educational practices. In our study, the learner controlled environment of the tool is considered as being “indirect instruction”. Several research efforts have shown that computer programs offer students greater control over their learning environments and have beneficial effects on students (Shoener, & Turgeo, 2001; Wooyong & Robert, 2000; Hargis, 2000; Kemp, Morrison & Ross, 1994; Hannafin & Sullivan, 1995; Shyu & Brown, 1992; Santiago, & Okey, 1992; Knowles, 1975). According to Schank students should control the educational process; not the computers (1993). In order to increase the learner’s control over the learning environment, organizing the instruction in a non-linear manner becomes important. In such an environment students have a higher degree of freedom regarding the method of study and the study material. However, there are studies showing that some students do not succeed in learner-controlled learning environments. For some it is hard to make decisions about what to study and in what order. They may have trouble in monitoring their own learning (Cho, 1995). In support of this, the findings of McNeil’s study (1991) show that a learner-controlled program is less effective than a computer controlled program in CAI at the elementary level. According to McCombs (1988 cited in Sleight, 1997), since students don’t know how to use strategies in a non-linear learning environment, they are having adjustment problems. Chang (2003) found that teacher-directed CAI was more effective in improving students’ achievement than student-controlled CAI, given the same learning content and overall learning time. Results from this study also revealed that students in the teacher-directed CAI group showed significantly more positive attitudes toward the subject matter than did those students in the student-controlled CAI group (Chang , 2003). According to Ellis and Kurniawan (2000) the flexibility in non-linear learning paths may increase complexity.

Method This study was not designed to examine the effect of the web-based tool on students’ learning, instead its purpose was to reveal some factors effecting students’ preferences pertaining to linear and non-linear learning paths. In this sense, the tool for this study was designed and developed to support both linear and non-linear instruction. By addressing the factors that affect students’ preferences in this environment, the study also aimed to guide other research studies to gain further insight on the different behaviors of the learners. Accordingly the following factors were analyzed: (a) Individual differences among the students (age, gender, preferences on instructions (preferring teacherbased instructions (direct instruction) or self-paced learning (indirect instruction)), perceptions on problem solving (whether they are a good problem solver or not) and their learning preferences such as visual and audio) (b) Student’s prior knowledge and familiarity with the material (e.g. computers, computer experience, familiarity with windows-based applications and their backgrounds) This qualitative case study focuses on the perspectives of the participants of the study in order to uncover the complexity of human behavior in such a framework and present a holistic interpretation of what is happening in this context. As McMillan & Schumacher (2001) advocate case studies focus on phenomenon “. . . which the researcher selects to understand in depth regardless of the number of sites or participants” (p.398). Qualitative 125

case studies also yield to better understanding of abstract concepts as this is the case for this study. Therefore, in this study the critical point is not the number of subjects but the depth and the quality of presenting the subjects’ learning experience with the system.

The Course This study was conducted on an entry-level Turkish language course at a mid-west university in the USA. Turkish is a less-commonly taught foreign language in the USA. This course is offered as an elective for students with different backgrounds. The number of students registering in this course in each semester is usually very limited. Turkish instructors whose native language is Turkish, but who have no formal language education offer these courses. Accordingly, the instructor turnover rate is very high. Additionally, new instructors sometimes need additional information and sources to prepare the lectures and to find out the answers for some of their questions. The course was organized in the form of 45-minute lessons five days a week. In general, on Mondays the instructor would introduce the topics, on Tuesdays she would introduce the grammar issues related with that lesson, the next day she would do different exercises related with the topic, on Thursdays she would introduce some songs, audio-visual materials related with the topics and finally on Fridays she would help the students in group exercises. The course instructor also organized after-school conversation hours every three weeks. The students were free to attend these conversation hours. For these after-school conversation hours, the course instructor invited some people interested in talking Turkish, to help students communicate with native Turkish language speakers while talking about topics related to daily life.

The Tool In this study, a web-based learning tool was developed for the entry-level Turkish language class (available at http://clio.dlib.indiana.edu/~ncagilta/tlepss.html and http://www.princeton.edu/~turkish/practice/tlepss.html). Several different forms of instruction such as sound, image and text were also provided. The contents of the tool were all adapted from the course textbook. The first unit of the course textbook bound the scope of this study. The contents of the tool were also enhanced with some sound files. There are 480 sound files in the system. The contents of the tool were fostered with some pictures (images) as well. Most of the images used for the lessons and examples were taken from Microsoft's clip art gallery (Microsoft, 2000). 307 images were used for this system. To assist the students in different applications, some tools such as text boxes, simulated Turkish keyboards, and indexes were also provided within the tool. The tool includes some instructions, examples and exercises. In Figure 1 an example page is shown.

Figure 1. An Example in the Tool The tool was designed and developed to support both linear and non-linear instruction and several practice alternatives. Non-linear instruction Non-linear instruction was provided through the Indexes (organized as Turkish or English) (Figure 2) to help students to select any topic from indexed list, to initiate their own learning and direct it. Students may choose to select and study a concept, or go through the instruction and the examples using the index. 126

Figure 2. English Index providing non-linear Instruction Linear Instruction Linear instruction was provided through the main menu shown in Figure 3. In this menu, the content is organized in the same order as introduced in the classical classroom environment. Students may follow the instructions in this linear order.

Figure 3. Main Menu providing linear Instruction

Data Collection The actual data collection process for this study was conducted during the first six weeks of the semester. The content of the tool covers the first three weeks of the course. At the beginning of the semester an orientation session was organized to introduce the main purpose of the research and how to use the web-based environment. Afterwards, students used the tool in parallel to their regular classes. The tool was not used during the classroom activities; rather students were provided with a CD version of the tool and were also able to access it via the Internet. During this period, students were asked to use the tool whenever they needed help. They had 127

opportunities to choose any subject within the tool and study it, or practice a chosen subject. In this sense, students had to decide when to use the tool, how long to use the tool and how to use it according to their preferences. They had opportunities to repeat sounds, lessons and any activities in the tool whenever they needed help. After three weeks, an interview was conducted with each student in the classroom and with the course instructor. After the individual interviews, an observation session was also conducted with each student. During the observations, each student was asked to use the tool as if they were alone. The observer recorded each step that the student followed. The observation sessions took approximately half-an hour. The main purpose of the observation session was to find out the students’ preferences regarding the tool and to investigate their preferred learning path (linear or non-linear). These interviews and observations were all conducted within five days. During the second half of the six week period, students followed the lessons without the support of any specific web-based tool designed for these lessons. After this three week period, the next round of interviews was conducted with the students and the course instructor to obtain comparative feedback on the benefits and weaknesses of the tool. These interviews were all conducted within three days. During the first and the second interviews several questions were presented to the students and the instructor to get a better view regarding the major variables analyzed in this study. During the interviews with the course instructor, a different session for each student was conducted.

Students There were ten students in the classroom. For the sake of anonymity, each student was assigned a different code, such as S1 and S2. Students were from different age groups and had different backgrounds. Meanwhile, there was a wide range of distinction among their preferences and their expectations during the learning process. Only S4, S5, S6, S8, S10 were a little familiar with the Turkish language because of a Turkish person whom they knew. The others were not familiar with the course content at all. There were four male students in the class; only three students had very low computer usage but not much familiarity with the web-based applications. Five students preferred to study by themselves (self-study) while others preferred to have teacher instruction. Tables 1 & 2 summarize students’ profiles. All the data shown here was collected by means of interviews and observations, as mentioned in the data collection section. Table 1 Students’ Profiles (I) Code

Age

Gender

Problem Solving?

S2 S3 S4 S7 S10 S1

25 21 53 25 18 42

M F F M F M

Good Not good Not good Pretty good Good Excellent

Studying with teacher/ self-studying Self Self Self Self Self Teacher

S5 S6

25 50

M F

Very Good Good

Teacher Teacher

S8 S9

22 61

F F

Good Good

Teacher Teacher

Learning preferences Visual learner Learning by listening Learning by writing and repetition Audiovisual, Drill-and practice, hands on, strode it on the board, do the exercises, verbalize it Learning by talking. Group learner, learning with traditional methods, Writing, talking and reading. Watching, writing, talking and listening Talking and listening, learning with traditional methods

F: Female M: Male The students’ learning preferences were different from each other. Some students preferred to learn by drill & practice and doing exercises. On the other hand some students preferred visual illustrations. This group felt that visual illustrations helped them to learn easily and remember what they had learned. 128

Table 2 Students’ Profiles (II) Code

Likes Computers?

S1 S3

Not always Very much

S2 S7

Very much Yes, but cannot sit in front of a computer for several hour A lot Yes Very much Yes, but cannot sit in front of a computer for several hours Not very much Not very much Average

S8 S4 S5 S6 S9 S10

Computer experience (years) 30 3

Familiarity With windows based applications Very high Very high

8 19

High High

10 3 3 10

High Middle Middle Low

16 3 10

Low Low

Background Central Eurasian Studies Computer Information Systems Criminal Justice Law, Mathematics Biology Psychology History Educator MS on education Chemistry

Limitations of the study Since the tool developed for this study is not a professional one, it has some limitations in the sense of the content included and user interface issues. Additionally, only ten subjects participated in the research.

Results In this section, the data collected for this study is presented to provide evidence for the students’ preferences on the provided environment. Thus, first how students used the tool was analyzed. Then, students’ preferences pertaining to linear and non-linear types of instructions, according to their individual differences and their background regarding the environment were analyzed.

How students used the tool Results indicated that, each student used the tool in several different ways. While some of them preferred linear instruction, others preferred non-linear instruction. One student who preferred to use the tool in a non-linear order stated that: If the teacher gives me a specific example, I go through the assignment; otherwise, I try to find the topic that we are learning in class and I study the topic simultaneously on the computer with the class. … I liked to use the system interactively. My preference was going through the index and looking for the specific things there. If I found something interesting I would explore it… Similarly, in that context another student stated that he used the tool in a non-linear way to get a better view about the tool. He also stated that, if it was like an exam, he would prefer to use it in a linear order. He stated: I like to play with things first and then when I get tired I go through each. I kind of experiment with it first and then go through each item randomly. But for an exam it helps to have it in linear, run through it from the beginning to the end. However, one student stated that, in some cases she preferred the linear instructions where some other cases she preferred non-linear instructions. She stated that: I liked to search through the content independently. I liked to study the tool by using the indexes. But, I preferred to look at different Turkish books while studying grammatical components, instead of studying it through the tool. I believe that manual methods work better for me while studying grammar. With regard to this issue, one student stated; even though she preferred non-linear instructions, she liked seeing linear and non-linear instructions together: 129

If everything had been designed in a non-linear form, it would have been like a dictionary. Having the structured menu is very helpful. I preferred the dictionary part mostly, but I also liked seeing the content of unit one, and sometimes I went through it. After I studied the first unit, I wanted to come back to Main Menu. But if everything had been just like in the dictionary, it would have been harder. On the other hand, some students preferred only the linear instructions. One student stated that:

I just started from the Main Menu, then I clicked on each one, I kind of way down the list, I did not get a chance to do the exercises in the book, but I worked the exercises on the CD. Another student who liked linear instructions reported that she followed the instructions in the book, and then went through the CD: I like some linear instruction because I am not trained completely [to be an] independent learner. The younger generation [is] yes. So I need some instruction. But I can do a little bit of independent searching. I just need practice.

While some of the students preferred mostly to study the lessons, others did the exercises. Some of them used the tool on a daily basis after the lessons whereas others preferred to use it during the weekends. Most of the students studied the lessons from both the tool (web-based environment) and the course textbook. Only one student preferred to study the lessons from the web-based environment and not to use the textbook at all. Some students preferred to use the tool by reading the instructions and studying the examples. Some students preferred to study by means of exercises and went through the instructions as necessary. All students reported that they used the tool easily. All students found the user interface easy to use, easy to learn and self-explanatory. They said they could easily reach the answers of their questions using the tool. However, S6 reported that she encountered difficulty while studying the lessons using the tool. For example she needed some help from other people while using the tool, but when she needed help she could not find anybody. Additionally, while she was using non-linear instruction she was lost in the application. She reported, I should have followed the book but I did different things on the CD. So, I was lost on the CD. I blindly clicked on this that and was lost in the computer. As the course instructor describes, the technology is not her (S6) favorite. The course instructor declared that she helped S6 in using the tool 4 or 5 times in the computer laboratory. She added that other students did not ask for help. The instructor reported that, Sometimes she becomes intimidated because technology is something that she is not familiar with. It took some time for her to get used to the technology. After she got used to it, she liked it. But still technology is not her favorite thing. She likes studying by traditional methods. For this reason we study together with her to correct some mistakes in her homework. She prefers studying by writing, speaking and grammar; it is her preference.

Individual differences According to the course instructor, using the tool in conjunction with the course curriculum improved classroom performance. She believes that the individual differences among students affect the classroom’s performance: The individual differences among students affect class performance seriously. In general freshmen are better in audio than the more senior students whereas the more senior students usually have problems in hearing the words and understanding the words in an audio exercise. If the students’ individual characteristics and their preferred ways of learning of in the classroom are similar, then they can become more active in the classroom [and] learn a lot. They are motivating each other as well as helping each other. If these individual characteristics are not similar, then I need to spend more time to building a common instructional style, which will be helpful for all [/most]of the students in the classroom. Most of the time, this also affects the general classroom performance. For example, sometimes, some students distract the others by asking a lot of questions and by not following the common preferred way of learning of the classroom. According to the course instructor, when students used the research tool, they were more prepared for the lessons. She did not spend much time on repeating those lessons. The instructor also reported that, students did not ask as many questions during the lessons in parallel to the tool as during the other lessons. She believes that 130

students found answers for most of their questions from the tool. Using the tool was time saving for the course instructor as well and she gained more time to organize her lessons and do other activities in the classroom. Students’ preferences on the provided environment were analyzed according to the data collected by means of interviews and observations. Accordingly, Table 3 and 4 summarize students’ preferences on the linear and nonlinear ways of using the tool in the sense of individual differences. Table 3. Students preferring linear paths of instruction Code

Age 18 22

Prefers teacher or self-study Self Teacher

Problem solving Good Good

S10 S8 S6

50

Teacher

Good

S4

53

Self

Not good

S9

61

Teacher

Good

Average

41

Preferred way of learning Watching, writing, speaking and listening Group learner, learning with traditional methods, Writing, speaking and reading. Learning by writing and repetition Talking and listening, learning with traditional methods

Gender Female Female Female Female Female

Table 4. Students preferring non-linear paths of instruction Code

Age

S1

42

Prefers teacher or Self-study Teacher

Problem solving Excellent

S5 S7 S2

25 25 25

Teacher Self Self

Very Good Pretty Good Good

S3 Average

21 28

Self

Not Good

Preferred way of learning

Drill-and practice, hands on, strode it on the board, exercises, verbalization Learning by speaking Audiovisual Visual Learner Learning by listening

Gender Male

Male Male Male Female

Age: In this study, “age” was a factor affecting students’ preferences on selecting the linear or non-linear paths in the tool (Table 3 and 4). For example, the average age of the students who preferred linear instruction was 41 where the average age of the students who preferred non-linear instruction was 28. In this context, during the interviews the course instructor said that “age” is an important factor affecting the way how students learn a language. She further reported: “Age” is also an important factor, not personally how old they are but when they learn a language, what system they use, what materials and techniques they use. So, when I say subject pronoun, the older students know what I am talking about. But I have to introduce these terms for the freshmen because they did not learn it that way. Similarly when I give them an Internet exercise, the freshmen can use it easily, but I have to spend hours with the older ones. Accordingly, while using the tool, mostly the students belonging to the younger generation (S2: 25 years old, S3: 21 years old, S5: 25 years old, S7: 25 years old) preferred to use non-linear instruction. On the other hand, the more senior students (S4: 53 years old, S6: 50 years old, S9: 61 years old) preferred to use linear instruction in the tool. Gender: All male students preferred to use the tool through non-linear instruction. Only one female student (S3) preferred to use the tool through non-linear instruction. Preferred way of learning: Students who preferred to use the tool through linear instruction (S4, S6, S8, and S9) preferred the following ways of learning: writing, repetition, watching, group-learner or traditional learning methods. Other students who preferred to use the tool through non-linear instruction (S1, S2, S3, S5 and S7) preferred the following ways of learning: Drill and practice, visuals, listening, speaking and audiovisuals. 131

Teacher or self-study preferences: Most students who preferred self-study (studying on their own) (S2, S3 and S7) also preferred to use the tool through non-linear instruction. On the other hand, other students who preferred studying with the help of the teacher (S6, S8 & S9) also preferred to use the tool through linear instruction. Perceptions on Problem Solving Capacity: Most students who preferred non-linear instruction described their perceptions on their problem solving capacity as very good, pretty good, or excellent (S1: excellent, S5: very good, S7: Pretty good).

Prior knowledge and familiarity with the material Tables 5 and 6 summarize students’ preferences on the linear or non-linear way of learning with the tool according to their prior knowledge and familiarity with the material. Since they were not familiar with the course content before entering to this course, students’ prior knowledge on the course content will not be discussed here. Table 5. Students preferring linear paths of instruction Code Familiarity With Likes Computers? Computer Window Applications Experience (years) S8 A lot 10 High S4 Middle Yes 3 S6 Low Yes, but cannot sit in front of a 10 computer for several hours S9 Low Not very much 16 S10 Low Not very much 3 8.4 Average Code

Table 6. Students preferring non-linear paths instructions Familiarity With Like Computers Window Applications

S1 S3 S7

Very High Very High High

S2 S5

High Middle Average

Not always Very much Yes, but cannot sit in front of a computer for several hour Very much Very much

Computer experience (years) 30 3 19 8 3 12.6

According to these results, only the familiarity with windows-based applications was an effective factor on students’ preferences. We could not find any relation between liking computers or not and their computer experience. Familiarity with window-based applications (FWWA): All students (except S8), whose FWWA is low or middle, preferred the linear learning path. Other students (except S5), whose FWWA is high or very high, preferred the non-linear learning path. Classroom Performance: Although this study is not organized to investigate the effect of the tool on students’ or classroom’s performance, according to the course instructor it improved the classroom performance. She reported that she did not spend much time on repeating those lessons included in the tool. According to the instructor, students did not ask as many questions during the lessons studied parallel to the developed tool as in the other lessons. She believes that students found answers for most of their questions from the tool. She declares that using this tool was also time saving for her: she gained more time for organizing her lessons and doing other activities in the classroom. According to the instructor, this tool helps students to study on their own: They [students] can study on their own, or study with the tool. It [the tool] will also, perhaps help develop a skill on using other technological tools, to learn a language. Again, you make it for them more fun and easer. [It helps] most of them to study on their own, so they are not always dependent on the teacher. So, upon that point of use, it is [the tool] very useful. 132

Computer Experience: The average computer experience of the students who prefer linear path of instructions (8.4) is slightly lover than that of who prefer non-linear path of instructions (12.6). Computer Affinity: Most students who prefer the non-linear path of instruction like to use computers very much (S2, S3, S5). The students who do not like to use computers (S9 & S10) also preferred the linear path of instruction.

Discussions and Conclusion The results of this study underlined that the learners’ preferred learning path (linear or non-linear) depends on their personal characteristics such as their age, perceptions on problem solving, teacher or self study preferences, familiarity with the windows based computer applications, gender and preferred way of learning. Earlier studies show that young students of age 7 or 8 are more successful on structured learning materials (Shin, Schallert, Savenye, 1994). Also, learner controlled programs are more successful than the program controlled when the learners are older (Hannafin, 1984). In our study, we have found that older students of age around 40 also prefer linear and more structured instruction. However middle age group students of age around 20-30 usually prefer non-structured and non-linear instruction. This shows us that while younger students and older students prefer more structured and linear way of learning, middle age group students prefer non-linear way of learning. We believe that, such tools can be used in the or outside the classroom to support current methods of instructions. The following section offers recommendations for instructors, designers and for further research.

Implications for the Instructors Since students’ preferences on the learning path of instructions (linear or non-linear) differ, it is not always possible to provide an appropriate method for each student in the classroom. In such cases, the instructor has to choose a method which is common for most of the students. Accordingly, technological tools providing several different alternative ways of instructional methods could help to guide students according to their individual preferences.

Implications for Further Research The factors analyzed in this study need to be evaluated in other learning environments with a different group of students. For example, learners’ familiarity with the most recent technological innovations and older technological applications should be analyzed as separate factors to get a better understanding of the design issues for such tools.

Implications for the Designers The studies carried in the last 20 years showed that older students benefit most from the direct instruction (Volet, 1995). In our study even the average computer usage score of students is 8.4 (Table 5), still the older students preferred direct instruction and they preferred to follow a linear path of instruction. Additionally, the students who have preferred non-linear path of instruction were more familiar with the windows-based computer applications. This shows that the students are not always well prepared for the new technologies, since technological developments occur very rapidly. This could be a big barrier for adapting technology into the traditional educational systems. In order to handle this problem, while designing new instructional systems, involving both the previous technologies and the current new technological approaches at the same time could be helpful. For example, in our study, we have found that individual differences among the students affect their preferences pertaining to linear and non-linear paths. The linear way of learning is more traditional. In that sense, it is not always easy to move learning from linear to non-linear organization. In order to benefit more from the non-linear way of learning and to ease students’ transition between these approaches, providing both methods at the same time and leaving the choice to the learner could be a good strategy. Such an approach helps students to 133

go between these two approaches and to get use to the non-linear instruction. This approach could be applicable to any instructional tool involving more recent technologies. In our study we could not find any relation between students’ previous knowledge on the course content, if they liked working with computers or not as well as their computer experience and their preferences on linear and non-linear instruction. Additionally, these factors need to be tested in other research studies, and further the results of this study need to be supported by other qualitative and quantitative research.

References Barker, P. (1997). Enhancing learning opportunities through electronic course delivery. Paper presented at the EDUTEC 97 Conference, University of Malaga, Spain, 26 October 1997. Barker, P., Giller, S., Richards, S., Banerji, A., & Emery, C. (1993). Interactive Technologies for Language Learning. In Proceedings of the ED-MEDIA '93 World Conference on Educational Multimedia and Hypermedia Florida, USA, 1993, 26-31. Braswell, R., & Brown, J. (1992). Use of interactive videodisc technology in a physical education methods class. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA, USA. Brown, B. W., & Liedholm, C. E. (2004). Student Preferences in Using Online Learning Resources. Social Science Computer Review, 22 (4), 479-492 Chambers, A. (2001). Introduction. In Chambers, A. & Davis, G. (Eds.), ICT and language learning. A European perspective, Lisse: Swets & Zeitlinger. Chan, W. M., & Kim, D. (2004). Towards Greater Individualization and Process-Oriented Learning Through Electronic Self-Access: Project “e-daf”. Computer Assisted Language Learning, 17 (1), 83-108. Chang, C. (2003). Teaching earth sciences: should we implement teacher-directed or student-controlled CAI in the secondary classroom? International Journal of Science Education, 25 (4), 427-438. Charney, D. (1987). Comprehending non-linear text: The role of discourse cues and reading strategies, Hypertext planning committee. In Proceedings of Hypertext ’87, Chapel Hill, North Carolina: The University of North Carolina, 109-120. Cho, Y. (1995). The nature of learner's cognitive processes in learner- and program-controlled hypertext learning environments. UMI, DAI-A 56/06, p. 2207, Publication Number, AAT 9534749. Cuban, L., Kirkpatrick, H., & Peck, C. (2001). High access and low use of technologies in high school classrooms: Explaining an apparent paradox. American Educational Research Journal, 38 (4), 813-834. Dalgarno, B. (2001). Interpretations of constructivism and consequences for Computer Assisted Learning. British Journal of Educational Technology, 32 (2), 183-194. Davies, G. (2001). New technologies and language learning: A suitable subject for research. In Chambers, A. & Davis, G. (Eds.), ICT and language learning. A European perspective, Lisse: Swets & Zeitlinger. Decoo, W. (2003). Language Methods and CALL: redefining our Relations. Computer Assisted Language Learning, 16 (4), 269-274. Ellis, R. D., & Kurniawan, S. H. (2000). Increasing the usability of online information for older users - A case study in participatory design. Instructional Journal of Human-Computer Interaction, 12 (2), 263-276. Gardner, H. (1983). Frames of Mind: The theory of Multiple Intelligences, New York, USA: Basic Books. Gay, G. (1986). Interaction of learner control and prior understanding in computer assisted video instruction. Journal of Educational Psychology, 78 (3), 225-227. 134

Gilbert, J. E., & Han, C. Y. (1999). Adapting instruction in search of ‘a significant difference’. Journal of Network and Computer Applications, 22 (3), 149-160. Goldberg, M. W. (1997). CALOS: First Results from an Experiment in Computer-Aided Learning. In Proceedings of the ACM's 28th SIGCSE Technical Symposium on Computer Science Education. Hannafin, M. J. (1984). Guidelines for determining locus of instructional control in the design of computerassisted instruction. Journal of Instructional Development, 7 (3), 6-10. Hannafin, R. D., & Sullivan, H. (1995). Learner control in full and lean CAI programs. Educational Technology Research and Development, 43 (1), 19-30. Hargis, J. (2000). The Self-Regulated Learner Advantage: Learning Science on the Internet. Electronic Journal of Science Education, 4 (4), retrieved June 30, 2006, from, http://unr.edu/homepage/crowther/ejse/hargis.html. Hirata, Y. (2004). Computer Assisted Pronounciation Training for Native English Speakers Learning Japanese Pitch and Duration Contrasts. Computer Assisted Language Learning, 17 (3-4), 357-376. Hitendra, P. (1998). An Investigation of the Effect of Individual Cognitive Preferences on Learning Through Computer-Based Instruction. Educational Psychology, 18 (2), 171. Hokanson, B., & Hooper, S. (2000). Computers as cognitive media: examining the potential of computers in education. Computers in Human Behaviour, 16 (5), 537–552. Howell, S. L., Williams, P. B., & Lindsay, N. K. (2003). Thirty-two Trends Affecting Distance Education: An Informed Foundation for Strategic Planning, retrieved April 13, 2006, from, http://www.westga.edu/~distance/ojdla/fall63/howell63.html. Kelley, T. M., & Stack, S. A. (2000). Thought recognition, locus of control, and adolescent well-being. Adolescence, 35, 531-551. Kemp, J. E., Morrison, G. R., & Ross, S. M. (1994). Designing effective instruction, NY, USA: McMillan. Kinzie, M. B., Sullivan, H. J., & Berdel, R. L. (1992). Motivational and achievement effects of learner control over content review within CAI. Journal of Educational Computing Research, 8 (1), 101–114. Knowles, M. S. (1975). Self-directed learning: A guide for learners and teachers. Chicago: Association Press. Koper, R., & Olivier, B. (2004). Representing the Learning Design of Units of Learning. Educational Technology & Society, 7 (3), 97-111. Kulik, J. A. (1994). Meta-analytic studies of findings on computer-based instruction. In Baker, E. & O'Neil, H. (Eds.), Technology Assessment in Education and Training, Hillsdale, NJ and Hove, UK: Lawrence Erlbaum. Levy, M. (2001). Coherence and direction in CALL research: Comparative Designs CALL & The challenge of Change in Exeter, Elm Bank. Lonfis, C., & Vanparys, J. (2001). How to Design User-Friendly CALL Interfaces. Computer Assisted Language Learning, 14 (5), 404-417. McManus, T. F. (1997). Self-regulated learning and web based instruction. In Carlson, P. & Makedon, F. (Eds); Proceedings of the World Conference on Educational Multimedia and Hypermedia, VA, USA: AACE. McMillan, J. H., & Schumacher, S. (2001). Research in Education: a conceptual introduction (5th Ed.), Addison Wesley. McNeil, B. J. (1991). Meta-analysis of interactive video instruction: A 10-year review of achievement effects. Journal of Computer-based Instruction, 18 (1), 1-6.

135

Microsoft (2000). Microsoft's clip http://cgl.microsoft.com/clipgallerylive/.

art

gallery,

retrieved

April,

13,

2006,

from,

Roblyer, M. D., & Edwards, J. (2000). Integrating educational technology into teaching (2nd Ed.), Upper Saddle River, NJ, USA: Merrill. Santiago, R., & Okey, J. (1992). The effects of advisement and locus of control on achievement in learnercontrolled instruction. Journal of Computer-Based Instruction, 119 (2), 47-53. Schank, R. C. (1993). Learning via multimedia computers. Technology in Education, 36 (5), 54-56. Shin, E. C., Schallert, D. L., & Savenye, W. C. (1994). Effects of learner control, advisement, and prior knowledge on young students' learning in a hypertext environment. Educational Technology Research and Development, 42 (1), 33-46. Shoener, H. A., & Turgeo, A. J. (2001). Web-Accessible Learning Resources: Learner-Controlled versus Instructor Controlled. Journal of Natural Resources and Life Sciences Education, 30, 9-13. Shyu, H., & Brown, S. (1992). Learner control versus program control in interactive videodisc instruction: What are the effects in procedural learning. International Journal of Instructional Media, 19 (2), 85. SIIA (2000). Research Report on the Effectiveness of Technology in schools, 7th edition, Software & Information Industry Association, Washington, DC, retrieved April, 13, 2006, from, http://www.sunysuffolk.edu/Web/Central/InstTech/projects/iteffrpt.pdf. Silva, R. (1999). Computer-Aided Instruction of Foreign Languages: the case of Beginning Brazilian Portuguese. Paper presented at the National Association for Self Instructional Language Programs (NASILP) Conference, Washington D.C., USA, October 1999. Simonson, M. R., & Thompson, A. (1997). Educational computing foundations (3rd Ed.), Upper Saddle River, NJ, USA: Prentice Hall. Sleight, D. A. (1997). Self-Regulated Learning during Non-Linear Self-Instruction, retrieved April, 13, 2006, from, http://www.msu.edu/~sleightd/srl.html. Spiro, R. J., Feltovich, P. J., Jackson, M. J., & Coulson, R. L (1991). Cognitive flexibility, constructivism, and hypertext: random access instruction for advanced knowledge acquisition in ill-structured domains. Educational Technology, 31 (5), 24-33. Stepp, J. (2002). Student Perceptions on Language learning in a technological environment: Implications for the millennium. Language Learning & Technology, 6 (1), 165-180. Stone III, T. T. (1996). The academic impact of classroom computer usage middle-class primary grade level elementary school children. Dissertation Abstracts International, 57/06-A (Order No. AAD96-33809). Volet, S. (1995). Process-oriented instruction: A discussion. European Journal of Psychology of Education, 10 (4), 449-459. Warschauer, M., & Kern, R. (Eds.) (2000). Network-based language teaching: Concepts and practice, Cambridge: Cambridge University Press. Wooyong, E., & Robert, A. R. (2000). The effects of self-regulation and instructional control on performance and motivation in computer-based instruction. International Journal of Instructional Media, 27 (3), 247.

136

Hsieh, Y.-C. J., & Cifuentes, L. (2006). Student-Generated Visualization as a Study Strategy for Science Concept Learning. Educational Technology & Society, 9 (3), 137-148.

Student-Generated Visualization as a Study Strategy for Science Concept Learning Yi-Chuan Jane Hsieh Department of Applied Foreign Languages, College of Commerce, Ching Yun University, Jung-Li 320, Taiwan Tel: (+886) 3-4581196 ext. 7903 Fax: (+886) 3-250-3014 [email protected]

Lauren Cifuentes Department of Educational Psychology, College of Education and Human Development Texas A&M University, College Station, Texas 77843-4225, USA Tel: (+1) 979-845-7806 Fax: (+1) 979-862-1256 [email protected] ABSTRACT Mixed methods were adopted to explore the effects of student-generated visualization on paper and on computers as a study strategy for middle school science concept learning. In a post-test-only-control-group design, scores were compared among a control-group (n=28), a group that was trained to visualize on paper (n=30), and a group that was trained to visualize on computers (n=34). The paper group and the computer group performed significantly better on the post-test than the control group. Visualization as a study strategy had positive effects whether students used paper or computers to generate their visualizations. However, no significant difference existed between the paper group and the computer group’s scores. Qualitative results indicated that students who were trained to visualize on paper or on computers had positive attitudes toward the use of visualization as a study strategy, and they engaged more in purposeful tasks during study time than the control group. However, several participants in the computer group felt negatively about the use of visualization; they claimed that visualization demanded too much time and effort. Attitudes toward visualization as a study strategy differed within groups indicating that visualization might interact with learner characteristics.

Keywords Visualization, Study strategy, Concept mapping, Science learning

Theoretical Background Constructivist theorists contend that learning occurs when learners actively construct their own knowledge and think reflectively when information and concepts are presented to them (Lee, 1997). Students build their understanding in the context of mentoring from instructors who help them find connections among concepts (Bransford, 2000; Julyan & Duckworth, 1996). “Connections that are obvious to an instructor may be far from obvious to a pupil” (Driver, 1983, p. 2). Such connections are made through experience over time. One strategy for providing students with constructivist learning experiences that also gives teachers access to the students’ understandings for effectively mentoring is student-generated visualization. Student-generated visualization is defined as student-created graphical or pictorial representations of sequential, causal, comparative, chronological, oppositional, categorical, and hierarchical relationships among concepts, whether hand drawn on paper or constructed on computers (Cifuentes & Hsieh, 2004a, 2004b, 2004c; Cifuentes, 1992; Wileman, 1993). Students’ study notes are visualizations if the concepts are presented in the form of direct representations, diagrams, matrices, charts, trees, tables, graphs, pyramids, causal chains, timelines, or even outlines (Cifuentes & Hsieh, 2004a, 2004b, 2004c). Wittrock (1990) stressed that successful comprehension requires learners to actively construct relationships among parts of a text, and between the text and knowledge and experience, and to invest their effort at generating headings, summaries, flow charts, pictures, tables, metaphors, and analogies during or after the reading process. Plotnick (1997) later listed several advantages of constructing visual representations of text: (a) visual symbols are quickly and easily recognized; (b) minimum use of text makes it easy to scan for a word, phrase, or the general ideas; (c) visual representation allows for development of a holistic understanding that words alone cannot convey (p. 3).

ISSN 1436-4522 (online) and 1176-3647 (print). © International Forum of Educational Technology & Society (IFETS). The authors and the forum jointly retain the copyright of the articles. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear the full citation on the first page. Copyrights for components of this work owned by others than IFETS must be honoured. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from the editors at [email protected].

137

One of the visualization techniques commonly used by learners for knowledge construction is concept mapping. Concept mapping is the “process for representing concepts and their relationships in graphical form, providing students with a visually rich way to organize and communicate what they know” (Anderson-Inman & Ditson, 1999, p. 7). As students construct concept maps, they actively revise and manipulate the information to be learned and look for meaningful concept connections (Kinchin, 2000). Concept mapping also “enhances students’ self-esteem through the sense of accomplishment of constructing a map and the subsequent realization that they can extract concepts to construct their own meaning” (Novak, 1998, p. 124). Several tools facilitate generating visual representations of concepts, including paper and pencil, pen or colored markers, computer graphics software, or concept mapping software, such as Inspiration™ or Microsoft Visio™. Constructivist theorists indicate that computers can be the “cognitive tools” to support learners’ thinking processes (Jonassen, 2000; Lajoie, 1993). Namely, computers can be used as tools for “analyzing the world, accessing information, interpreting and organizing individuals’ personal knowledge, and representing what they know to others” (Jonassen & Reeves, 1996, p. 694). Anderson-Inman (1996) noted that the creation of visualization on computers made the learning process more accessible to students, and it helped mitigate the frustration and confusion felt by students while constructing concept maps with the paper-and-pencil approach. The use of computers for constructing visual representations of concepts offers several advantages. Students can manipulate graphical representations with ease, and can perform subsequent revision and dynamic linkage of concepts. In addition, computers support learners’ production of sophisticated-looking graphics regardless of their level of artistic skill (Cifuentes & Hsieh, 2004a, Anderson-Inman & Ditson, 1999; Anderson-Inman & Zeitz, 1993; Lanzing, 1998). Researchers, such as Anderson-Inman (1992), and Anderson-Inman, Knox-Quinn, & Horney (1996), found that the use of computer-based concept mapping tools as a vehicle to synthesize information during study time had a positive impact on learning. Computer graphics software, such as AppleWorks™ and PhotoShop™ allow students to access a variety of tools that help them draw and paint objects to visually organize and represent what they know. Jonassen (1996) indicated that drawing and painting programs are powerful expressive tools for learners to visually articulate what they know, and to make their knowledge structures more easily interpretable by other viewers. Skilled learners can use drawing and painting tools for image generation to facilitate making sense of their ideas and producing an organized knowledge base (Jonassen, 1999). When the drawing and painting programs are used as “cognitive tools” for mindful engagement, learners’ thinking processes can be supported, guided, and extended (Derry & Lajoie, 1993). Nevertheless, there are several concerns regarding the use of computers during study time. Downes (2000) noted that children between 5 and 12 years old usually perceive the computer as both a tool and a toy. When children are thinking and using the computer as a toy, it is “playable.” Such a conception of the computer enables children to complete their work-related tasks more through playful means. Without adult guidance, children who use computers during study time are more likely to spend their time fiddling around instead of concentrating on purposeful tasks. Computer-related distractions, such as technological problems, and multimedia, or software features also have an impact on students’ effective use of computers during study time. Generally, research studies indicate that visualization on computers has improved student learning. In studies in which computers have been a distraction, those researchers concluded that computers would have been beneficial given controlled elements in the school settings, explicit instruction in when, what and how to visualize concepts on computers, the selection of the appropriate computer tool with students’ high competency in its usage, and frequent feedback regarding accuracy of students’ visualization. In this study, further investigation into the effect of the use of computer-based drawing and painting tools on learning was conducted. Each of the concerns regarding the use of computers during study time was addressed in the student visualization training.

Research Questions This study explored the use of visualization as a study strategy for middle school science concept learning. The relative effectiveness of student-created visualization on paper and on computers was investigated. The research questions addressed through the quantitative method included: (1) Did students trained to visualize on paper perform better on a comprehension test than those not trained to do so? (2) Did students trained to visualize on computers (using MicrosoftTM drawing and painting tools) perform better on a comprehension test than those who not trained to do so? (3) Was there any difference on comprehension posttest scores between students who trained to visualize on computers (using MicrosoftTM drawing and painting tools) and those who trained to visualize on paper? 138

Other questions addressed through the qualitative method were listed as the following: (4) What was the difference among groups in their level of engagement during study time? (5) What were the elements in the school setting and learners’ behaviors that might contribute to the effectiveness or lack of effectiveness of the use of MicrosoftTM drawing and painting tools to generate visualizations during study time? (6) What were students’ attitudes toward generating visualizations during study time? (7) Had students ever been exposed to the visualization skills that were identified in the provide visualization workshops? Based on previous studies, we expected that students who received visualization training on either paper or computers would outperform students who did not receive training. We provided visualization training to help the participants develop the abilities to organize, to monitor, and to control their generative processes. MicrosoftTM drawing and painting program was selected as a visualization tool because the students were competent in using those tools.

Methods This study used the quantitative and qualitative methods to investigate the research topic. The participants were 8th graders (N=92) in six intact science classes at a rural, public, junior high school in the Southern Texas. Two classes were randomly assigned to be the control group, two classes to be the visualization/paper group, and two classes to be the visualization/computer group. All participants received the given treatment as part of their curricular activity in regular science class periods. Because of the lack of random selection and random assignment of individuals to groups, evidence of groups’ equivalence was especially important. According to the chi-square tests results, the three groups did not differ across age or ethnicity, but did differ in gender. The visualization/computer group had significantly more male students than the other two groups (p< .05). Further, in order to determine whether the three groups differed in their perception of the five biological topics (Energy, Cells, Homeostasis, Coordination, and Transport in Plants and Animals), students were asked to report the extent (I knew none/some/a lot of/all of the information) to which they had been previously exposed to the information presented in the studied essays after the treatment. The frequency counts of the extent to which the students had been exposed to the studying content in studying essays was shown in a 5 x 4 chi-square contingency table. Five Pearson chi-square tests results indicated that there were no significant differences among the proportion of students in the three groups who checked the extent to which they had been exposed before to the five biological topics. Accordingly, the students in each group did not differ in their prior knowledge on the content in studying essays. It was suspected that the participants’ prior knowledge of the content shown on the science essays did not confound their comprehension test scores. In addition, in order to gain a better understanding of the participants’ computer use at school and at home, a “Computer Use Survey” was administered prior to the treatment. The results indicated that the three groups did not differ significantly in the frequency of using a computer at school and at home, in the amount of time spent in using a computer at school and at home, in the number of computer courses taken in the past, in their performances in previous computer course(s), in the average amount of time using a computer at home each time, and in the frequency of using computer tools to support various learning tasks. Procedures A posttest-only control group design was used. The control group attended 100-minute non-visualization workshop. Each day, students watched a 25-miunte science-related videotape in their regular classroom. Videos were carefully selected to assure that content did not include concepts to be covered in the experimental studying materials. On the first, second, third, and fourth days after the workshop and on days five and six, students were given five different science essays for unguided and independent study. When students finished their studying, they handed in their study notes to the test administrator, and they were given a test covering comprehension of the science essays. As for the visualization/paper group, students attended a 100-minute paper-form visualization workshop, which lasted 25-miunte per day. On days one, two, three, and four, students were instructed in how to identify each of the three types of structures (cause-effect, sequence, and comparison-contrast) and in how to use visual conventions (fishbone diagram, flowchart, matrix/table) to represent these structures on paper. On the first, second, third, and fourth days after the workshop and on days five and six, students were given the same science essays to study as the control group. Students had to use their learned visualization skills to graphically represent 139

the interrelationships among concepts on paper. When students finished their studying, they handed in their study notes to the test administrator, and they were given the same comprehension test as the control group. The visualization/computer group attended a 100-minute paper-form visualization workshop, which lasted 25 minutes per day. During the workshop, they were asked to perform their visualizing work on computers, using MicrosoftTM drawing and painting tools. The visualization workshop was delivered in the school computer lab. The content and the structure of the workshop were the same as for the visualization/paper group except that the teacher modeled visualization on computers rather than on paper and students generated their visualizations on computers. On the first, second, third, and fourth days after the workshop and on days five and six, students were given the same science essays to study as the control group. Students had to use the learned visualization skills to graphically represent the interrelationships among concepts on computer. When students finished their studying, they handed in their study notes to the test administrator, and they were given the same comprehension test as the control group. After the completion of the comprehension posttest, all participants were asked to verbally describe their study strategy used to prepare for the comprehension posttest. The visualization/paper and the visualization/computer groups were further asked to respond two questions: (1) How did you feel about the generation of visualizations during study time? Did it help you learn the content? and (2) Have you ever learned the visualization strategy before? During the entire study, the participants’ science teacher delivered the treatments. During the workshop, one of the researchers helped this teacher in guiding students in how to properly represent what they learned. The science teacher and the researchers also provided feedback to the visualization groups regarding the appropriateness of their visualizations during their study time. Following the treatments, the researchers administered the comprehension posttest for all groups. Essays studied by groups and the comprehension posttest The five science essays for students to study were excerpted from an American biology textbook, Biology: the web of life (Strauss & Lisowski, 1998), adopted in the 9-12 grade Texas science curriculum. The illustrations were eliminated in order to create the text-based document. The higher level reading passages were used to assure a high level of difficulty and a lack of student exposure to the content. The first day after the treatment video-watching or visualization workshop, all participants were asked to spend a maximum of 20 minutes studying an essay on science concepts on “Energy”, and then they took a paper-andpencil comprehension test containing five multiple-choice items. On the second day, after the video-watching or visualization workshop, all participants were asked to spend a maximum of 20 minutes studying an essay on “Cells”. After that, a paper-and-pencil comprehension test containing two multiple-choice items were administered. Likewise, on day three, after the video-watching or visualization workshop, all participants were asked to spend a maximum of 20 minutes studying an essay on “Homeostasis”, and then they took a paper-and-pencil comprehension test containing eight multiple-choice items. On the fourth day, after the video-watching or visualization workshop, all participants were asked to spend a maximum of 20 minutes studying an essay on “Coordination”. After students studied the written verbal science text, a paper-and-pencil comprehension test containing six multiple-choice items was administered. On day 5, all participants were asked to study an essay on “Transport in Plants and Animals.” They had a maximum of 50 minutes to study on the fifth day. When the study period time was over, the researchers collected all students’ essays. On the sixth day, the researcher handed the same essay back to the students, and they had a maximum of 35 minutes to complete their studying. After that, students were given a comprehension test containing nine multiple-choice test items to measure their understanding of the reading materials that they had spent time studying since day 5. The test items administered on each day were combined to form a 30-item multiple choice comprehension test. Students did not receive the results of their comprehension posttest each day; instead, the researchers graded each participant’s total testing items to form their final scores in the last day of the study. The test items were derived from the Taiwan Educational Department Test Bank for Middle School Biology (Taiwan National Institute for Compilation and Translation, 2000) that evaluated students’ higher level of thinking skills, which 140

required them to apply comprehension, analysis, and synthesis skills (Bloom & Krathwohl, 1956) to answer questions. The test items were constructed and validated by three content experts to be appropriate for this study. For the purpose of scoring students’ responses to the comprehension test items, 3.5 points were given for a correct answer and no credit was given for incorrect or unanswered questions. The reason to assign 3.5 points to a correct answer was for the test to have face validity, and have no detrimental result on the total score. The comprehension test had a reliability coefficient of 0.71 (coefficient alpha).

Results One-way analysis of variance indicated that there was a significant difference among the three treatment groups, F (2, 89) = 20.363, p 0.05. Similar tests conducted to detect diferences due to gender and spatial experience showed all the groups were statistically equivalent except for spatial visualisation pre-test measure indicating high spatial experience participants were more accurate than the low spatial experience counterparts. The absence of gender differences for both measures of spatial ability especially in mental rotation were most surprising as most literature reported otherwise. Table 1 summarises the means and standard deviations of the pre-test and post-test measurements. Table 1: Means and standard deviations for the spatial ability measures Mental Rotation Spatial Visualisation Conditions (max 30) Reaction time (secs) (max 32) Pre Post Pre Post Pre Post Experimental 1 Females High exp. 13.70 19.40 24.00 27.10 124.88 100.67 (4.69) (2.63) (2.71) (1.20) (42.81) (32.38) Low exp. 12.46 16.15 19.77 22.54 118.43 126.90 (2.66) (2.64) (1.79) (2.50) (40.51) (32.00) Males High exp. 15.60 20.10 24.60 27.00 113.31 94.70 (4.03) (3.96) (2.37) (1.25) (25.68) (24.41) Low exp. 14.23 19.15 19.92 22.15 113.98 119.46 (3.92) (3.83) (1.15) (2.11) (42.27) (32.70) Total 13.91 18.57 21.78 24.39 119.10 97.68 (Females+Males) (3.86) (3.57) (2.97) (3.01) (34.87) (28.08) Experimental 2 Females High exp. 14.00 17.88 22.13 24.00 142.32 119.56 (4.04) (2.75) (2.36) (2.92) (34.06) (30.16) Low exp. 12.93 15.07 21.07 22.67 124.58 117.87 (2.63) (2.02) (3.56) (3.41) (38.07) (31.90) Males High exp. 15.67 20.00 22.11 24.11 115.80 105.58 (4.00) (2.87) (3.62) (3.57) (39.86) (43.79) Low exp 13.36 17.50 20.00 23.14 137.96 129.61 (3.43) (2.07) (3.68) (3.61) (39.53) (28.43) Total 13.78 17.26 21.13 23.33 130.03 119.34 (Females+Males) (3.46) (2.89) (3.44) (3.37) (38.24) (33.23) Control Females

High exp. Low exp.

Males

High exp. Low exp.

14.83 (4.31) 12.82 (2.74)

18.67 (2.07) 14.59 (3.02)

19.83 (3.54) 19.82 (3.50)

21.00 (3.80) 22.35 (3.32)

138.52 (29.80) 141.85 (38.21)

117.97 (27.36) 127.74 (36.85)

15.70 (3.77) 14.31 (3.17)

18.00 (4.11) 16.92 (2.81)

19.60 (3.37) 20.69 (3.04)

21.80 (3.68) 21.85 (3.58)

137.79 (40.33) 150.28 (24.25)

124.29 (40.83) 131.92 (25.32) 153

Total (Females+Males)

14.13 (3.40)

16.52 (3.43)

20.02 (3.27)

21.91 (3.44)

142.92 (33.54)

126.90 (33.02)

Spatial Visualisation Three-way interaction between method of instructions, gender and level of spatial experience was not found to be significant, F(2,126) = 0.58, p = .56 by the analysis of variance performed. Similarly, no three-way interactions among the three independent variables were found to be signififant. The main effect for method of instructions was significant, F(2,126) = 3.47, p = .03, showing better performances for participants receiving training in EDwgT compared to the other two methods. Gender factor produced significant main effect, F(1,126) = 9.91, p = .002 indicating males to be more accurate in spatial visualisation tasks than their female counterparts. A highly significant main effect of spatial experience was detected, F(1,126) = 21.60, p = .0005 revealing better spatial visualisation performance for high spatial experience participants compared to their low spatial experience counterparts. Table 2 summarises the results of the main effects and interactions for spatial visualisation post-test. Table 2: ANOVA for the Spatial Visualisation Test df Sum of Squares Mean Square Method 2 60.96 30.48 Gender 1 87.16 87.16 Spatial Experience 1 190.01 190.01 Method x Gender 2 11.28 5.64 Method x Spatial Experience 2 2.01 1.01 Gender x Spatial Experience 1 27.84 27.84 Method x Gender x Spatial 2 10.22 5.11 Experience Error 126 1108.37 8.80 *p