Transactional distance in an undergraduate ... - Semantic Scholar

Knowledge-Based Systems 71 (2014) 41–51

Contents lists available at ScienceDirect

Knowledge-Based Systems journal homepage: www.elsevier.com/locate/knosys

Transactional distance in an undergraduate project-based systems modeling course Niva Wengrowicz a,b, Yehudit Judy Dori c,a, Dov Dori d,b,⇑ a

Department of Education in Science and Technology, Technion, Israel Institute of Technology, Haifa, Israel Faculty of Industrial Engineering and Management, Technion, Israel Institute of Technology, Haifa, Israel c Department of Electrical Engineering & Computer Science, Massachusetts Institute of Technology, Cambridge, MA, USA d Engineering Systems Division, Massachusetts Institute of Technology, Cambridge, MA, USA b

a r t i c l e

i n f o

Article history: Available online 13 June 2014 Keywords: Project-based learning Systems engineering Object Process Methodology (OPM) Engineering education Transactional distance Collaborative learning Visualization Conceptual modeling

a b s t r a c t Assessing the level and quality of collaboration between students working in project teams is a complex task. The main goal of our study was to develop and validate an online questionnaire for assessing the quality of distance teamwork collaboration in a project-based systems modeling course. The research goals included assessing the transactional distance (TD) perceptions among peer students who had collaborated in jointly constructing conceptual system models of projects carried out by distant researchers and the TD between the students on one hand and the distant researchers on the other hand. The research questions were aimed at validating the TD questionnaire as a tool for assessing TD. The research population included undergraduate students who participated and collaborated via a visualization-based environment as part of the EU VISIONAIR infrastructure project. The students interacted both among themselves and with remote researchers across Europe. Reliability and inter-correlation tests have indicated internal structure validity and reliability of the TD questionnaire. Correlation with other student outcomes indicated content validity by criterion. Experiencing visualization-based environments was a key factor in student satisfaction. Based on our findings and the collaboration literature, TD may serve as an alternative assessment tool for evaluating the quality of collaboration among peer students and researchers. Ó 2014 Elsevier B.V. All rights reserved.

1. Introduction Modern engineering education programs aim to endow students with a broad base of knowledge, skills, and attitudes necessary to become successful young engineers [7]. The engineering education goal is to train students to be able to conceive, design, implement, and operate complex, value-added engineering products, processes, and systems in modern, team-based environments. An important consequence of this trend is that beyond the exposure of students to the body of technical knowledge and the product, process, and system building skills specific to their engineering profession, they also need to acquire interpersonal skills and additional personal and professional traits. These insights formed the basis for the CDIO – Conceive Design Implement Operate educational framework [6]. ⇑ Corresponding author at: Engineering Systems Division, Massachusetts Institute of Technology, Cambridge, MA, USA. Tel.: +1 6172301662. E-mail addresses: [email protected] (N. Wengrowicz), [email protected] (Y.J. Dori), [email protected] (D. Dori). http://dx.doi.org/10.1016/j.knosys.2014.05.022 0950-7051/Ó 2014 Elsevier B.V. All rights reserved.

The CDIO approach is designed to raise the quality of engineering education programs [30], and most of the CDIO features are related to experiential learning [6]. This approach emphasizes the importance of active and hands-on learning in both the classroom and modern learning workspaces. CDIO enables students to be exposed to the experiences that they will encounter as engineers during their professional lives. To enable these kinds of experiences, the CDIO syllabus contains significant elements of projectbased learning [7].

1.1. The research goal The main goal of this study was to develop and validate an online questionnaire for assessing the quality of distance teamwork collaboration on carrying out projects in the spirit of CDIO by measuring the perceptions of peer students and distant researchers during their interactions regarding the transactional distance (TD), a key concept of this work, which is explained in detail in the sequel.

42

N. Wengrowicz et al. / Knowledge-Based Systems 71 (2014) 41–51

In what follows we present the theoretical background of the teaching and learning methods – project-based learning and collaboration, the assessment method of such collaboration, and the concept of transactional distance. 1.2. Project-based learning Project-based learning (PBL) is a teaching method in which students are given realistic problems characterized by not having a single correct answer. Guided through a process of analyzing the problem, researching alternatives, arguing for and against them, the students present a recommended solution [22]. Proponents of the PBL method clarify that it provides real-world and real-time learning opportunities that replicate the type of problems students will encounter and solutions they will use throughout their academic and professional lives [12,20]. PBL is characterized by authentic investigation, collaboration among peers, the use of technology to support inquiry processes, and delivery of an end product [26,27,42]. Through their active participation in the project execution process, students are encouraged to form original opinions and express individual standpoints. The project fosters students’ awareness of the complexity of systems they would tackle and encourages them to explore the consequences of their own values [48]. PBL was tested at the levels of elementary, junior, and high school, as well as in higher education, and was found to be effective for promoting self-efficacy, meaningful learning, and the development of higher order thinking skills [1,33,44,45]. Collaboration, discussed in the next section, is a key feature of PBL. 1.3. Collaboration Thomson et al. [46] based their theoretical and empirical definition of collaboration on a comprehensive review of the literature and a systematic analysis of multiple definitions of collaboration across many disciplines. They have defined collaboration as a process in which autonomous actors interact and jointly create rules and structures governing their relationships. They added that collaboration processes involve shared norms and mutually beneficial interactions. Dillenbourg [8] defined collaborative learning as ‘‘two or more people [who] learn or attempt to learn something together.’’ Johnson [24] emphasized that collaborative learning pedagogy has shifted the focus from the teacher–student interaction to the role of peer relationships. Collaborative learning is part of the constructivist approaches, also known as active approaches and student-centered pedagogy theories [19]. Pedagogical methods, including collaborative learning, which build on these theories, create learning situations, such as laboratories, field studies, simulations, and case studies with group discussions, which enable learners to engage in active exploration and/or social collaboration [19]. These theories present learning as a social process that takes place through communication with others. The learner actively constructs knowledge by formulating ideas into words and graphic illustrations, and these ideas are built upon through reactions and responses of others. In other words, Hiltz and his colleagues [19] claimed that collaborative learning is not only active; it is also interactive. Dillenbourg [8] indicated that collaborative learning mechanisms directly affect cognitive processes. The first of these mechanisms is the conflict or disagreement mechanism, which is based on social factors that prevent learners from ignoring conflict and force them to seek additional information and find a solution. Internalization is another mechanism, in which the concepts, conveyed by the interactions with more knowledgeable peers, are progressively integrated into the learner’s knowledge structures. When integrated, they can be used in the student’s own reasoning

mechanisms. Finally, the self-explanation mechanism is founded on the finding that while less knowledgeable members learn from the explanations of more advanced peers, the more able peers also benefit, because the need to generate and deliver an explanation improves the knowledge of the explainer. Explaining to others may be more beneficial to the explainer when the material is complex than when the material is simple. Such self-explanation processes are the essence of the interactions that occur naturally in collaborative learning. Based on the variety of definitions and explanations of collaborative learning, we define collaborative learning as a learning process in which learners acquire knowledge and gain understanding via a mutually beneficial explanation process. One of the main principles of CDIO and PBL is the ability to assess and assure the quality of the learning and collaboration process and outcomes [30].

1.4. Assessment of collaboration Assessment, defined as a collection of information on students’ outcomes [38,39], is commonly applied to evaluate students. Alternative assessment is applied to evaluate students on the basis of their active performance in using knowledge in creative ways to solve worthy problems [39]. Embedded assessment comprises recurring activities that are indistinguishable to students from instructional activities, enabling a comparison of students’ current understanding with the expectations of the curricular goals [25]. The combination of alternative and embedded assessment can potentially yield a powerful set of tools for measuring learning effectiveness, enhancing learning outcomes [29], and fostering higher order thinking skills [13]. As noted, a main goal of engineering education is to train the engineers of the future to collaborate and work as part of a team. Since assessment is an important part of education processes in general, and since it serves as a means for quality assurance of CDIO in particular [30], it becomes necessary to measure the quality of learning that takes place during peer collaboration. Although it might be possible to extend the general definition of collaboration of Thomson et al. [46] for collaborative learning, it was not possible to use this definition operationally for measuring the quality of collaborative learning. These researchers claimed that ‘‘few instruments to measure collaboration exist, and those that do are difficult to adapt outside the immediate context of a particular study’’ [46]. Accordingly, they have conceptualized the collaboration process in terms of five dimensions: governance, administration, mutuality, norms, and organizational autonomy. Based on these dimensions they developed a collaboration assessment instrument. The variables and items in this instrument indicate that it measures collaboration between organizations, but it does not measure the mutually beneficial explanation process occurring among the collaborating stakeholders and their learning process outcomes. In order to measure the effects of collaboration in their newly developed learning environment, Hwang and Karnofsky [23] divided collaboration into three dimensions: collaborative situations, interactions, and processes. For measuring these collaboration dimensions, they observed and inquired about specific uses of their software and hardware, such as patterns of keyboard strokes and mouse movements, patterns of monitor use, and patterns of shared meaning tools. Their operational definition was adapted to their unique environment and related to the use of the environment in the context of collaborative learning. However, it did not measure the mutually beneficial explanation process which assesses the quality of collaboration among the collaborating people themselves.


Hiltz and his colleagues [19] measured the learning perception, length of reports, and quality of the proposed solution for evaluating and comparing the quality of collaborative learning outcomes between different leaning environments. Transactional distance, defined and explained next, is another teaching–learning outcome that can be measured and serve as an elaborate collaboration metric. 1.5. Transactional distance Transactional distance (TD) is one of the major theories lying at the basis of distance teaching and learning. Developed by Moore [34–36], the TD theory suggests that rather than the physical or temporal distances that separate the participants, the most crucial role in distance education is pedagogy. Moore [34,35] defined TD as a psychological, pedagogical and communicational distance, which might lead to misunderstanding among the collaborators. Distance education environments feature unique characteristics of separation between the participants, and therefore we can expect unique corresponding teaching and learning behaviors [34]. The physical separation leads to a sense of distance, which opens the door for misunderstandings between the teacher and the students, as well as among student peers. Moore [34,35] claimed that TD is a relative concept. It can serve as an attribute of leaning environments; it can be measured in various kinds of teaching–learning environments and enable comparison among such environments. Giossos et al. [17] attempted to position the TD theory within an epistemological framework of realism. They referred to TD as an outcome of the teaching–learning activity, which is a specific, subjective, and personal distance, resulting from the teaching and learning activities, and affecting the people involved in these activities. They emphasized that TD is not just an environmental attribute, but a distance that the participants experience and perceive. In other words, TD is not an abstract understanding distance, but rather a specific, subjective, and personal distance, a byproduct of the explaining activity that takes place during the teaching–learning process. Thus, expressions such as ‘‘You don’t understand me’’ or ‘‘You are not following’’ indicate large TD due to absence of mutual understanding or shared perception of ideas, feelings, or situations. Adopting the realistic epistemology of Giossos and his colleagues [17,47] have suggested that students making an effort to understand the learning material are influenced by the teaching activity. Their effort, if not successful, makes them feel a subjective distance. The teacher, who tries to help the students understand the subject matter, is also influenced by the teaching activity, and failed attempts on the teacher’s part cause subjective distance feelings on her or his side as well. Accordingly, Wengrowicz and Offir [47] have defined teacher transactional distance (TTD) as a teacher’s perceptions of TD. The definition of TTD includes a teacher’s perceptions towards the teaching process, evaluation of one’s ability to communicate with the students, and the level of satisfaction from the teaching process in various settings. 2. Research method This study focuses on assessment of TD in emerging projectbased learning and visualization-rich collaborative course environments, where the distinction between the roles of the teacher and the students becomes blurred, as it is replaced by peer students, collaborating both with each other and with distant researchers. Following Moore [34,35], we define TD theoretically as the combination of pedagogical, psychological, and communication distances, which may lead to misunderstandings. Our operational

43

definition of TD is derived from the approach of Giossos and his colleagues [17]. We examined the students’ TD perceptions as a result of the collaborative distance and face-to-face learning processes based on students’ evaluation of their ability to (1) communicate with their peers and with researchers aboard—the communication TD, (2) know and explain whether and to what extent they understood their peers and vice versa—the pedagogical TD, and (3) their satisfaction from the collaborative environment in the context of explaining and learning—the psychological TD. The research findings are expected to validate an assessment instrument described below and promote understanding of TD among peers who collaborate and cooperate in carrying out projects in visualization-rich environments from a distance. In the next section, we introduce Object-Process Methodology, one of the two modeling methodologies in the study reported in the paper and the technical infrastructure used during this study. 2.1. Object-Process Methodology Science and technology are learned and developed through exchange of information, much of which is presented visually. Pavio [40], Mathewson [31], and Mayer [32] suggested that human cognition is divided into two major processing subsystems: the verbal and the non-verbal, and that knowledge is represented and manipulated through visual and verbal channels. Dori and Belcher [11] claimed that science teaching that uses the visual and verbal channels together can enhance learning and understanding processes and improve students’ learning outcomes. Object-Process Methodology (OPM), developed by Dori [9], is a holistic graphical and textual paradigm for the representation and development of complex systems in a formal yet intuitive framework. OPM enables systems engineers to design and represent systems simply in a single model, expressed in both graphics and equivalent natural language. By using a single holistic hierarchical model for representing structure and behavior, clutter and incompatibilities can be significantly reduced even in highly complex systems, thereby enhancing their comprehensibility. Any OPM model comprises only two elementary building blocks: stateful objects – things that exist and represent the system’s structure, and processes – things that happen to the system’s objects and transform them. Processes transform objects by creating them, consuming them, or changing their states. OPM has a single diagram type, with which one can model the function, structure, and behavior of a system in virtually any domain and at any level of complexity by recursively refining processes of interest and objects that these processes transform. Two semantically equivalent modalities, one graphic and the other textual, are used to describe an OPM model. The graphical OPM model is translated on the fly to a subset of natural English, complementing the visual representation with a textual one, catering to ‘‘left brainers’’ and ‘‘right brainers’’ alike. This combination makes it ideal for collaboration and for communicating knowledge and ideas. Comparisons between OPM and other standard languages have pointed out to benefits of OPM compared with other conceptual modeling languages [41,43] and synergies [18]. OPM is in the process of becoming ISO Standard 19450 [3]. In the experiments presented and discussed later in the paper, OPM was a used as a modeling methodology in the visualization-rich peers and distance collaboration environments. 2.2. Research setup The research was conducted within the framework of a project-based system modeling course, Mini-Project in Industrial Engineering, taught at the Faculty of Industrial Engineering at the Technion, Israel Institute of Technology. The course aims to expose

44


Fig. 1. The System Diagram (SD) – the top-level object-process diagram of the model constructed by the students in the course.

undergraduate industrial engineering and management students in their third semester to working in a real international projectbased environment. This collaboration was carried out by engaging the students in jointly constructing a conceptual model expressed in terms of an OPM model. All the projects the students worked on were part of the EU VISIONAIR1 project, aimed at creating a European infrastructure for conducting state-of-the-art research in visualization. 2.2.1. VISIONAIR description VISIONAIR is a European funded infrastructure project that grants researchers access to high level visualization facilities and resources. Among other objectives, VISIONAIR enables transnational access (TNA) to the 29 consortium labs across Europe with the objective of supporting visualization-related research by helping researchers across Europe with the definition, modeling, and subsequent execution of research projects they propose. Research groups interested in visiting one of the labs of VISIONAIR start by submitting a research proposal. The proposal is reviewed, and if accepted, the visit takes place and the project defined by the proposer carried out in collaboration with the researchers at the host laboratory. Both physical access and virtual services are offered by the infrastructure, free of charge, based on the excellence of the project proposal submitted. 2.2.2. Course conduct and resulting model One of the students’ projects carried out during the course called for modeling the trans-national access (TNA) process in order to document it and provide a basis for an information management system for TNA within VISIONAIR. Along the course, there were three IP-based virtual conference meetings between the course students and their professor on one side, and the customers—the distant researchers—on the other side. The distant 1

http://www.infra-visionair.eu/.

researchers described to the students the TNA process and presented the requirements. The students presented to the researchers the model as it evolved, and the distant researchers provided feedback. In what follows, we briefly present the top-level parts of the OPM model the students constructed during the course. Fig. 1 presents a screenshot of OPCAT- object-process case tool [10], showing the first diagram of the OPM model. The top part of Fig. 1 shows the system diagram (SD) – the top-level Object-Process Diagram (OPD) of the model. The function of the system, as expressed by the central ellipse representing the main process of the system, is Visualization-based Trans-National Researching. Around it are the main objects, represented as boxes: Research Group – the group who visits the laboratory of the VISIONAIR Consortium, the Project Set – the set of Project Proposals, the Knowledge & Expertise that is expected to be gained by the Research Group, and the Project Assessment Report, which summarize and analyze the performance and contribution of the visualization-based trans-national researching process. The bottom part of Fig. 1 is the Object-Process Language (OPL) paragraph, which consists of OPL sentences generated automatically by OPCAT in response to each graphic input of the modeler. For example, the bottom sentence is: Visualization-based TransNational Researching yields Project Assessment Report, Project Proposal, and Knowledge & Experience. As the OPD shows, these are indeed the tree outputs of the Visualization-based TransNational Researching process. In Fig. 2, the Visualization-based Trans-National Researching process from the system diagram of Fig. 1 is in-zoomed, showing three subprocesses: Pre-Project Processing, Project Executing, and Project Assessing. Fig. 3 shows the process Pre-Project Processing from Fig. 2 in-zoomed, exposing its four subprocesses. As this pattern demonstrates, an OPM model comprises a tree of OPDs, each resulting from zooming into some process in its ancestor OPD. All the OPDs are part of the OPM mode; they are


45

Fig. 2. The visualization-based trans-national researching process from the system diagram of Fig. 1 is in-zoomed, showing three sub processes: pre-project processing, project executing, and project assessing.

consistent and together contain all the model facts that are necessary to understand the system’s function, structure, and behavior. 2.3. Research sample and design The students (n = 17) in the Mini-Project course, who were part of the VISIONAIR effort, worked initially in two different groups (Group 1, n = 7; Group 2, n = 10), and were asked to conceptually model one of the VISIONAIR projects, as illustrated in Fig. 4. The students in each group collaborated with their peers, as well as with the distant researchers who defined the requirements of the project. The student–peers collaboration tools were emails, Skype conference calls, SMS, chat rooms and OPCAT. The student–researcher collaboration tools were emails, Skype or IP-based video-conference that was set in advance (there were three organized distance meeting in total), and OPCAT. The collaboration among peers was more flexible compared to the collaboration with the distant researcher. For data collection, we used the mixed study method and adopted qualitative and quantitative approaches, including webbased questionnaires, semi-structured interviews, and content analysis for the written sources. These sources included reflection reports, open-ended responses to questions, project proposals, and project summary reports.

Following Dori and Belcher [11], who have shown that visualization technologies can support meaningful learning in an undergraduate electromagnetism course at MIT, our study adopts a mode of assessment that is commensurate with the new visualization technologies developed and applied within this EU project. We also examined various aspects of the communication and interaction among students and between the students in Israel and researchers in France and the UK via video conferencing sessions of various kinds. For the purpose of this assessment, we have developed and analyzed a TD questionnaire to assess the effectiveness of using various visualization technologies for collaborative PBL among the undergraduate students. The questionnaire, which was validated by three experts in educational technologies, measures students’ perceptions of TD using the following three factors. (1) The communication transactional distance (TD-communication), which describes the students’ subjective feelings regarding their ability to communicate with others and the ability of others to communicate with them while working on the project. (2) The understanding transactional distance (TD-understanding), which describes the students’ subjective feelings regarding their ability to explain themselves using a technology-rich learning environment and to know whether others understood them.

46


Fig. 3. The process pre-project processing from Fig. 2 is in-zoomed, showing its four subprocesses.

Fig. 4. The two groups of students who collaborate with their peers and with the distant researchers (ESML stands for Enterprise Systems Modeling Laboratory, http://esml.iem.technion.ac.il/, where the students carried out the two projects).

(3) The satisfaction transactional distance (TD-satisfaction), which describes the students’ subjective feelings regarding their satisfaction from the learning environment of the course, the tools at their disposal, and the manner in which these tools contributed to their ability to collaborate and work on the project. The close-ended part of the TD questionnaire contained 30 items, of which 15 were related to the students’ peer-to-peer collaboration TD (PRTD) and 15—to the student–researcher collaboration TD (DRTD). Fig. 5 displays the three TD measures, TD-communication, TD-understanding, and TD-satisfaction, in the context of the two types of collaborations, PRTD and DRTD. Each factor consists of five items. The TD-communication factor contains items such as Our group members have several ways to communicate with each other and We have several ways to communicate with the researchers. Examples of the TD-understanding items

include After giving an explanation to my peers, I could tell whether they had understood me and After giving an explanation to the researchers, I could tell whether they had understood me’’. Finally, examples for TD-satisfaction items are I am satisfied with my peers’ collaborative work and I am satisfied with the researchers’ collaborative work. The students were asked to indicate their agreement with the statements on a Likert scale from 1 (agree to a very small extent) to 5 (agree to a very great extent). They could also choose 0 for any irrelevant item. Internal consistency (Cronbach’s alpha) was calculated to determine the reliability of the questionnaire. In this test, the internal consistency coefficient was calculated for all the three PRTD types combined (a = 0.89) and for each PRTD type separately (aPRTD-communication = 0.77; aPRTD-understanding = 0.86; aPRTD-satisfaction = 0.85). The internal consistency coefficient was calculated also for the student–researcher collaboration TD items (DRTD, a = 0.96) and for each DRTD measure separately (aDRTD-communication = 0.90; aDRTD-understanding = 0.90; aDRTD-satisfaction = 0.89). Based on the findings of the reliability test, eight measures were calculated for each participant. The scores ranged from 1 to 5, where a higher score means higher participant’s perception of closeness, i.e., perception of a smaller transactional distance. The questionnaires’ open-ended part consisted of 12 questions that focused on students’ perceptions about distance-dependent understanding and collaborative working. In addition, students were asked to write a project summary and a reflection report, and to respond to a semi-structured interview. The students’ responses were read, reread, and gradually analyzed from a descriptive–interpretive perspective. In order to establish research trustworthiness, three researchers were involved in data analysis and interpretation, applying investigator triangulation. 2.4. Research objectives and hypotheses Our research objectives were to (1) develop and validate an online questionnaire for assessing TD perceptions of collaborating

47


Fig. 5. TD measures in the context of the two student collaboration types – peer-to-peer collaboration (PRTD) and student–researcher collaboration (DRTD).

students who use advanced visualization and distance technologies, and (2) determine the satisfaction level of students who collaborate with their peers and with distant researchers. Four hypotheses were derived from these objectives: (1) There would be significant differences in TD between perceptions resulting from collaborative learning of peers and those resulting from the collaboration with the distant researchers. The expected difference was that TD perception among peers would be higher than the TD perception between these students and the remotely-located scientists with whom they collaborated (note that in this context a high TD perception means perception of closeness between the interacting parties). (2) There would be significant positive correlation between the TD perceptions resulting from the collaborative learning of the peers and those resulting from the collaboration with the distant researchers. (3) There would be significant positive correlation between the TD perceptions and students’ outcomes. (4) The learning outcomes—communication TD, understanding TD, number of learned skills, and the personal project grade—will be found as significant predictors of students’ satisfaction from the course.

Table 1 TD means, standard deviations and t-test results for differences between groups. TD measure

PRTD PRTD-communication PRTD-understanding PRTD-satisfaction DRTD DRTD-communication DRTD-understanding DRTD-satisfaction

Group 1

Group 2

M

SD

M

SD

3.84 3.80 4.06 3.80 3.58 3.53 3.78 3.69

.63 .54 1.12 .80 .86 .95 1.03 .80

3.95 3.89 3.91 4.05 3.68 3.54 3.50 3.92

.69 .73 .75 .80 .93 1.26 .88 .95

t-test

P value

.34 .24 .29 .59 .20 .01 .57 .48

ns ns ns ns ns ns ns ns

The preliminary analysis was intended to test if we can statistically combine the two student groups (n1 = 7, n2 = 10) into one research group of 17 students. Independent samples t-tests were conducted to compare the TD measures in group 1 and group 2 in order to ensure that there were no significant differences between those groups so we can statistically treat both as one group. Table 1 presents the means, standard deviations and independent t-test results for differences between the TD measures of the two groups. These findings indicate that there were no significant differences in TD measures between the two groups, as illustrated in Fig. 6, enabling us to combine the two research groups into one.

student–peers collaboration and those resulting by the collaboration with the distant researchers. In order to test this hypothesis, a paired-samples t-test was conducted to compare collaboration TD measures in student–peers and student–researcher conditions. Table 2 presents the TD measures’ means, standard deviations, and paired samples t-test results for differences between peers and distant research project team collaboration. In line with our first hypothesis, the findings in Table 2 indicate that there were significant differences in three of the four TD measures between student–peers collaboration TD (PRTD) and student–researchers collaboration TD (DRTD). The PRTD means were significantly higher (i.e., peers had a perception of a lower distance) than the DRTD in two of the three measures. The reflection reports and the interviews indicated that peer collaboration characteristics were more flexible than the collaboration with the distant researchers. Many of the participants referred to the fact that the distance meetings with the researchers were rigid because ‘‘these meetings were set in advance’’ and ‘‘only one student spoke for the entire group’’. Moore [34,35] claimed that TD is influenced by the degree of the communication structure flexibility. As the structure is less rigid and more flexible, TD is reduced. The significant difference between these two types of TD provides evidence for construct validity by group comparison and reinforces the choice of TD as a theoretical framework for constructing assessment techniques for collaborative PBL involving advanced visualization and distance technologies, which was the first objective of our study.

3.2. Research hypothesis 1

3.3. Research hypothesis 2

The first research hypothesis was that there would be significant differences in TD between perceptions resulting by

The second research hypothesis was that there would be significant positive correlation between the student–peers collaboration

3. Data analysis and results 3.1. Preliminary analysis

48


Fig. 6. Means comparison between groups in peer-to-peer collaboration TD (PRTD) and student–researcher collaboration TD (DRTD). Table 2 TD measures means, standard deviations, and t-test results for differences between peers and distant research project team collaboration.

*

TD measure

M

SD

t-test

df

PRTD (overall) DRTD (overall) PRTD-communication DRTD-communication PRTD-understanding DRTD-understanding PRTD-satisfaction DRTD-satisfaction

3.91 3.64 3.86 3.53 3.96 3.55 3.96 3.83

.65 .88 .65 1.12 .83 .95 .78 .88

1.75*

15

1.30

15 *

13

*

15

1.97 1.91

p 6 .05.

TD and those resulting from the student–researchers collaboration. For assessing the relationship between the TD measures, a Pearson product-moment correlation coefficient was computed. In accord with the second hypothesis, PRTD and DRTD were high and strongly correlated, r(16) = 0.72, p < 0.01. Table 3 presents the inter-correlations matrix between the TD measures. The matrix in Table 3 shows high, strong, and positive correlations between both the three PRTD measures and the three DRTD measures. These findings provide evidence of internal structure validity of the questionnaire. They also indicate high, strong, and positive correlation between TD perceptions resulting from peer students collaboration and those resulting from the student– researcher collaboration, confirming the second research hypothesis and providing further evidence of content validity. Although there are significant correlations between most of the factors, as can be seen in Table 3, there is no correlation between unrelated factors, such as DRTD-communication with PRTD-understanding. 3.4. Research hypothesis 3 The third research hypothesis was that there would be a significant positive correlation between the collaboration TD

perceptions and other students’ learning outcomes. For assessing the students’ outcomes, we examined students’ individual project summaries, their reflection reports, and their interviews. We analyzed the personal project summaries with respect to four aspects of the learning outcomes: (1) modeling context understanding, (2) modeling process understanding, (3) modeling methodology understanding, and (4) report quality. Context understanding focuses on understanding what they did (‘‘We developed a model that showed the main processes of. . .’’). Process understanding relates to their explanations about how they designed and constructed the conceptual model (‘‘We organized and mapped various possible scenarios and then. . .’’). Methodology understanding is the understanding of the reasons for modeling; why they needed to conceptually model systems (‘‘It enabled us to simplify the VISIONAIR processes.’’). Finally, report quality refers to indices of clarity, structure, and correctness of details. Each criterion was rated in the range of 0 to 3 as follows: 0 = missing; 1 = exists but incomplete; 2 = good, and 3 = very good, exceptional, or innovative. Based on the criteria scores, the project summary grade was calculated for each participating student. Personal and interpersonal skills are one of the major teaching tasks of engineering education [6]. Therefore, we analyzed the students’ reflection reports and their interviews with respect to five skills that had been targeted in the course and which emerged during the content analysis: (1) learning a new language and modeling tool, (2) practicing in dynamic and autodidactic learning, (3) understanding the nature of the industrial engineer role, (4) understanding the nature of conducting a global project, and (5) the importance of teamwork. Based on the skills analysis we counted the number of different skills that each student mentioned. For assessing the relationship between the TD perceptions and the other student outcomes, a Pearson product-moment correlation coefficient was computed. Table 4 presents the correlations matrix between three TD measures and students’ outcomes. The

Table 3 Inter-correlations matrix between the peer-to-peer collaboration TD (PRTD) and student–researcher collaboration TD (DRTD) measures.

PRTD-communication PRTD-understanding PRTD-satisfaction DRTD-communication DRTD-understanding DRTD-satisfaction *

p 6 .05. ** p 6 .01.

PRTD-communication

PRTD-understanding

PRTD-satisfaction

DRTD-communication

DRTD-understanding

DRTD-satisfaction

– .50* .76** .47* .36 .69**

– .66** .43 .62** .54*

– .63** .37 .96**

– .60** .75**

– .43*

–


matrix in Table 4 indicates high, strong, and positive correlation between PRTD, DRTD and total TD on one hand and students’ outcomes on the other hand. Since we relate to TD and to the students’ outcomes as results of the same collaborative learning process, these findings provide evidence for content validity of the TD questionnaire by these two criteria. 3.5. Research hypothesis 4 The fourth research hypothesis was that the learning outcomes, namely the communication TD, understanding TD, number of learned skills, and the individual project grade, will be found as significant predictors of a student’s satisfaction from the course. In order to test this hypothesis, a stepwise multiple regressions test was conducted. The predictors were peer-to-peer student communication and understanding TD measures, student–researcher communication, and understanding TD measures, the individual project grade, and the number of learned skills. The findings of the regression are presented in Table 5. These regression results show that PRTD-communication was entered in the first step. The percent of explained variance was 57.9% and the regression equation reached statistical significance, F(1, 12) = 16.50, p < .01. According to the standardized regression coefficients (b), PRTD-communication made a positive significant contribution to explaining the variance, b = 0.76, p < .01. Thus, as peer-to-peer student collaboration through visualization and conferencing technologies increases, so does the course satisfaction. DRTD-communication was entered in the second step in addition to PRTD-communication. The percent of explained variance increased to 75.5% and the regression equation reached statistical significance, F(2, 11) = 16.97, p 6 .001. According to the standardized regression coefficients (b), PRTD-communication contributed positively and in a significant way to explaining the variance, b = 0.50, p < .05. DRTD-communication also made a positive significant contribution to explaining the variance, b = 0.49, p < 0.05. Thus, the more the student collaborated with his peers and with Table 4 Correlations matrix between peer-to-peer collaboration TD (PRTD), student– researcher collaboration TD (DRTD) and total TD to students’ outcomes.

PRTD DRTD Total TD * **

Individual project summary grade

Number of skills in reflection report

.55* .58** .59**

.56* .59** .63**

p 6 .05. p 6 .01.

Table 5 Stepwise regression results for predicting students’ satisfaction via student learning outcomes. Step I b PRTD-communication .98 DRTD-communication R R2 F df-Regression df-Residual *

p 6 .05. p 6 .01. *** p 6 .001. **

.76 .58 16.50** 1 12

Step II SD

B

t

b

.24 .76 4.06** .65 .37 .87 .76 16.97*** 2 11

SD

B

t

.23 .50 2.85* .13 .49 2.81*

49

the distant researcher through visualization and conferencing technologies, the more her or his satisfaction from the course increased. TD-understanding, the number of learned skills, and the individual project grade were not found as significant predictors of course satisfaction. The testimonies that the students wrote in their reflection reports about their interactions with each other and with the distant researchers strengthen these findings. The sentences students wrote include the following: (1) ‘‘We worked together with OPM and I am glad that I learned how to use this tool.’’ (2) ‘‘The cooperation with distant researchers was interesting, challenging and important, and I would like to experience this in additional courses.’’ (3) ‘‘I loved the contact with the distant researchers, which was the uniqueness of this course.’’ (4) ‘‘Conference calls and the connection with the distant researchers were very helpful for understanding the essence of this global project.’’ 4. Discussion and conclusion The main objective of the study was to develop and validate an online questionnaire for assessing the TD perception among peer students who collaborated via visualization-based environment within the VISIONAIR project. The questionnaire content was validated by three experts in educational technologies. The findings of the reliability indicate evidence of internal structure validity of the questionnaire. The differences we found in TD perception among peers and those resulting from interaction with distant researchers indicate construct validity. As we expected, the peer-to-peer student TD results were lower than those of the student–researcher TD. The significant, high, and strong factors correlation we found provide further evidence of content validity. We referred to the transactional distance perceptions as one of the outcomes of the collaborative PBL. The positive correlation we found between the TD perceptions and other student outcomes further strengthens the content validity of the TD questionnaire. As a whole, the reliability and validity results provide ample assurances that this tool can be trusted to faithfully measure and evaluate the collaborative learning quality, and that it can be adapted outside the immediate context of this particular study. As mentioned in the theoretical background section, there are only few instruments to measure collaborative learning outcomes, and those that exist are adapted to specific research environments [46]. Our collaboration TD questionnaire is independent of a particular collaboration research environment or setting, because it measures humans’ subjective perception about their ability to explain and communicate ideas and to understand each other. Baranyi and Csapo [2] stated that cognitive info-communications (CogInfoCom) focuses on support and expansion of cognitive processes by using info-communication devices and tools so that the capabilities of the human brain can be extended beyond the physical and temporal distances. Using various devices and tools for measuring and comparing their effectiveness while collaborating, requires, among others, a tool that can measure and compare the psychological, pedagogical, and communicational distance. The TD questionnaire that was developed as part of this study was designed to do just that. We designed the TD questionnaire based on several criteria that have been suggested in the literature [34,36]. Our TD measurement tool has two limitations. The first relates to the nature of the tool and the second concerns the timing of measurement. We collected subjective perceptions of each subject at the end of the course. It would be interesting to extend the measurement of the pedagogical distance between the peers by using objective measures in addition

50


to the subjective perception. Mental cloning-based software is state-of-the-art technology that enables software engineers to design the application’s user interface based on cognitive analysis of the user [15,14,16,28]. Through behavioral observation followed by facial and voice analysis, these researchers analyzed the engagement of their users. Integrating a technology such as this to measure the peer-to-peer engagement will provide means to measure in real time the pedagogical distance between the peers and then analyze the quality of the collaboration. Such measurement methods are not only non-intrusive, and as such are more objective; they also enable measuring this distance during the course of the collaboration activity, not just at its end. The second research objective was to evaluate the students’ satisfaction level. Students expressed satisfaction from being exposed to and experiencing work in a visualization-based environment. The quantitative analysis findings were reinforced by the qualitative analysis results of the students’ reflection reports. Findings of previous studies explain and reinforce our findings too. Several studies had investigated course satisfaction predictors in technology-based learning environments [1,4,5,21,37]. The technological dimension was an important factor among those that contributed to students’ satisfaction. The contribution of using technologies to raising the level of satisfaction is particularly notable with experienced users [21]. Bourne and his colleagues [4] found that the convenience achieved by using technologies increases student satisfaction. Focusing on the rapid feedback that the use of technologies enables and its effect on student satisfaction, Chen and his colleagues [5] have also found evidence for the contribution of technologies to satisfaction. Our participants were somewhat experienced with using technologies. The visualization-based modeling environment has contributed to their ability and convenience to collaborate and transfer knowledge and ideas, as well as to the rapid feedback they got from peers and distant researchers. All these explain the contribution of the use of technologies to the students’ high satisfaction level we found. This study points to the need of tailored student-centered pedagogies that are commensurate with emerging technology-rich distance education and research environments. It is not the technology per se, but its contribution to the learning and the feeling of students’ self-esteem. Our study is of a pioneering nature in that it is the first to investigate transactional distance (TD) in a visualization-rich, projectbased environment which involves students interacting with researchers rather than with their professors. The participants in such learning environments play the role of a student and a teacher interchangeably. The environment in which the research was conducted emulates a real setting, in which engineers, developers, and researchers work collaboratively, often over continental distances, using visualization and conferencing technologies. Science and engineering educators and university professors are likely to find this research valuable, as it exposes them to the concept of TD from several viewpoints: (a) providing a standardized way of measuring collaboration between learners; (b) presenting new ways of designing a project-based learning approach for undergraduate courses; and (c) suggesting an additional option to technology-enabled learning in such environments. This study shows how instructors and their students can adopt new course formats that exploit emerging visualization technologies for the benefit of advancing science and engineering education while providing tangible benefits to collaborating researchers. Our TD tool can be viewed as a candidate for measure TD, given that there are no similar measurement tools and the literature definitions of collaboration point towards the criteria that our tool employs. Indeed, even though our TD measurement tool has two limitations, it opens the door for further research, which will validate the collaborative TD questionnaire in different settings and cultures and examine it on a larger number of participants. Since

this is the first time this type of TD questionnaire is developed and implemented, additional studies are needed to examine its ability to differentiate between various collaborative pedagogies, the relationships between collaborative TD and other collaborative learning outcomes, and new designs of visualization-based collaborative environments.

Acknowledgement This work was funded by the EU FP7 Project #262044 – VISIONAIR – Vision Advanced Infrastructure for Research and the Technion Postdoctoral Fellowship.

References [1] M. Barak, T. Ashkar, Y. Dori, Learning science via animated movies: its effect on students’ thinking and motivation, Comp. Educ. 56 (3) (2011) 839–846, http:// dx.doi.org/10.1016/j.compedu.2010.10.025. [2] P. Baranyi, A. Csapo, Definition and synergies of cognitive infocommunications, Acta Polytech. Hung. 9 (1) (2012) 67–83. [3] A. Blekhman, D. Dori, Tesperanto – a model-based system specification methodology and language, in: 23rd Annual INCOSE International Symposium, Philadelphia, PA, USA, 2013. [4] J. Bourne, D. Harris, F. Mayadas, Online engineering education: learning anywhere, anytime, J. Eng. Educ. 94 (1) (2005) 131–146, http://dx.doi.org/ 10.1002/j.2168-9830.2005.tb00834.x. [5] J.C. Chen, D.C. Whittinghill, J.A. Kadlowec, Classes that click: fast, rich feedback to enhance student learning and satisfaction, J. Eng. Educ. 99 (2) (2010) 159– 168, http://dx.doi.org/10.1002/j.2168-9830.2010.tb01052.x. [6] E.F. Crawley, D.R. Brodeur, D.H. Soderholm, The education of future aeronautical engineers: conceiving, designing, implementing and operating, J. Sci. Educ. Technol. 17 (2008) 138–151. . [7] E.F. Crawley, J. Malmqvist, W.A. Lucas, D.R. Brodeur, The CDIO syllabus v2.0. An updated statement of goals for engineering education, in: Proceedings of 7th International CDIO Conference. Denmark, 2011 . [8] P. Dillenbourg, What do you mean by collaborative learning?, in: P. Dillenbourg (Ed.), Collaborative-learning: Cognitive and Computational Approaches, vol. 1, Elsevier, Oxford, 1999, pp. 1–19 . [9] D. Dori, Object-Process Methodology, Springer, Berlin Heidelberg, Berlin, Heidelberg, 2002. doi:10.1007/978-3-642-56209-9. [10] D. Dori, I. Reinhartz, A. Strum, Developing complex systems with objectprocess methodology using OPCAT, in: 22nd International Conference on Conceptual Modeling, October 13–16, 2003, Chicago, pp. 570–572, doi:http:// dx.doi.org/10.1007/978-3-540-39648-2_46. [11] Y. Dori, J. Belcher, Learning electromagnetism with visualizations and active learning, in: J.K. Gilbert (Ed.), Visualization in Science Education, vol. 1, Springer-Verlag, Berlin/Heidelberg, 2005, pp. 187–216, doi:http://dx.doi.org/ 10.1007/1-4020-3613-2_11. [12] Y. Dori, A. Silva, Assessing international product design & development graduate courses: the MIT-Portugal program, Advan. Eng. Educ. 2 (2) (2010) 1– 30. . [13] Y. Dori, R. Tal, M. Tsaushu, Teaching biotechnology through case studies – can we improve higher order thinking skills of nonscience majors?, Sci Educ. 87 (6) (2003) 767–793, http://dx.doi.org/10.1002/sce.10081. [14] H. Fujita, J. Hakura, M. Kurematsu, Virtual medical doctor interaction based on transactional analysis, in: SoMeT, 2009, pp. 503–517. [15] H. Fujita, J. Hakura, M. Kurematsu, S. Chida, Y. Arakawa, Empirical based techniques for human cognitive interaction analysis: universal template design, in: SoMeT, 2008, pp. 257–277. [16] H. Fujita, J. Hakura, M. Kurematu, Intelligent human interface based on mental cloning-based software, Knowl.-Based Syst. 22 (3) (2009) 216–234, http:// dx.doi.org/10.1016/j.knosys.2008.11.005. [17] Y. Giossos, M. Koutsouba, A. Lionarakis, Reconsidering Moore’s transactional distance theory, Euro. J. Open Dist. ELearn. 2 (2009) 1–6. . [18] Y. Grobshtein, D. Dori, Generating SysML views from an OPM model: design and evaluation, Syst. Eng. 14 (3) (2011) 327–340. [19] S.R. Hiltz, N. Coppola, N. Rotter, M. Toroff, R. Benbunan-Fich, Measuring the importance of collaborative learning for the effectiveness of ALN: a multimeasure, in: J. Bourne (Ed.), Online Education: Learning Effectiveness and Faculty Satisfaction, vol. 1, 2000, pp. 101–119. [20] C.E. Hmelo-Silver, Problem-based learning: what and how do students learn?, Educ Psychol. Rev. 16 (3) (2004) 235–266, http://dx.doi.org/10.1023/B:EDPR. 0000034022.16470.f3. [21] K.-S. Hong, Relationships between students’ and instructional variables with satisfaction and learning from a Web-based course, Inter. High. Educ. 5 (3) (2002) 267–281, http://dx.doi.org/10.1016/S1096-7516(02)00105-7.

N. Wengrowicz et al. / Knowledge-Based Systems 71 (2014) 41–51 [22] C. Hsieh, L. Knight, Problem-based learning for engineering students: an evidence-based comparative study, J. Acad. Librarian. 34 (1) (2008) 25–30, http://dx.doi.org/10.1016/j.acalib.2007.11.007. [23] S. Hwang, P. Karnofsky, Defining, measuring, and improving collaboration though evaluating the cornell library collaborative learning computer laboratory, Unpublished Cornell Class Report, 1–4, 2005 . [24] D. Johnson, Student–student interaction: the neglected variable in education, Educ. Research. 10 (1) (1981) 5–10. . [25] C. Kennedy, N. Brown, K. Draney, M. Wilson, Using progress variables and embedded assessment to improve teaching and learning, in: Annual Meeting of the American Education Research Association, Montreal, Canada, 2005 . [26] E.S. Klein, E. Merritt, Environmental education as a model for constructivist teaching, J. Environ. Educ. 25 (3) (1994) 14–21, http://dx.doi.org/10.1080/ 00958964.1994.9941953. [27] J.S. Krajcik, C. Czerniak, C. Berger, Teaching Children Science: A Project-Based Approach, McGraw-Hil, Boston, 1999. [28] M. Kurematsu, M. Ohashi, O. Kinosita, J. Hakura, H. Fujita, A study of how to implement a listener estimate emotion in speech, in: Proceedings of the 2009 Conference on New Trends in Software Methodologies, Tools and Techniques: Proceedings of the Eighth SoMeT_09, IOS Press, 2009, pp. 531–540. [29] D. Lawton, N. Vye, J. Bransford, E. Sanders, M. Richey, D. French, R. Stephens, Online learning based on essential concepts and formative assessment, J. Eng. Educ. 101 (2) (2012) 244–287, http://dx.doi.org/10.1002/j.2168-9830.2012. tb00050.x. [30] J. Malmqvist, A comparison of the CDIO and EUR-ACE quality assurance systems, Int. J. Qual. Assur. Eng. Technol. Educ. 2 (2) (2012) 9–22, http:// dx.doi.org/10.4018/ijqaete.2012040102. [31] J.H. Mathewson, Visual-spatial thinking: an aspect of science overlooked by educators, Sci. Educ. 83 (1) (1999) 33–54, http://dx.doi.org/10.1002/ (SICI)1098-237X(199901)83:13.0.CO;2-Z. [32] R.E. Mayer, Cognitive theory and the design of multimedia instruction: an example of the two-way street between cognition and instruction, New Direct. Teach. Learn. 2002 (89) (2002) 55–71, http://dx.doi.org/10.1002/tl.47. [33] N. Mills, A guide du routard simulation: increasing self-efficacy in the standards through project-based learning, For. Lang. Ann. 42 (4) (2009) 607– 639, http://dx.doi.org/10.1111/j.1944-9720.2009.01046.x.

51

[34] M.G. Moore, Editorial: is teaching like flying? A total systems view of distance education, Am. J. Dist. Educ. 7 (1) (1993) 1–10, http://dx.doi.org/10.1080/ 08923649309526806. [35] M.G. Moore, Theory of transactional distance, in: D. Keegan (Ed.), Theoretical Principles of Distance Education, 1st ed., Routledge, 1993, pp. 22–38 . [36] M.G. Moore, The theory of transactional distance, in: M.G. Moore (Ed.), Handbook of Distance Education, third ed., Routledge, 2013, pp. 66–85. [37] L. Motiwalla, S. Tello, Distance learning on the internet: an exploratory study, Inter. High. Educ. 2 (4) (2000) 253–264, http://dx.doi.org/10.1016/S10967516(00)00026-9. [38] D. Nevo, How can teachers benefit from teacher evaluation?, J Pers. Eval. Educ. 8 (2) (1994) 109–117, http://dx.doi.org/10.1007/BF00972258. [39] D. Nevo, School-Based Evaluation: A Dialogue for School Improvement, Pergamon, Oxford, 1995. p. 213. [40] A. Paivio, Mental Representations: A Dual Coding Approach. Mental Representations, second ed., Oxford University Press, NY, 1990. [41] M. Peleg, D. Dori, The model multiplicity problem: experimenting with realtime specification methods, IEEE Trans. Softw. Eng. 26 (8) (2000) 742–759. [42] J.L. Polman, R.D. Pea, Transformative communication as a cultural tool for guiding inquiry science, Sci. Educ. 85 (3) (2001) 223–238, http://dx.doi.org/ 10.1002/sce.1007. [43] I. Reinhartz-Berger, D. Dori, OPM vs. UML: experimenting with comprehension and construction of web application models, Empir. Softw. Eng. 10 (1) (2005) 57–80. [44] S.P. Schaffer, X. Chen, X. Zhu, W.C. Oakes, Self-efficacy for cross-disciplinary learning in project-based teams, J. Eng. Educ. 101 (1) (2012) 82–94, http:// dx.doi.org/10.1002/j.2168-9830.2012.tb00042.x. [45] R. Tal, S. Argaman, Characteristics and difficulties of teachers who mentor environmental inquiry projects, Res. Sci. Educ. 35 (4) (2005) 363–394, http:// dx.doi.org/10.1007/s11165-004-8163-y. [46] A. Thomson, J. Perry, T. Miller, Conceptualizing and measuring collaboration, J. Pub. Admin. Res. Theory 19 (1) (2007) 23–56, http://dx.doi.org/10.1093/ jopart/mum036. [47] N. Wengrowicz, B. Offir, Teachers’ perceptions of transactional distance in different teaching environments, Am. J. Dist. Educ. 27 (2) (2013) 111–121, http://dx.doi.org/10.1080/08923647.2013.773701. [48] U. Zoller, Problem-solving and the ‘‘problem-solving paradox’’, in: S. Keiny, U. Zoller (Eds.), Conceptual Issues in Environmental Education, Peter Lang, New York, 1991, pp. 71–87.