Using Concept Maps To Assess Design Process Knowledge - CiteSeerX

Session F1G

Using Concept Maps To Assess Design Process Knowledge Judith E. Sims-Knight1, Richard L. Upchurch2, Nixon Pendergrass3, Tesfay Meressi 4, Paul Fortier 5 Plamen Tchimev6, Rebecca VonderHeide7, and Madeleine Page8 Abstract - If engineering educators are to incorporate assessment of student learning outcomes into their curricula, they need assessments that are reliable, valid and feasible within the time constraints of coursework. We are engaged in an NSF supported project to develop such measures for design skill. This paper describes our exploration of the use of student-generated concept maps to assess students’ understanding of how various aspects of the design process go together. Students in three seniorlevel engineering courses constructed concept maps of the design process. The resulting maps could be reliably sorted into patterns that presumably represent distinctly different ways of understanding the process. In addition, subpatterns of the concept maps were used to assess specific units of knowledge (e. g., the relation between feasibility, on the one hand, and requirements and preliminary design, on the other). These two components comprise an easily created report that provides detailed and useful pointers toward course and curricular improvement. Index Terms – Design Process, Assessment, Continuous Improvement, Concept Maps INTRODUCTION If engineering educators are to incorporate assessment of student learning outcomes into their curricula, they need assessments that are reliable, valid and feasible within the time constraints of coursework. We are engaged in an NSF supported project to develop such measures for design skill. Our overall strategy is to develop assessments for each component of expert design skill. Experts know the facts about design practice, they understand how the facts fit together, and they know how to go about creating designs. This paper will describe our exploration of the use of studentgenerated concept maps to assess students’ understanding of how various aspects of the design process go together. Concept maps, broadly conceived, are network diagrams in which concepts (nouns) are nodes and the relationships between concepts (verbs) are links. They are directed, acyclic graphs consisting of a set of concepts and a non-empty set of

relationships (associations between concepts). Researchers in this area [1] – [4] have made a case that concept maps provide a measure of structural knowledge, that is, that there may be a relationship between what is actually known by the learner and the external representation of this knowledge as a concept map. Concept maps have been used widely in education [5] – [7]. They have been successfully used as pedagogical tools [89] and as assessment devices [10] –[11]. Further research indicates teachers are able to use student created concept maps to identify those specific areas within the curriculum that should be modified in order to promote a greater understanding of the subject matter [12] – [14]. Much of the research on concept maps has scored them by analyzing propositions, that is, by extracting node-linknode combinations [6] – [7]. In directed graphs, which are networks in which the link is directional (arrows), the propositions are clearly analogous to the subject-verb-object structure of sentences. Propositions are then typically scored against a template of propositions considered correct. In openended domains such as design process, there is no one right answer. In such circumstances novices’ propositions are compared to propositions on which experts agree (e. g., [11]). Overall patterns in concept maps have also been found to be indicative of cognitive status [15]-[16]. Hart [15] developed a classification scheme pattern based on spatial layout. His scheme included four patterns that differentiated among students’ grades on an architectural design project. Particularly striking was that all the students who got A’s had one pattern, which he called branching. The existing research and use of concept maps has focused on the use of concept maps of declarative knowledge (often to represent chapters in texts). It is our belief that procedural knowledge is also structured in a way that can be represented by networks (although not by strict tree-structure concept maps). The various patterns used by Hart may represent different ways our students understand the design process. If students understand the design process only as a series of phases, they will be likely to use his linear pattern to describe that sequence of events. This would be overly simplistic, because it captures neither the iterative nature of

1

Judith Sims-Knight, Department of Psychology, UMASS Dartmouth, No. Dartmouth, MA 02747-2300, [email protected] Richard Upchurch, Department of Computer and Information Science, UMASS Dartmouth, No. Dartmouth, MA 02747-2300, [email protected] Nixon Pendergrass, Department of Electrical & Computer Engineering, UMASS Dartmouth, No. Dartmouth, MA 02747-2300, [email protected] 4 Tesfay Meressi, Department of Mechanical Engineering, UMASS Dartmouth, No. Dartmouth, MA 02747-2300, [email protected] 5 Paul Fortier, Department of Electrical and Computer Engineering, UMASS Dartmouth, No. Dartmouth, MA 02747-2300, [email protected] 6 Plamen Tchimev, CIS Department, University of Massachusetts Dartmouth, Dartmouth, MA 02747-2300, [email protected] 7 Rebecca VonderHeide, Department of Psychology, UMASS Dartmouth, No. Dartmouth, MA 02747-2300, [email protected] 8 Madeleine Page, Department of Psychology, UMASS Dartmouth, No. Dartmouth, MA 02747-2300, [email protected] 2 3

0-7803-8552-7/04/$20.00 © 2004 IEEE October 20 – 23, 2004, Savannah, GA 34th ASEE/IEEE Frontiers in Education Conference F1G-6

Session F1G the process nor the richness of the associations among nodes. Students who understand the iterative nature of the design process and the interconnections among the concepts would likely use one of the other patterns, most particularly Hart’s branching pattern. In this study we describe our explorations of the utility of concept maps in assessment of design skill. THE TASK The expert concept map task was developed by consensus among a team of three engineers (one computer, one electrical and one mechanical), one computer scientist, and one psychologist. The team, after long discussion, argument, and reflection, agreed on a list of ten nouns and six verbs to represent the design process. The nouns are: customer, design, feasibility, need, product, requirements, tentative design, testing, tradeoffs, and user. The verbs are: drives, evaluates, has, includes, influences, and yields. The students’ task was to use some or all of the verbs provided to link with arrows the ten nodes, which could be placed anywhere on the page. METHOD Participants The sample consisted of students from three different senior courses—software engineering, required of both computer science and computer engineering students, a senior design project course required of computer engineering and electrical engineering students, and a senior database course that is a technical elective in the electrical and computer engineering department. The total of 166 students included all students who finished these courses in 2002 and 2003. Materials and Procedure An instructional packet was developed to introduce students to concept mapping in the context of procedural knowledge. Students practiced by filling in portions of a partially completed map that represented the structure of role-playing games, a domain with which most students are familiar. Students were given feedback about the maps and suggestions about how to use concept maps to represent deep knowledge. The instructions also included instructions on using IHMC Concept Map Software (CMAP) [17]. The instructional packet was passed out in class along with instructions to go to a computer laboratory to create their own map. The students then completed a postmortem on their experience with the concept map.

RESULTS Classifying Patterns We developed a rule-based scoring system designed to capture Hart’s patterns (Hart did not report his system). The patterns are web (one or two central concepts with links to at least five other nodes), cat’s cradle (a number of nodes with at least three links), linear (a chain of at least 5 nodes, which typically represent stages in the design process), and branching (an overall integrated pattern with subsets of connected links) (see Figure 1 for an example of each). Two patterns (cat’s cradle and linear) were subdivided into extreme and moderate variants). Overall reliability for concept map sorting was 96% agreement and reliabilities for individual categories ranged from 91 to 100% agreement. Examination of the examples demonstrates that some of the patterns have face validity. The linear pattern suggests a conception of the design process as a series of steps to be undertaken one at a time. Hence this pattern tends to capture the process model rather than to specify the relationships among the concepts. There is no indication that the student is aware of the need to iterate or to apply issues of feasibility, tradeoffs, and testing at more than one stage in the process. The web pattern makes one concept primary. It thereby suggests that the student does not understand that requirements and tentative design are just as important as design or product and that she or he does not understand the need to apply issues of feasibility, tradeoffs and testing at several points in the design process. The cat’s cradle and branching patterns, in contrast, do reflect students’ awareness of the need for iteration and address issues of feasibility, tradeoffs and testing at more than one stage in the process. Cat’s cradle patterns, however, typically have too many links and reflect some fundamental misconceptions or lack a full integration/assimilation of the concepts. The example in Figure 1b, for example, has tradeoffs isolated from requirements and tentative design and feasibility influencing testing, but not tentative design. The prevalence of these patterns varies as a function of major, χ2(5) = 15.3, p = .009 (see Table 1). That there are differences in frequency of pattern as a function of major indicates that the way students are taught will influence their conceptions of the design process. The overall patterns of concept maps can be used as a basis for continuous improvement. The standard rule to identify problems that require improvement is 20 percent. For example, if 35 per cent of students in a class are creating pure linear maps and the instructor believes a branching pattern represents better structure, then he or she can redesign the course to help students grasp the iterative nature of the design process and the desirability of addressing feasibility, trade-offs, and testing early and often in the design process.


Session F1G

a. Branching

b. Cat’s Cradle

c. Web

d. Linear FIGURE 1 PATTERNS OF CONCEPT MAPS


Session F1G TABLE 1 FREQUENCY OF CONCEPT MAP PATTERNS IN COURSES IN COMPUTER AND INFORMATION SCIENCE (CIS) AND ELECTRICAL AND COMPUTER ENGINEERING (ECE)

Pattern

ECE courses

CIS course

Pure linear

14

8

Linear with branches

14

12

Cat’s cradle high

6

28

Cat’s cradle low

13

27

Web

14

21

4

5

Branching Classifying Subpatterns

Classifying overall patterns, while useful, does not capture all the information that concept maps carry. Examination of subpatterns permits instructors to focus on misconceptions of specific constructs. Figures 2 and 3 show an analysis of students’ responses with respect to two features. In each case the pattern at the top was that deemed desirable (by the authors). The report indicates the percent of the students in the class got the whole pattern, and then the percent who got pieces of the pattern. Figure 2 shows that no one in the class understood the relationship between feasibility, on the one hand, and requirements and tentative design, on the other hand. In fact, fewer than 20 per cent got any part of the pattern and 38 per cent linked feasibility with design, which is much too late in the design process. This result tells the instructor that she or he needs to redesign instruction to promote understanding of how and why feasibility should be addressed when developing requirements and tentative design. Figure 3 shows the report of the software engineering students’ understanding of design phases. Students had somewhat greater understanding of the relationships among the phases of the design process than they did feasibility. Although only 13.6 per cent got the whole pattern, approximately half the class got each of the three pairs (not the same students). Still, all of these results would be identified as loci for improvement, since in no case did more than 80% exhibit the desired pattern (i.e., in all cases over 20 percent misunderstood).

FIGURE 2 ASSESSMENT OF STUDENTS’ UNDERSTANDING OF FEASIBILITY

FIGURE 3 ASSESSMENT OF STUDENTS’ UNDERSTANDING OF RELATIONSHIP AMONG DESIGN, TENTATIVE DESIGN, AND PRODUCT

DISCUSSION Requiring students to construct a concept map of the design process when given the concepts and links was an effective means of revealing the structure of their underlying understanding of how pieces of the design process relate to one another. Such knowledge is part of domain-specific expertise. Both whole map analysis and analysis of crucial substructures provided useful information about what students understood and provided both general and specific information that could be used as a basis for continuous improvement.

Use of concept maps as an assessment of design skill does not require that all instructors agree on a single best map, which is good because the concept maps of experts often vary [18]. Using concept maps as assessment tools can accommodate such differences. For example, imagine that Professor A considers the linear pattern ideal and Professor B prefers the branching pattern. Both use the concept map construction task several times during a course. Both classes at the beginning would be expected to create a variety of maps. By the end of the class a larger number of Professor A’s


Session F1G students would create linear maps, whereas a larger number of Professor B’s student would create branching maps. In both cases application of the quality model has led to less variability in students, which is the goal. The nodes and links of the concept map construction task can also be varied. The procedures described here would be unchanged if other terms were added or substituted. The version used in this study set as a criterion that the fewest concepts and verbs as possible be used, because of the demands of another task that was being coordinated with it. A fair criticism of this work is that it has not demonstrated that the maps truly reflect students’ understanding. We suggested interpretations of the whole map patterns largely on the basis of face validity. It will require further research to demonstrate, for example, that students who create linear maps are truly less likely to revisit feasibility, trade-offs and testing at more than one phase of development or that students who make a link from feasibility only to design would not actually consider feasibility while writing requirements. Nonetheless, it seems unlikely that this task has no validity, because so much past research has found validity in the use of concept maps [1]-[16], [18]. In addition, we will be addressing these issues in a future paper. Making the construction of concept maps required assignments, as homework or as graded assignments, increases the probability that students will take them seriously. Although the data in this paper came entirely from one situation—students individually creating maps in CMAP—we have informally tried having students create a joint map in small groups. This procedure worked well, both in terms of the discussion generated and the quality of the product. The technique described in this paper is quite practical. It requires approximately one hour of students’ time to do and we are developing computerized scoring of the propositions. The categorizing of the whole-map patterns is essentially procedural and so a set of 50 can be done in fewer than two hours. In conclusion, this paper showed that a concept map construction task can be used effectively to assess students’ understanding of the design process in ways that can provide pointers for course and curricular improvement. ACKNOWLEDGMENT This work was done under the auspices of grant No. 0127751 from the National Science Foundation Assessment of Student Achievement program.

[4]

Wilson, J. M. (1994). Network presentations of knowledge about chemical equilibrium: Variations with achievement. Journal of Research in Science Teaching, Vol. 31, No. 10, pp. 113-117

[5]

Turns, J., Atman, C. J., & Adams, R. "Concept maps for engineering education: A cognitively motivated tool supporting varied assessment functions," IEEE Transactions on Education Special Issue on Assessment, Vol. 43, May 2000, pp. 164-173

[6]

McClure, J., Sonak, B. and Suen, H. “Concept map assessment of classroom learning: reliability, validity and logistical practicality,” Journal of Research in Science Teaching, Vol. 36, No. 4, 1999, pp. 475492

[7]

Ruiz-Primo, M. A., & Shavelson, R. J. Problems and issues in the use of concept maps in science assessment,” Journal of Research in Science Teaching, Vol. 33, No. 6, 1996, pp. 569-600

[8]

Coleman, E. “Using explanatory knowledge during collaborative problem solving in science,” Journal of the Learning Sciences. Vol. 7, No. 3 & 4, 1998, pp. 387-427

[9]

Parkes, J.T., Suen, H.K., Zimmaro, D.M., & Zappe, S.M. “Structural knowledge as a prerequisite to valid performance assessment scores,” Presented at the National Council on Measurement in Education, Montreal, Canada, 1999

[10] McClure, J. R. “Concept mapping and acquisition on cognitive skills: Concept maps as a tool to study skill acquisition,” American Educational Research Association, Montreal, Canada, April 1999 [11] Herl, H. E., Baker, E. L., & Niemi, D. “Construct validation of an approach to modeling cognitive structure of U.S. history knowledge,” Journal of Educational Research, Vol. 89, No. 4, 1996, pp. 206-218 [12] Bolte, L.A. “Using concept maps and interpretive essays for assessment in mathematics,” School of Science and Mathematics, Vol. 99, No. 1, 1999, pp. 19-30 [13] Jones, G., Carter, G., & Rua, M. “Children’s concepts: Tools for transforming science teachers’ knowledge,” Science Education, Vol. 83, No. 5, 1999, pp. 545–557 [14] Weiss, L.B. & Levision “Tools for integrating women’s health into medical education: Clinical cases and concept mapping,” Academic Medicine, 75(11), 2000, pp. 1081-1086 [15] Hart, I. “Visualising structural knowledge,” Fifth International Conference for Computers in Education, Beijing, October, 1998 [16] Kinchin, I. M., Hay, D. B., & Adams, A. “How a qualitative approach to concept map analysis can be used to aid learning by illustrating patterns of conceptual development.” Educational Researcher, Vol. 42, 2000, pp. 43-57 [17] Institute for Human and Machine Cognition, University of West Florida, http://cmap.ihmc.us/ [18] Britton, B. K. & Tidwell, P. “Cognitive structure testing: A computer system for diagnosis of expert-novice differences.” In P. D. Nichols, S. F. Chipman, & R. L. Brennan. Cognitively diagnostic assessment. 1995, pp. 251-278

REFERENCES [1]

Jonassen, D.H., Reeves, T., & Hong, N. “Concept mapping as cognitive learning and assessment tools,” Journal of Interactive Learning Research, Vol. 8, 1998, pp. 289–308

[2]

Diekhoff, G., & Evans, S. H. “Development and evaluation of a learning strategy program.” Journal of Educational Psychology, Vol. 71, 1979, pp. 64-73

[3]

Markham, K., Mintzes, J. and Jones, G. “The concept map as a research and evaluation tool: Further evidence of validity,” Journal of Research in Science Teaching, Vol. 31, No. 1, 1994, pp. 91-101
