Expertiza: Students Helping to Write an OOD Text Edward F. Gehringer
Luke M. Ehresman
Dale J. Skrien
North Carolina State University Department of Computer Science Raleigh, NC 27695-8206 +1 919-515-2066
North Carolina State University Department of Computer Science Raleigh, NC 27695-8206 +1 919-889-9189
Colby College Department of Computer Science Waterville, ME 04901-8840 +1 207-859-5851
[email protected]
[email protected]
[email protected]
ABSTRACT Students in a masters-level object-oriented design class participated in several exercises related to improving a not-yetpublished textbook in object-oriented design. Each student was asked, sometime during the semester, to improve an explanation from the book, to make up an example of a concept described in the book, and to create an exercise for a chapter in the book. Their contributions were peer-reviewed by other members of the class. A strong majority of the students (29 of 49) indicated that they learned a lot from doing the assignments, and 27 of 49 of them agreed or strongly agreed that the assignments were enjoyable—a good measure of their engagement. The textbook author found the feedback very helpful in revising the textbook.
Students there were asked to do homework assignments that involved improving a draft object-oriented design textbook used in the course. Both the students and the textbook author found this a very useful exercise, and it has greatly influenced the final draft of the text. Our experience demonstrates that a broad range of students in this course were capable of helping with the creation of educational materials—reusable learning objects—that can be used to good effect by subsequent generations of students. Based upon our experience, we encourage other instructors teaching object technology to assign exercises to create resources that will be of use to future students. While this is easiest and most flexible using an electronic peer-review system like ours, it is also possible to do peer review during class time.
Categories and Subject Descriptors
3. THE RATIONALE
1.1 D.3.2 [Programming Languages]: Language classifications: Object-oriented languages, K.3.2 Computer and Information Science Education
General Terms Languages
Keywords Peer review, reusable learning objects.
2. INTRODUCTION Active learning is an approach to education where students “learn by doing,” by performing a task instead of studying the material. Collaborative learning means working with others in order to learn new things. Active and collaborative learning have proven themselves markedly more effective than individual learning. Johnson et al. [JMJN 81, JJ 89] present two meta-analyses, based on dozens of other published studies, that show the benefits of cooperation for students. This paper reports on an active and collaborative learning exercise undertaken at North Carolina State University in Fall 2005. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. OOPSLA 2006 Educators’ Symposium, October 22–26, 2006, Portland, OR, USA. Copyright 2006 ACM 1-58113-000-0/00/0004…$5.00.
Peer review is a familiar concept for anyone in academia. Not only is it our main vehicle for encouraging scholarship and enhancing the knowledge base, but for the last thirty years or so, it has been used extensively in classes. Students benefit, both from reviewing others and having their own work reviewed. As assessors, they spend time analyzing, diagnosing misconceptions, and offering suggestions for improvement [VCBM 95]. As recipients of assessment, they are forced to write for an audience of their peers where it is their responsibility to make themselves understood, rather than relying on the instructor or TA to decipher their intentions. Moreover, their peers, having only a few (usually one to four) other students to review, can give them more feedback than they would ever receive from the course staff. This makes it much more feasible to ask students to revise their work in response to reviewers’ comments, and thereby improve their own work product. The whole experience encourages students to invest deeper thought and greater effort into their work. Many studies of peer review have found that students felt they thought more critically and found the work more challenging [Stef 94, OMR 00]. What’s new about our Expertiza approach is that it is geared toward creating resources (called “reusable learning objects”) that can help future students to learn the material. This represents a paradigm shift for homework assignments. Traditionally, homework has served only for students to demonstrate mastery of the subject. Every student is assigned the same work, which precipitates a lot of redundant effort. Once graded, work is “thrown away,” never to benefit anyone but the student who did it. By contrast, our methodology assigns different pieces of work to different students. The students’ work is then peer-evaluated, and when it is deemed good enough by a “panel” of the student’s peers, it can then be offered as a learning object to other students.
In a large class, several students can independently be assigned each piece of work, improving the chances that one of the submissions will be good enough to serve as a resource in the future.
reviews) make it impossible to submit a finished product obtained from an external source [Ster 04]. (Multiple deadlines, by the way, also make it more difficult to be a “slacker,” thereby helping students keep up with the course material.)
3.1 The peer-review process
The fact that students are doing customized assignments [CW 02] means that instead of being able to find a co-conspirator anywhere in the class, they are limited to only the small set of students who chose the same piece of work as they did. Since the set is so small,
Note how peer review is essential to this process. Only by getting the whole class involved in doing reviews can enough specialized feedback be given to each author on his/her work. There is no “learning curve” in assessment, as there are no more than a handful of submissions on any particular topic. There is no way that the course staff, no matter how astute or well motivated, could give the kind of personal attention through multiple rounds of review that peer evaluators can. Of course, students are not automatically good assessors. We use two main mechanisms to raise the quality of reviews. First, students are presented with a rubric consisting of several questions, which they answer for each of the submissions they review. The rubrics for this experiment are given in Table 1. Second, after the review period is over, students are asked to rate their peers’ reviews using a similar rubric. In other words, student x rates student y’s review of student z’s work. This actually does not take very much effort, because in most cases, one can quickly tell whether a review is careful just by scanning the submission and the review of it. The quality of reviewing is factored into the student’s grade for the assignment (typically, reviewing counts for 1/4 to 1/5 of the grade. Student-assigned grades are used as the grade for the assignment, unless the student appeals to the instructor; this tends to happen in 10% to 15% of the cases.
3.2 Electronic peer review While peer review can be performed in a face-to-face class setting, there are great benefits in making it a computer-mediated communication process. Electronic review can be double blind, so that neither author nor reviewers know who each other are. Distance-education students can participate on the same basis as the rest of the class. This was important in our experiment, since DE students constituted about half of the students in the class. In fact, electronic peer review is one of the few ways that activelearning exercises can be performed effectively with distance-ed students. Finally, electronic peer review avoids the overhead of a manual process. Papers have to be collected and passed out to reviewers. While a single individual can meet with a single peer reviewer face-to-face, the logistics become much more complicated when each student is to receive multiple reviews, and each student is to be reviewed by a different subset of his/her peers. An electronic system can automatically determine the best-reviewed of a set of submissions on the same topic, and assist with getting permission to publish these for outside students to see. We use the peer-review system PG [Gehr 01]. This system has been enhanced to offer support for dividing up the work among students and automatically “publishing” it for others to learn from. In principle, however, another electronic peer review system such as the Daedalus Integrated Writing Environment [Bowe 92, Crav 94] and Calibrated Peer Review [CF 00, PW 04] could be used.
3.3 Ancillary benefits of the Expertiza process The Internet has enormously increased the opportunity for students to turn in work that is not their own. The Expertiza platform incorporates several strategies that can diminish the risk. Multiple deadlines (submission, review, resubmission, review of
Table 1. Rubrics for peer review of student enhancements to textbook Rubric for explanations 1. Is the explanation correct? 2. Is the explanation clearer than the one currently in the book? (1 = much less clear; 3 = as clear; 5 = much clearer) 3. Is the explanation more complete than the one currently in the book? 4. Can the explanation be understood having only read the preceding material in the text (i.e., does it avoid forward references)? 5. Is the explanation as concise as possible, given what it needs to explain? 6. Does the explanation seem to be original? 7. Other — how would you rate the submission on factors not reflected in the scores on other questions? Rubric for examples 1. Does the example fully illustrate the concept being explained? 2. Is the example easy to understand, i.e., as clear as it could be and still illustrate the concept? 3. Does the example model the real world, i.e., could it be implemented in practice? 4. Is the example code elegant? 5. Does the example use up-to-date Java code or UML? 6. Does the text that accompanies the example explain it well? 7. Is the example more useful than the examples in this section of the book? 8. Does the example seem to be original? 9. Other — how would you rate the submission on factors not reflected in the scores on other questions? Rubric for exercises 1. Is the exercise relevant to the material covered in the textbook? 2. Would doing the exercise help students to learn the material? 3. Are the exercise and the answer stated clearly? 4. Is the answer to the exercise correct? 5. Is the correct answer well defined (rather than open-ended)? 6. Does the exercise seem to be original? 7. Other ñ how would you rate the submission on factors not reflected in the scores on other questions? the same peer reviewer(s) can evaluate most or all submissions in this set, which makes cheating easy to detect. And when students are asked to improve on a submission from a previous class [Swai 05], there is minimal risk that they could accomplish the task by Web-surfing.
The Expertiza approach makes “formative assessment” easier to achieve. Formative assessment is feedback to students to show both students and instructor how well the students are learning the material. It is distinguished from “summative assessment,” which is used to assign grades. Black and William [BW 98] reviewed 250 journal articles and book chapters on formative assessment to establish whether it raises academic standards. They found that the use of systematic formative assessment produced significant learning gains. Formative assessment can take many forms, but increasingly often consists of machine-scorable tests administered by computer. All the major learning-management systems (e.g., Blackboard) have interactive testing systems, and there are many stand-alone systems that can administer these tests as well. With the Expertiza approach, students can be assigned to create machine-scorable questions over lecture material, and these questions can be peer-reviewed by other students. The best questions can then be chosen for inclusion in mastery quizzes. In fact, this could be made an in-class assignment. After 10 or 15 minutes of lecture, the class could be split up into groups of students, with each group being asked to make up two or three questions. These questions can then be presented to other groups in the class, in a kind of “review jeopardy” [PJ 01]. The class can vote on which questions are worthy of use on future formative – assessment quizzes. When students are assigned (as they were in our experiment), to make up exercises on material from the textbook, the community gains material that can be assigned for homework or as test questions in future classes. One major publisher of textbooks in computing has expressed interest in this approach as a way of creating exercises for revised editions of textbooks. Finally, with the Expertiza approach, large classes produce more resources, and better resources to be used in the future. Faculty will never run out of questions to ask, and students, learning from materials that have been iteratively improved by past classes, will become more expert in designing good materials themselves. Flying in the face of traditional wisdom, this actually makes teaching large classes an advantage.
4. ENHANCING THE SKRIEN TEXT For the Fall 2005 semester, the first author of this paper adopted a prepublication copy of the third author’s textbook, An Introduction to Object-Oriented Design and Design Patterns using Java, to be published by McGraw-Hill. The book was used in a masters-level course in object-oriented design and programming. Typically about 10% of the students in the course are advanced undergraduates, and about 1/3 of them are Computer Engineering students (the course is cross-listed with ECE). There is a large distance-education component; half of the 78 students were taking the course by watching lectures through media files over the Web. The course covered eight of the ten chapters in the textbook. There were three different kinds of peer-reviewed assignments during the semester. •
Improving an explanation of some concept from the textbook. After every two chapters or so, we would discuss in class which were the most difficult concepts. Among the concepts chosen were— o
Overloading vs. overriding
o o o o •
Adding another example of a concept covered in the text, such as o o o o
•
Inheritance vs. interfaces—when to use each When is polymorphism not a sufficient reason to subclass? Overriding the equals method The clone method
public vs. private methods, including a discussion of accessor methods when not to use class methods or instance methods simple examples of common refactorings examples of specific design patterns, shorter than the book’s running example relating to a drawing application
Writing a back-of-the-chapter exercise for one of the chapters in the book.
In order to cover topics throughout the book uniformly, and while they were still fresh in students’ minds, we had one-third of the students do each task in each round of peer-reviewed assignments. That is, for the first due date in September, we had one-third of the students do explanations, one-third do examples, and one-third do exercises. For the second due date in October, a different onethird did each type of assignment, and ditto for the third due date in November. In all cases, students submitted their work, got feedback from their peers, resubmitted revised work, and then were reviewed again and graded by their peers. Following that, their reviews were reviewed by a different set of students. To make the process more interesting, to better identify the best submissions, and to have good submissions more thoroughly refined, we selected the eight best-reviewed submissions in each category for a “playoff” round in which a different set of reviewers gave feedback, and the authors were permitted to revise their submissions yet again. At the conclusion of the playoffs, each submission that made the playoffs received 10% extra credit. The second-place submission in each category (explanation, example, and exercise) received 25% extra credit, and the winning submission received 50% extra credit. One student rode two winning entries to a semester average of greater than 100%, the first time this has happened in the class in more than 10 years.
5. RESULTS Students reacted positively to this experiment. After the end of the semester, they were asked to fill out a Web survey that asked 10 questions about the textbook and the peer-reviewed assignments. Of the 78 students in the class, 49 responded, a response rate of 63%. Of these 49 students, 29 either agreed (13) or strongly agreed (13) that they learned a lot from doing the peerreviewed assignments (Figure 1). Twenty-seven of them said they enjoyed or strongly enjoyed the assignments (Figure 2). On the positive side, one student commented, These assignments turned into an extremely effective learning tool since it forces the student to research the concept and offer their own example, explanation, or exercise.
Of the students who did not like the assignments, several mentioned the multiple deadlines. Others also mentioned the problems with the peer-review system (we were
switching to the Apache Struts framework during the first
pointed out that there should never be two schedulers for landings
18 25
16 14
20
12 10
15
8 10
6 4
5
2 0 Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Figure 1. I learned a lot from doing the peerreviewed assignments related to the text.
review period, and were fixing bugs as they were reported). One area for improvement is in the quality of the reviewing: Lot of interest generated initially. But poor reviews partly due to poor understanding of people about how to review caused a waning interest. Perhaps a training to student on how to review so that a uniform scale exist.
This suggests that a classroom discussion of reviewing, using a submission from another semester, would be valuable next time around.
6. THE TEXTBOOK AUTHOR REACTS When writing the object-oriented design textbook that was used in the class, the author struggled to come up with good, simple explanations, examples, and exercises for each of the concepts he wanted to present. Through the Expertiza experience, the students in the course came up with a large amount of material for possible inclusion in the text. In fact, the author plans to use sixteen student contributions in the final version of the textbook. Some students provided new examples of concepts. For example, in the text the author explained the State pattern by way of a drawing application. In that application, a mouse click in the drawing pane had different behavior depending on which tool in the toolbar was currently selected. The current selection could be viewed as the current state of the application. A student felt that a better understanding of the State pattern could be obtained by thinking of a Swiss Army knife. The state of the knife, and so the way it will behave, depends on the particular blade of the knife that has been exposed. Another student came up with a good analogy for the Singleton pattern by considering air-traffic controller software. The student
0 Strongly Agree
Agree
Neutral
Disagree
Strongly Disagree
Figure 2. I enjoyed doing the peer-reviewed assignment related to the text.
and takeoffs from the same runway. A third student came up with an example of the Prototype pattern involving a default auto insurance policy that can be cloned and then edited to meet the specific needs of each customer. Another student explained that a picture of a bedroom is a Memento of that room in that, at any time in the future, the bedroom can be restored to its state at the time of the picture by using the picture to do the restoring. In other cases, the students provided examples of various concepts discussed in the book that were not already accompanied by an example. For example, in the book the author discussed the Rename Method refactoring and the problems that can be caused by renaming a method that overrides an inherited method or is overridden in a subclass or is one of the methods in an interface implemented by the class. However, the author provided no examples in the text nor did he give suggestions regarding what to do in the case where you cannot rename the method. One of the students came up with a simple example of this situation using the paint method of a JComponent and suggested a way to handle the situation. His solution was to leave the paint method's signature alone and instead use the Extract Method refactoring to extract the body of the paint method into a new method that is more appropriately named. Several other students expanded on the book's section on coding conventions to address issues the author had overlooked, such as the preferred use of capitalization and braces. Other students devised useful new exercises for the text. For example, some students came up with exercises involving adding new features to the drawing program discussed in the text, such as a new tool for rotating figures in the drawing.
Another student noticed that in one chapter the Factory pattern was introduced but there were no exercises at the end of that chapter directly related to that pattern. So the student created an exercise asking the reader to use the Factory pattern to create dining-room tables, where the particular kind of table that was created depended on the amount of money the buyer wanted to spend on the table. One student noticed that the Mementos that were created in a drawing program in the text in order to undo and redo changes to the drawing could also be used to save drawings to a file and to open those files. This observation led to an exercise asking the reader to implement the saving and opening of files of drawings. Other students created extensive exercises that were more like large programming projects. The fact that all the students were required to submit solutions to the exercises they submitted was also beneficial to the author. Some students felt they could come up with better explanations of design patterns than the explanations in the text. One student explained the Observer pattern by noting that human beings do not repeatedly check the door to see if someone is waiting to come in and they do not repeatedly pick up the phone to see if someone is on the line waiting to talk to them. Instead a doorbell and a telephone ringer are used to notify the person that someone is at the door or trying to reach them on the phone. Some students created explanations of design patterns that were not covered in the book. For example, one student motivated the Flyweight pattern by considering the thousands of blocks displayed in a SuperMario game. Instead of creating thousands of Block objects, the game can create just a few different blocks in a few different states and the game board can have thousands of pointers to those blocks. When a block in the game board needs to change, the game can just change the pointer instead of the block. Some student contributions that were intended as explanations or examples ended up as exercises. For example, some students suggested that, to ensure class invariants hold, assertions that check those invariants should always be included at the end of all modifier methods in a class. Since such checks are not always appropriate, the author instead included an exercise in the book asking the readers to explain the advantages and disadvantages of such a scheme. One limitation of student-suggested improvements to the text is that some students lack good writing skills. However, even poorly-written contributions were helpful to the author because all contributions, no matter how well written, needed to be rewritten to fit the author's style of writing. Some contributions forced the author to rethink what he presented and how he presented it. For example, one section of the book covered class and loop invariants and method preconditions and postconditions but didn't mention assertions. Several students commented about this omission. The author agreed that the inclusion of assertions is appropriate and so rewrote the section accordingly. Also students found some subtle, non-trivial errors in the implementations of some methods in the book, such as the equals method for Java classes, forcing the author to rework that section.
In summary, the main benefit of Expertiza to the textbook author is that it provided a way for students to act as coauthors or editors to go over the text and suggest improvements. The particular benefits of using students to do the editing and authoring include the following: • There are many students using Expertiza and so the author received many suggestions. • The suggestions are from the students' perspective instead of the perspective of an instructor or editor. • The suggestions reveal issues that confuse students and the suggestions provide explanations of those issues. • The suggested examples and exercises are presumably ones that some students found interesting and so have a greater chance of being appealing to other students than ones developed by instructors. • The suggested exercises are presumably ones at the appropriate level of difficulty for students. • The examples suggested by the students are often incomplete. Such examples provide good exercises for the book, in which the author presents the given example and asks the reader to complete it. • Those students who have prior experience in industry have sometimes come up with examples based on that experience.
7. FUTURE WORK In addition to being used to review writing, Expertiza can also be applied to the review of code. In this year’s offering of our OOD course, we are going to have students pair up, with each pair selecting a piece of code (Expertiza source code!) to refactor. Students will be asked to use as many refactorings as they can to improve the quality of their module of code. Then other students will be assigned to review their refactorings, in such a way that each student reviews three different refactorings of the same piece of code. In general, students will review refactorings of different code than they themselves refactored. The winning refactorings, based upon peer review, will be awarded extra credit. We believe this will serve as an excellent vehicle for students to learn refactoring, as well as to become familiar with Expertiza so that they can extend it in projects later in the course.
8. ACKNOWLEDGMENTS This work is supported by the National Science Foundation’s Division of Undergraduate Education under a CCLI (Course, Curriculum, and Laboratory Improvement) grant, by the NCSU Faculty Center for Teaching and Learning under an Instructional Grant, and by the NCSU Information Technology Division under a LITRE (Learning in a Technology-Rich Environment) grant. The authors acknowledge the contributions of the entire Expertiza group, including Susan Conger, Prasad Wagle, David Edelman, Michael Smith, and D. J. Malmberg.
9. REFERENCES [BW 98] [Bowe
Black, P., and Wiliam, D., “Assessment and classroom learning,” Assessment in Education 5:1, pp. 7-74, 1998. 92] Bowen, L.F., “The Daedalus Integrated Writing Environment,” Computers and Composition 10, November 1992, pp. 77–88.
[CF 00]
Chapman, Orville L. and Fiore, Michael A., “Calibrated peer review,” Journal of Interactive Instruction Development 12:3, Winter 2000, pp. 11–15.
[CW 02]
Clayton, Michael and Watkins, Adam, "Assessment in the Digital Arts," Syllabus 15, April 2002, pp. 31–34.
[Crav 94]
Craven, J., “A newmodel for teaching literature classes,” T.H.E. Journal 22, August 1994, p. 55–57.
[Gehr 01]
Gehringer, Edward F., “Peer review and peer grading in computer-science courses,” Proceedings of SIGCSE 2001: Thirty-Second Technical Symposium on Computer Science Education, Charlotte, Feb. 22–25, 2001, pp. 139-143.
[JMJN 81] Johnson, D. W., Maruyama, G., Johnson, R. T., Nelson, D., and Skon, L., “Effect of cooperative, competitive and individualistic goal structures on achievement: A metaanalysis,” Psychological Bulletin 89, pp. 47-62. [JJ 89] Johnson, D. W., & Johnson, R. T., Cooperation and competition: Theory and research. Edina, MA: Interaction Book, 1989. [OMR 00] Orsmond, Paul, Merry, Stephen, and Reiling, Kevin, “The user of student derived marking criteria in peer and self-
assessment,” Assessment and Evaluation in Higher Education 25:1, March 2000, pp. 23-38. [PW 04]
Plutsky, Susan, and Wilson, Barbara A., “A comparison of three methods of teaching and evaluating writing: A quasiexperimental study,” Delta Pi Epsilon Journal 46:1, Winter 2004, pp. 50–61.
[PJ 01] Pollock, L. and Jochen, M. 2001. Making parallel programming accessible to inexperienced programmers through cooperative learning. In Proceedings of the Thirty-Second SIGCSE Technical Symposium on Computer Science Education (Charlotte, North Carolina, United States). SIGCSE '01. ACM Press, New York, NY, 224-228. DOI= http://www.lib.ncsu.edu:2234/10.1145/364447.364589. [Stef 94] Stefani, L. A. J., “Peer, self, and tutor assessment: Relative reliabilities,” Studies in Higher Education 19:1, 1994, pp. 69-75. [Ster 04] Sterngold, Arthur, “Confronting Plagiarism: How Conventional Teaching Invites Cyber-cheating,” Change 36:3, May-June 2004, pp. 16–21. [Swai 05]
Swain, Harriet, “Nip double trouble in the bud,” The Times Higher Education Supplement No. 1676, p. 58, January 28, 2005
[VCBM 95] Van Lehn, K. A., Chi, M. T. H., Baggett, W., and Murray, R.C. (1995), Progress report: Towards a theory of learning during tutoring. Pittsburgh, PA: Learning Reasearch and Development Center, University of Pittsburgh.
