Roschelle 73..107

Interactive Learning Environments 2004, Vol. 12, Nos. 1–2, pp. 73–107

ESCOT: Coordinating the Influence of R&D and Classroom Practice to Produce Educational Software From Reusable Components Jeremy Roschelle and Chris DiGiano SRI International

ABSTRACT In a 3-year project, a consortium of university, nonprofit and commercial educational software developers formed a testbed for the rapid assembly of educational software from reusable component tools. This testbed incorporated interactive learning tools from a variety of university, nonprofit, and commercial developers, and hosted decentralized authoring teams consisting of teachers, developers and educational technologists. Within its testbed, the Educational Software Components of Tomorrow (ESCOT) project achieved notable success in (a) producing a large collection of technology-rich learning activities with reuse rates estimated at 90% and (b) scaffolding authoring teams in successful and rewarding collaborative development experiences. Fortunately, since ESCOT was funded as a National Science Foundation research project, we have had time to reflect on the conditions that led to our achievements. In this article, we reflect on the three activities of the project that we believe were most responsible for its success: (1) the selection of a unit of software production (2) the development of a strategy to allow reuse of interoperable software components and (3) the structuring of a distributed, team-based authoring process. We observe that a common characteristic across these activities was reciprocal influence from both the fields of researchbased software development and teaching practice.

INTRODUCTION Tools such as calculators, spreadsheets, and graphing calculators have a potentially profound role in enabling large numbers of students to learn complex mathematical ideas (Kaput, 1992). Tools can reduce some of the cognitive complexity of calculating with mathematical algorithms, allowing students to focus on conceptual understanding. Tools can present mathematical ideas in visual and interactive modes that are better tuned to students’ learning capabilities. Tools can make mathematical tasks more authentic; for example, Address correspondence to: Jeremy Roschelle. E-mail: [email protected] 10.1080/1049482042000300904$16.00 # Taylor & Francis Ltd.

74

JEREMY ROSCHELLE & CHRIS DIGIANO

a spreadsheet can help students apply the kinds of analyses that are used in business. And in some cases, such as ideas surrounding chaos and complexity, it is hard to engage with mathematical ideas in the absence of tools. Recognizing this potential, the National Council of Teachers of Mathematics (NCTM) recently revised its principles, adding the principle that ‘‘technology is essential’’ (2000). Despite the strong research backing for the use of technological tools in education and much research exploring the optimum design of tools for particular concepts, progress towards a usable collection of tools for education has been slow (Laleuf & Spalter, 2001). A recent survey (Becker, Ravitz, & Wong, 1999) shows only one research-based software tool in widespread use in mathematics education, The Geometer’s Sketchpad+. Hundreds of other innovative tools funded by the National Science Foundation and others have either never left the lab or never attained a large market. The underlying problem is at least partially on the supply side. In 1997, in the course of a planning project that eventually led to the Educational Software Components of Tomorrow (ESCOT), we surveyed the thousands of educational applets indexed by the Educational Object Economy (EOE) project (Spohrer, Sumner, & Shum, 1998). We identified many serious problems that made these widely available applets essentially unusable, such as: Lack of connections to curriculum & standards Lack of a lesson or activity to contextualize use Lack of clear learning objectives. These problems persist; there still is not a sufficient supply of educationallyappropriate, ready-to-use tools. Business-as-usual research and development (R&D) projects appear unlikely to meet this need, as most such projects are oriented to hypothesis testing by building proof-of-concept software. Policy setting organizations, such as the CEO Forum, are now turning their attention to the question of how quality digital educational content might be produced (CEO Forum on Education & Technology, 2000). Against this backdrop, we believe our ESCOT testbed stands out as a model worth understanding, refining, and replicating. ESCOT was a large, coordinated effort. The project was led by SRI International in Menlo Park, CA, with key subcontractors at the Math Forum (now located at Drexel University); the University of Colorado, Boulder; the University of Massachusetts, Dartmouth; KCP Technologies, Inc. (in Emeryville, CA); and the University of Missouri. All together, we had 7 major partners and 34 lesser collaborators (e.g.,

COORDINATING R&D AND CLASSROOM PRACTICE

75

organizations that contributed software, volunteers, or contractors). A key to the ESCOT network was the involvement of teachers who participated in design teams. Over the three years of the project, a total of 19 teachers were involved, hailing from 11 different states. During years 2 and 3 of the project, ESCOT produced 46 interactive mathematics puzzles in a regular series (weekly for year 2 and biweekly for year 3), which is now archived at http://www.escot.org. We delivered each puzzle – an interactive ‘‘curriculet’’ as we now call it – according to a tight time schedule and labor budget, assembling the curriculet from components derived mostly from third-party software. Our punctual delivery of software contrasts with the norm for software engineering: a 1994 survey of large software projects revealed that over 50% of projects were over budget typically by a factor of two, over schedule by a factor of two, and did not meet about two thirds of the original functional specifications (Standish Group International, 1994). Our use of components also contrasts with other projects. Although component reuse is a well-documented best practice in software engineering (IEEE Computer, 1997; Myers, 1997; Poulin, 1999), we are not aware of published accounts of large-scale reuse of third-party software. In educational technology R&D in particular, our experience is that for various reasons the common practice is to avoid resources ‘‘not invented here.’’ Each ESCOT curriculet was produced by combining successful educational tools from multiple vendors with some small amount of new, special-purpose code. The tools were interconnected and worked together in a single window to achieve a coherent educational purpose. For example, AgentSheets+ (a powerful simulation construction engine) or The Geometer’s Sketchpad (the best-selling mathematics application) was combined with graphing tools from the SimCalc Project. Our curriculets included a problem context that motivated and assessed students’ progress. This problem context consisted of a story that introduced the interactive tools to the students and a set of assessment questions, which included both short answer and longer, more indepth items. Each interactive curriculet was linked through a teacher support page to NCTM standards. The rich educational context of our curriculets stands in contrast to the majority of applets we surveyed on the EOE and other Web sites. By reflecting on how ESCOT achieved these successes, we hope to shed light on replicable practices that produce predictably high-quality digital learning resources within tight schedules and using limited labor (DiGiano, Roschelle, & Chung, 2001). We believe the key challenges these practices

76


must overcome are (a) the general lack of coordination among independent software producers in our field and (b) inadequate levels of capital available to fund large teams of research-based producers (Roschelle & Kaput, 1996). In this article we discuss three cornerstones of our approach. First, we argue that the careful selection of a target unit of production, a regularly published interactive curriculet, laid the foundation for ESCOT’s success. While other units of production might have worked, our curriculets had relatively unique aspects that made them well-suited for scalable, distributed production and testing. In particular, they were well-bounded, adoptable by teachers, readily connectable to curricula and standards, embedded in a data-gathering framework, and relevant across topics. We provide this section first, because it describes the educational context in which we worked. Second, we describe how ESCOT’s conscientious employment and continual refinement of a reuse and interoperability strategy made scheduled, labor-limited production possible. Important aspects of our strategy include a product line approach, an emphasis on supporting domain-specific design patterns, a continual push to identify and eliminate obstacles to reuse, and an orientation to building social networks, not just technological repositories. Third, we discuss how the interplay of central organization and distributed teamwork enabled parallel production of multiple curriculets on a tight schedule and with limited labor. We call our process ‘‘integration teams.’’ Particularly important aspects of our process include the definition of team roles, the establishment of a common timeline, the incorporation of multiple checkpoints in the timeline, and centralized support for communication and testing. After discussing each of these points, we discuss some of the challenges that lie ahead. In particular, ESCOT has been less successful than we would have liked in one area: the ease of preparing school-based computers to use our software. In the discussion that concludes this article, we argue that a major lesson from the ESCOT project is the importance of coordinating the reciprocal influence of research-based software development and teaching practice. In producing its interactive curriculets, ESCOT mediated between issues important to the R&D community and issues central to classroom practice. Perhaps more interestingly, we believe the nature of the ESCOT experience engendered reciprocal influence between these two fields by, for example, providing opportunities for members to better appreciate each other’s value systems. This coordination of influence is most evident in three aspects of ESCOT work: our choice of a unit of production, our approach to software


77

reuse and interoperability for educational applications, and our support for collaborative design teams. 1. Educational Factors: Selecting a Target Unit of Production ESCOT’s goal was to create technology in support of reform-oriented mathematics teaching in middle school. In particular, we oriented our efforts towards five recently published textbooks underwritten by the NSF for middle school mathematics: Connected Math, Math in Context, MathScape, MathThematics, and Pathways. Early in the project, a team led by David Barnes, then at the University of Missouri, reviewed these curricula, creating a database of technology needs within each of their units. Within this broad context, we organized a group of teachers and developers to brainstorm specific educational needs at the level of particular concepts or lessons that children have trouble with (see Section 3 for more details). To address this list of specific needs, we began to envision a series of targeted interactive tools that we would publish to middle schools. Rather than creating our own series from scratch, we elected to leverage the ‘‘Problem of the Week’’ service run by the Math Forum, which had been offered for several years prior to ESCOT. Each week the service offers a challenging new math puzzle, inspired by the National Council of Teachers of Mathematics standards. The ESCOT team decided it would add interactive tools to this existing format to create a series of ESCOT Problems of the Week (EPoWs). We have dubbed these EPoWs interactive ‘‘curriculets’’ because of the way they combined curricular materials and technology into a modest-sized bundle: there was a story, which introduced and motivated exploration, followed by one (and occasionally more) applet tools, followed by a short set of questions. As the series evolved, many curriculets tended to include additional common features. In particular, the tools tended to support a transition from ‘‘guess and check’’ to more reflective, analytical solutions by including a feature to record guesses for later analysis. Further, the first of 2 to 4 questions tended to be a simple 1, and factual, but the remaining questions tended to focus on explanation, problem-solving strategy, and generalization.

AN EXAMPLE CURRICULET The ‘‘Fish Farm’’ curriculet in Figure 1 illustrates these features. In this problem, students were asked to reflect on how to distribute a fixed number of

78


Fig. 1. The Fish Farm curriculet.

male and female fish among three ponds. They had to satisfy three mutual sets of constraints, since the owner of each pond wanted a different ratio of male to female fish. Students could experiment with combinations of fish by dragging and dropping animated fish into ponds. Not only did the transfer of fish affect the animation; it also triggered simultaneous updates to linked representations that included pie graphs. The design team presented Fish Farm in a context that included a carefully designed story to motivate student involvement and exploration. Further, a set of driving questions was provided to orient students to thinking about (a) the strategy they used to solve the problem, (b) mathematical generalizations of the problem they solved, and (c) how they could argue for the correctness of their solution. Each EPoW curriculet was delivered to schools via an interactive Web site. Students were able to submit their solutions to the curriculet to the Web site, and receive mentoring. In particular, the Web form routes student solutions to a ‘‘back office’’ where the solutions are placed on a queue. A pool of mentors works on the queue of solutions, providing a response to each student who submits. Often these responses ask the students to improve their solution. At


79

the end of the submission period, the mentors summarize the best solutions and post them to the Web for all to see. All student responses to the mentoring service were logged to a database, which was then available for analysis. Further, mentors interacted with students, and these interactions were also logged. These logs were then available for analysis. We performed a cursory analysis immediately after the conclusion of every EPoW, and the resulting feedback was made available to the design team for that curriculet and future curriculets. A result of this analysis, for example, was that early curriculets did not generate as much evidence of higher-order thinking as we anticipated; later curriculets added tools and questions to provoke more reflection, generalization, transfer, etc. In addition, we have used these data in a more detailed evaluation study, which also included videotaping in selected classrooms that were using EPoWs and interviews with teachers. This analysis is in preparation for subsequent publication. Below we reproduce part of one 13-year-old’s solution to the Fish Farm curriculet. This solution was one of those highlighted by the mentors as exemplary. Notice that the student described her strategy in terms of ratios and used arguments about the properties of mathematical operations with ratios to justify her approach. Angel had 8 male fish and 8 female fish in her pond. Molly had 3 male fish and 1 female fish in her pond. Gar had 2 male fish and 4 female fish in her pond. I first put one male and one female in Angel’s pond, 3 males and 1 female in Molly’s pond, and 2 males and 4 females in Gar’s pond. I thought I could put the rest into Angel’s pond, but I noticed that there were unequal amounts of males and females. So to make it equal, I put one more male fish and 2 more female fish in Gar’s pond, that would still be the same as 1:2. That left me with the same amount of male fish and female fish, so they could all go into Angel’s pond (that would still equal 1:1). Since I did everything slowly, I made sure that my amounts of fish were equal to the ratios. All I did was get the total amounts and then reduce them, and the reduced number should’ve equaled the ratio. I knew I got everything right when the bricks turned green. REFLECTIONS ON OUR UNIT OF PRODUCTION As we began to assemble our initial component set in the our first year and started solving interoperability problems, it became clear we would have to set

80


some bounds on how we would use the software in order to focus the production effort. Further, we believed that we would have better success in organizing the members of our project team, which spanned institutional boundaries, if we were creating a common kind of object. In response to these concerns, we arrived at the curriculet structure, which bound EPoWs in three ways: 1. In time: EPoWs were intended to occupy approximately 40 min of a class period, although we aimed to make them tough enough to take even the best students 10 min, and interesting enough that a good teacher could extend them to last all week. 2. In pedagogy: most EPoWs were exploratory and open-ended, but all had to have at least one short-answer or multiple-choice question that was straightforward for a mentor to grade. 3. In curriculum: users already expected a Problem of the Week series to address a coherent grade level and/or topic area. It was therefore natural to constrain our EPoWs to middle school math (more specifically, we often aimed at 7th grade). While constrained in productive ways, the EPoW curriculet was sufficiently open to accommodate the ideas and aspirations of many different individuals. Some team members were interested in highly constructivist math learning approaches. Some were interested in helping students overcome particularly difficult conceptual barriers. Some were interested in a particular kind of tool. In every case, the individuals could find some aspect of their interests that could be profitably represented as an interactive curriculet. In retrospect, the choice of the EPoW as ESCOT’s unit of production was critical to the success of the ESCOT process. From the design perspective, the clear pedagogical objective of an interactive curriculet provided a common focus for an ESCOT integration team, but the various elements of the curriculet allowed work to be divided along disciplinary lines. The fact that our curriculets were published on a regular basis meant that we were forced to schedule our production process carefully, which had the nice effect of distributing the workload across the year. The weekly, mentored structure also made it possible for us to test every curriculet we produced and use the resulting feedback in upcoming designs. From the perspective of adoptability, an important benefit of EPoW curriculets was that they were easy to relate to curriculum and standards and could be used by teachers in many different ways. Some teachers used them as a


81

computer lab exercise; some teachers used EPoWs for whole-class discussion; some teachers made EPoWs optional, for extra credit; some students did EPoWs on their own, without an assignment from their teacher; and at least one teacher used EPoWs as the basis of her whole curriculum. Given that curriculum publishers remain reluctant to incorporate technology directly, the fact that curriculets could supplement the mathematics curriculum in such a flexible way was a great asset. The consistent yet flexible structure of our curriculets and their bounded scope helped the technical process considerably, since they required similar types of Java applets that could be assembled via components (to be discussed further in Section 2). The similar structure and scope of curriculets also allowed us to generalize across our experiences in producing them according to a schedule and delivering them to classrooms. This made it possible to better support integration teams with standard processes and checkpoints (to be discussed further in Section 3). In short, we believe that many of ESCOT’s accomplishments can be explained by our decision to generate a series of interactive curriculets. We would not go so far as to claim that a curriculet series is the perfect format for all technology-supported curricula. For example, there is a place for broader interventions developed over an extended period that require teacher training. However, our experience indicates that the regular creation of more focused products can enable even modest research efforts to be highly productive and achieve classroom impact. Abstracting from the specifics of our EPoWs, we suggest that a good unit of production is:

Well-bounded enough to structure the production process General enough to encourage participation from many contributors Easily adoptable by a diversity of target users Readily aligned to curricula and standards Deliverable on a regular schedule that spreads the work load over time Susceptible to standardized automated data collection.

2. Technical Factors: Reuse and Interoperability Strategy ESCOT’s project title refers to ‘‘component software,’’ and, indeed, much of our work has revolved around a technical vision of modular, mix and match software tools (Roschelle et al., 1999). This vision builds on the longstanding component software goal (McIlroy, 1968) and the newer work of the Educational Object Economy to apply that vision in education (Roschelle,

82


Henderson, Spohrer, & Lilly, 1997). We have argued that educational software production will not be able to meet all of teachers’ and students’ needs within the typical model of creating independent, standalone software applications, and that reuse and interoperability among separately produced component tools is necessary to overall success (Roschelle, Kaput, Stroup, & Kahn, 1998; Roschelle et al., 1999). In formulating our reuse and interoperability strategy, we drew upon the Software Engineering literature, especially with regard to component-based systems (Jacobson, Griss, & Jonsson, 1997; Myers, 1997; Poulin, 1999; Rine, 1997). Overall, within this literature, the jury is still out on how effective component-based engineering is in real-world practice. A raison d’eˆtre of ESCOT, therefore, has been to further define and evaluate the viability of component software in education. ESCOT is not alone in its overall quest. Indeed, a variety of educational projects are addressing interoperability in education, although with many different orientations. Five specific levels of interoperability are: 1. 2. 3. 4. 5.

Descriptors of objects Databases of student records Course delivery systems Media elements Tools such as spreadsheets, graphs, simulations and geometry visualizations.

The Instructional Management System (IMS) project has focused on ‘‘metadata,’’ card catalog-like descriptors of digital learning objects and activities, following related work with the Dublin Core (http://dublincore.org/) and the IEEE Learning Technology Standards Committee (http://ltsc.ieee.org/). Digital Library efforts, such as the National Science Digital Library (Wattenberg, 1998) are also concerned with reuse and interoperability at the card catalog, or descriptor, level. Another area of emphasis on interoperability in education is in the back office systems used to track student registration, attendance, grades, etc. The School Interoperability Framework (http://www. schoolsinterop.org/) is one effort in this area. Course delivery systems, such as Blackboard and WebCT support reuse more generally by providing an overall framework in which learning takes places and in which domain-general tools (like bulletin boards and chat rooms) may be reused. The Sharable Content Object Reference Model (SCORM) defines a non-product-specific Web-based learning ‘‘Content Aggregation Model’’ and ‘‘Run-Time Environment’’ for learning objects (http://www.adlnet.org). Some projects, such as the ‘‘Courseware Conversion Factory’’ (http://www.teknowledge.com/training/research.


83

htm) have focused on reuse of general features of lessons, such as paragraphs of text, images, or movies. ESCOT, by way of contrast, has focused on the granularity of learning tools (Roschelle et al., 2000). Although we use the broad term ‘‘educational software,’’ we focused specifically on the tool or applet level. At this level, an assembly supports students’ integrated use of multiple representations (e.g., graphs, tables, and invented representations such as the fish pond simulation from our earlier example) to explore mathematical meaning and solve problems within a narrowly defined activity and curricular context. To create the parts for these assemblies we employed open software toolkits (diSessa, 1997; Roschelle & Jackiw, 2000) such as The Geometer’s Sketchpad and AgentSheets to generate preconfigured components. These components could then be coupled to other ESCOT components, such as a slider or a graph. This is a variation on the typical vision for open-software toolkits, which involves teachers doing the configuration to form ‘‘microworlds’’ for particular investigations or activities. In our case, ESCOT staff (as part of the integration teams described in the next section) did the configuring to craft the particular representation needed to support the design of a new curriculet. Other educational projects have explored a tool-level component approach, although they have focused on components developed within a single project, engineered from the beginning to form a coherent set. For example, through informal joint work, we are aware of several investigators, including Ron Thornton, David Hestenes, Fred Goldberg, and Bill Finzer, who explored educational component software in the context of OpenDoc. OpenDoc (Feiler, 1996) was a component framework that was heavily promoted by Apple Computer, Inc. and then suddenly canceled. More recently, the E-Slate project, based in Greece, has pursued a trajectory very parallel to ESCOT, focusing similarly on JavaBeans technology (the two projects have had annual joint project meetings). E-Slate (Birbilis, Koutlis, Kyrimis, Tsironis, & Vasilou, 2000) has a much more refined authoring tool than ESCOT and a noteworthy collection of components (oriented mostly to topics other than middle school mathematics). Boxer (diSessa, 2000) is another component-oriented project. Whereas ESCOT’s components are mostly black boxes, Boxer is exploring a white-box orientation. Boxer typically aims at providing software with its source code, so that a technically inclined user can usually explore the full source code of anything they see in the Boxer environment. Several other projects have produced series of mathematical applets from components. Two exemplar sites are Project Interactivate (http://www.shodor.org/interactivate/)

84


and the National Library of Virtual Manipulatives for Interactive Mathematics (http://matti.usu.edu/nlvm/). In contrast to the projects and products listed above, ESCOT is unique in producing an applet series that draws upon existing components with origins outside the project lead’s organization. The components in our library were not produced by a single organization, nor were they engineered with the requirement that they fit together as a whole. We did work with some organizations to improve the compatibility of their components, but (especially for the final iteration of our interoperability strategy) this was largely to ensure that the parts conformed to the JavaBeans standard rather than to our own idiosyncratic conventions.

DEFINING AND EVALUATING REUSE IN ESCOT Within the tool level described above, we defined ‘‘reuse’’ to mean building new tools (particularly the applet portion of a curriculet) by incorporating large chunks of functionality from existing parts, with the intention that most of the code would be assembled from existing resources, rather than handcrafted code for the particular applet. To accomplish reuse, ESCOT accumulated a modest-sized library of existing parts. A limited collection of components can simplify the cognitive demands on the developer, who must make a ‘‘build or buy’’ decision when faced with a need for new functionality. Cognitive studies of reuse (Fischer, 1987; Fisher & Redmiles, 1995) indicate that locating, comprehending, and adapting someone else’s code can be a surprisingly onerous task. Indeed, recent empirical findings in component-based development suggest that a small collection (20–200) of high-quality components that are well understood by a development team is generally preferred over a huge collection of uncertain-quality components (Poulin, 1999). Experience in component-based development also points to the value of a ‘‘product line’’ approach, in which components are selected for their ability to support a well-defined range of applications (Hopkins, 2000; Poulin, 1999). In ESCOT’s case, we had a natural product line: the series of interactive curriculets we planned to produce each year. Through a triangulation among other data sources, including the database of needs described in Section 1, we developed a list of approximately 40 components that made sense for our product line (Roschelle et al., 2001).


85

These included tools specific to curricular themes (e.g., for geometry, algebra, probability and statistics), as well as some utilities specific to education (e.g., data recorders, tools to support reflection). We started our component collection with three ‘‘anchor tenants:’’ SimCalc, The Geometer’s Sketchpad, and AgentSheets. These were known to be powerful tools that could generate a wide range of needed representations, and thus we anticipated that time spent familiarizing our integration teams with these systems would be a good investment. Analogous to the large department store in a mall, an anchor tenant tool could serve as the main attraction in a curriculet’s functionality. To provide supplemental controls, displays, and calculations we acquired other prebuilt components, such as an algebraic-expression evaluator and a Logo interpreter. Finally, we built several small-grained components, such as a widget for entering and displaying fractions and a logical component for aggregating values from other parts. Our decision to build a few components for ourselves followed some poor experiences with ‘‘found’’ components identified via general searches on the Web or in the EOE collection (Roschelle et al., 2000). In all, we assembled a collection of about 30 components. The Fish Farm curriculet (Fig. 1) illustrates the use of this library. AgentSheets is the anchor tenant component in the applet for the curriculet, proving the interactive simulation of fish swimming in ponds. The data about the fish count in each pond is dynamically linked to three pie chart components and to ‘‘number-edit’’ components (in this case, in view-only mode) that show the ratios. In addition, there are some utility ‘‘text label’’ components in the applet. That this applet is a composite of different underlying parts is invisible to most users; the applet functions as one coherent tool, with the purpose of allowing students to see how different combinations of objects produce mathematical ratios. In addition, the same components were also used in other applets, in different assemblies for different educational purposes. Early in the project, we encountered some of the tradeoffs that make reuse hard: 1. Knowing versus Learning. Especially when working within tight schedule constraints, developers have to choose whether to spend time to learn to use a new component, or to build the component themselves based upon what they already know (Fischer, 1987). The tradeoff can be especially difficult when developers feel relatively confident in their estimates of time to build it themselves and less confident about how long it will take them to learn the new component.

86


2. Reuse versus Production Values. Components are almost always less finely tuned to the task at hand then custom code would be. Thus it may not be possible to achieve as highly polished a look and feel as one could with custom code. 3. Packaging versus Building. It takes time to package software as a reusable component, and the time spent in packaging work is not spent adding features or building useful final products with the software. Our response to these very real tradeoffs has been to take a pragmatic, rather than a dogmatic approach to components. The ESCOT team did not insist that every final product be assembled completely out of preexisting components; on the contrary, we encouraged developers to create new code where small pieces of custom functionality were needed. Developers had to write some of these pieces in Java and incorporate them into the target applet. However, developers often could create these pieces with the help of testbed tools high-level, graphical authoring environments such as AgentSheets and The Geometer’s Sketchpad. We dubbed these tools ‘‘component generators’’ because they were not used directly in any applet but instead were employed to produce applet-specific functionality. A third way to add some additional code to applets was through scripting languages, including Logo, Python, and JavaScript. In reviewing the revisions of the core architecture, one important trend was to gradually centralize the interoperability burden: we sought to make it easier and easier for owners of individual components to make their components support domain-specific interoperability patterns. With each release, we also increased the number of test cases in the system, and thus built up confidence in the quality of the core interoperability code. The core team was not limited to architectural design. They were also charged with pragmatic assignments targeted at making reuse easier. For example, the core team produced a template project that could be copied to form the basis of a new applet. The team modified a freeware authoring tool to support direct packaging of designs as running applets. It created an applet deployment framework to ease the burden of getting applets to run across all the browser and operating system combinations we supported. Perhaps most importantly, the core team created a series of events at which it could listen to the needs of developers and the problems they encountered in reuse. These included periodic postmortem analysis meetings, preview and feedback sessions on new architectural designs, and formal advisory board

87


meetings. Further, the core team organized a social network of developers. In practice, a developer with a lead role in a particular integration team would often call or email other developers for help in resolving obstacles with components that the other developers knew best. This made the knowingversus-learning tradeoff much easier; developers are more confident about reuse when they know they have other developers supporting them. To measure the impact of reuse, we analyzed all our software changes during the last year of the project for a subset of EPoW curriculets. Table 1 shows the results, which are also depicted in Fig. 2. Table 1. Reuse Statistics for Six Interactive Curriculets. Curriculet

Lines changed (%)

Lines created (%)

Total (%)

Zooming Irrationals Galactic Suzie Scale n Pop Search and Rescue

0.4 0.2 1.4 3.7 2.3 1.1

6.3 10.1 15.7 4.2 5.1 6.4

6.7 10.3 17.1 7.9 7.4 7.5

Average Standard deviation

1.5 1.32

8.0 4.3

Fig. 2. Analysis of ESCOT reuse by curriculet.

9.4 3.92

88


We based our analysis on a Concurrent Versions System (CVS) repository that tracked every version of our source files. This repository can report all the newly added or changed lines in our source files. We analyzed the six curriculets produced in our second year for which all changes were logged in our repository; the other 10 curriculets used components in their applets whose source code was not accessible to us but which was tweaked to support particular applets. We counted lines of source code total, as well as the new lines created and the existing lines changed.1 As shown in Table 1, ESCOT appears to have achieved approximately 90% reuse. By way of comparison, projects cited as exemplifying reuse report levels ranging from 60–93% (DOD Software Reuse Initiative Program Management Office, 1995). Of course, this is only a rough comparison because the degree of difference between ESCOT artifacts may be different from the differences between other cases of reuse. Indeed, if all of our curriculets involved graphing of linear equations, one would not be surprised to see 90% reuse or better. One can visit http://www.escot.org to see the wide variety of interactive curriculets produced within our general focus on middle school mathematics. Given the variety of our curriculets we believe that 90% reuse is a meaningful level, especially given that our developers were distributed geographically. Defining and Evaluating our Interoperability Strategy ESCOT’s goal of embedding preexisting mathematics tools from various vendors into component assemblies required a carefully thought-out interoperability strategy. Our approach needed to address three key challenges: 1. Capturing the state of a component assembly during design time and restoring this later when students launch the applet for a curriculet (this includes the states of individual components and their interconnections); 2. Displaying the composite tool in one window, with one coherent user interface; and, especially, 3. Enabling real-time dynamic linkages among the mathematical functions and data displayed by each component. 1 The count was performed for each applet after removing any source files not actually used in the production code for that applet. We added a ‘‘binary objects’’ adjustment to the count of source lines for each applet. This adjustment was based on binary objects that were reused as components. To compute this adjustment we multiplied the binary size of the components by a source-to-binary ratio. We determined the source-to-binary ratio by measuring both source and binary sizes for our source tree and computing a ratio.


89

The last challenge was a benchmark problem for ESCOT, as the potential educational benefits of linking multiple representations are well known (Goldenberg, 1995; Kaput, 1992; Kozma, Russell, Jones, Marx, & Davis, 1996). But could we link measurements of geometric figures in one vendor’s geometry component to graphs in another vendor’s graphing component? Could those measurements update in real time as the student used a mouse to distort the shape of the geometric figure? In this article, we limit our scope to describing our interoperability strategy; a planned publication will contain engineering details, including an account of how and why our approach diverged from standard Java features (DiGiano, Roschelle, & Chung, In Preparation). At the core of our technical interoperability strategy were three main ideas: 1. Identifying common ‘‘patterns’’ of connections between educational components and making these trivial to accomplish with our framework, 2. Targeting domain-specific interoperability needs, and 3. Supporting a range of component types from generic JavaBeans to components customized to take advantage of ESCOT enhancements. Identifying Common Patterns of Connections To identify common types of connections between components, we reflected on our own software designs and reviewed the literature for popular techniques. We captured our findings as ‘‘software design patterns’’ (Gamma, Helm, Johnson, & Vlissides, 1995), a standard structure for communicating programming strategies. We report elsewhere on some of the key patterns we have identified such as Coordinated Properties, Shared Model, and Invoke on Event (DiGiano & Roschelle, 2000). The pattern form gave us a succinct medium for internal discussions, allowing us to think through changes to our interoperability framework without prematurely committing to a particular implementation. Furthermore, our connection patterns offered a common vocabulary for vetting our plans with ESCOT stakeholders. Consistent with the typical form of design patterns, we identified examples for each design pattern that allowed us to reason concretely about the tradeoffs between alternative interoperability strategies. Targeting Domain-Specific Interoperability Needs Early in the project, the ESCOT team recognized some critical interoperability needs that were specific to educational tools and not well supported by conventional interoperability approaches. One such need was support for

90


polyarchy among components, where any one of several linked components can determine an application parameter. For example, a simulation-based curriculet could feature several components for adjusting the rate of the simulation, such as a number entry box, a slider, and a bar chart. For pedagogical purposes, it can be important for the user to be able to change any one of these representations and observe how the related representations respond. Polyarchy is not as common in business applications where there tends to be a one-to-one correspondence between control components and application parameters. Another educationspecific need was support for capturing a history of the state of one or more components so that students could reflect on the choices they made and better strategize their next moves. Conventional interoperability frameworks for JavaBeans make no special accommodations for such data collection. By contrast, ESCOT we provided a standard data recorder component that made it easy to monitor an arbitrary number of components and display aggregate data in tabular or other forms. Supporting a Range of Component Types To reduce barriers to contributing components to the ESCOT testbed, we sought to support a variety of component types, ranging from unaltered JavaBeans to components customized to take advantage of ESCOT-specific enhancements. Our framework could recognize – or to use component parlance, ‘‘discover’’ – the interoperability characteristics and other attributes of candidate components in three different ways. First, our framework recognized the properties and event-handing capabilities of components that followed standard JavaBeans conventions. Second, our framework checked for ESCOT-specific metadata that exposed component capabilities that were unadvertised by JavaBeans conventions. Finally, our framework recognized a component that implemented ESCOT-specific interfaces, enabling the component to alter its capabilities dynamically. Such changes could be in response to (1) a developer linking to or from the component while creating the assembly, or (2) a change to the component’s configuration during assembly or while the assembly initializes itself after being launched by a user.2 Our three-tiered approach was designed to 2 Some component generators used in ESCOT, such as Geometer’s Sketchpad, actually created configuration files intended as input to a generic ‘‘player’’ component – in the case of Sketchpad, the Java Sketchpad applet component. Before loading a configuration file, such player components essentially had nothing to connect to other components. After loading a configuration file at run time, the component initialized run-time data structures that reflected the capabilities requested by the configuration file.


91

allow component developers to incrementally leverage ESCOT enhancements as the need for such enhancements became clear over the course of our curriculet series. Along with these architectural evolutions, we tried to maintain authoring tools to make it easier to assemble components through drag and drop operations. We recognized early on that we did not have a sufficient budget to build a serious authoring tool, so we focused throughout the project on tuning available tools to fit our needs. Although we have no formal evaluation of our interoperability strategy, we know that it served our curriculet production needs well. The design patterns we identified elevated our technical conversations to an efficient level and led to a framework that supported dynamically linked components across the range of our library. Our domain orientation helped us meet our pedagogical goals by enabling students to explore multiple representations and reflect on the history of their actions. Finally, our support for a range of component types allowed us to leverage well-tested third-party components, and yet we were able to compose these heterogeneous parts into unified assemblies.

REFLECTIONS ON REUSE AND INTEROPERABILITY We hired an external evaluator, Dr. Ted Kahn of DesignWorlds, to interview the developers with whom we worked and assess the effort involved in (1) ensuring compatibility of their components within our interoperability framework and/or (2) using the framework to connect components for a particular curriculet. At the end of our second year, when his evaluation was complete, Dr. Kahn reported: This success is especially impressive when we realize that this project is still less than two years old. The kind of sophistication in communication, collaboration, and project management that must be in place to support this kind of real production and presentation of interactive Web-based learning experiences was in good part a synergy . . . . Such collaborative successes with technologies that are still not mature and in the environment of very rapid changes in the world of the Web shows a kind of creative agility and adaptability of this testbed that is highly unusual, even among seasoned commercial companies in many different industry sectors (not just education).

92


Dr. Kahn’s remarks highlight the fact that reuse and interoperability are a management, communication, and collaboration challenge, not just a technical challenge. To overcome these challenges, our experience is that a dedicated team is needed to help developers locate, comprehend, and adapt foreign code. This team can further help by organizing the most useful components into a small, comprehensible set that can serve a limited ‘‘product line’’ of assemblies – in our case EPoWs. The ESCOT experience demonstrates how such a product line can narrow the interoperability and deployment requirements of a project, avoiding an endless quest for the ‘‘perfect’’ framework. Instead, we were able to focus on recurrent developer needs and design patterns that would address them. We iteratively refined our interoperability support to ease the burden of adoption on developers and thus to encourage them to choose reuse over building components from scratch. 3. Team Factors: Integration Team Process A third major factor in the success of ESCOT’s production process was careful structuring of the interplay between centralized management and distributed curriculet production teams over the course of a planned timeline.3 Each team that produced an interactive curriculet worked independently, but within an overall framework and with frequent checkpoints established by a central management team. Figure 3 shows the interplay between the centralized and decentralized elements over time. At the start of each of the 2 years of curriculet production, the management team organized key production resources and constraints. The resources included a collection of software components, curricular materials that illustrated classroom needs, and example curriculets. The constraints established a target scope for an interactive curriculet and a set of criteria a candidate EPoW would have to address. For example, these criteria included clear identification of an important and difficult math concept, a meaningful role for technology, a context that would motivate most students, an approach that could help students learn, and the feasibility of developing the curriculet on a limited budget. 3

(Repenning, Ionannidou, Payton, Ye, & Roschelle, 2001) provides an account of our process as it stood at the end of its first year. While this account misses the year 2 refinements, it does speak more strongly to software-engineering aspects of the integration-team process than the account we provide here.


93

Fig. 3. Interplay Between Integration and Management Teams.

These resources and constraints set the stage for a face-to-face workshop, held in the summer in Swarthmore, PA. A roughly even number of developers and teachers were invited to this workshop, along with some experts in the Problem of the Week format and in educational technology. The workshop was carefully facilitated to establish a climate of mutual respect and trust among the participants. As a group, the workshop attendees identified important math concepts, and then subgroups consisting of a mix of teachers, software developers, and educational technologists formed teams to brainstorm the design of interactive curriculets relating to selected concepts. The goal of each team was to develop a storyboard that characterized a proposed curriculet and to present their storyboard to the other attendees. After the storyboards were created, the leadership of the ESCOT project reviewed the proposed curriculets and determined which were technologically viable, which covered standards and middle school curricula best, and which were sufficiently defined to be implemented in a reasonable period of time. We emphasize that our top criterion was identifying a compelling mathematics education opportunity (‘‘Where’s the math?’’) and that other criteria were secondary filters.

94


Choosing among storyboards that rated highly on all criteria, the teams then came up with a schedule of weekly curriculets for the school year. A different team signed up for each interactive curriculet (some individuals participated in multiple teams, but except in the case of a few curriculets that used common storylines, teams generally disbanded after publishing their one curriculet). A team minimally consisted of a teacher, a developer, and a producer. In practice, there were usually two teachers, and often the lead developer was supported by additional developers who worked behind the scenes. The producer’s job was to keep the curriculet on schedule and to make sure the team worked together effectively. Producers worked on multiple teams and were expected to support each other in learning from prior experiences and solving emerging problems in teams. One centralized production team supported all the integration teams. The roles on this team included: 1. Executive Producer: overall responsibility for schedule and quality 2. Component Coordinator: maintained the component collection used by all teams 3. Testing Coordinator: managed a pool of testers used for all curriculets 4. Teacher Support Coordinator: ensured quality of the materials designed to support teachers’ use of the curriculets 5. Head Mentor: managed the mentoring function of live EPoWs. About 2 months before a curriculet was scheduled to go ‘‘live,’’ each integration team met to start their major work. They met either on the phone or using AOL’s Instant MessengerTM tool, to review the storyboards they had developed during the summer. Their first task was to refine these storyboards into a workable design. Often at this stage much work went into designing the ‘‘driving questions’’ that students would be asked to answer, and into defining a feasible approach to assembling the required technology. Once the design was clear enough to begin development, the centralized production team was given an opportunity to review and critique the design. The integration team then took on roles to develop the curriculet. Most often teacher members of the team were responsible for the textual aspects of a curriculet: the motivating story, the driving questions, and the teachersupport page. Developers were responsible for the assembly and testing of the Java applet. Producers were responsible for facilitating the process and keeping everything on track. The team met via conference call or AOL Instant


95

Messenger at last once each week to discuss progress and problems and continued to talk over email during the week if necessary. Two weeks before the curriculet was to go ‘‘live,’’ it was due for testing. During the first year, the executive producer and other Math Forum staff gave feedback to the teams. That turned out to be a lot of work, so we recruited an external testing group the second year, which included students and teachers who worked for an hourly rate. The team then had about 10 days to implement changes based on the feedback before the curriculet would go live. While each curriculet was live, students could submit solutions and receive mentoring. A small pool of mentors, drawn mostly from the ESCOT team, responded to each student or group of students by email. The head mentor managed this process and coached mentors on the appropriate tone of their remarks to students and the types of hints that are helpful. The mentors used rubrics provided by the Integration Team to grade the student responses. At the close of each EPoW, the head mentor and executive producer selected best solutions to ‘‘highlight’’ on the web page as models. It is prestigious for a student’s work to be selected as a highlighted solution for a Problem of the Week. Simultaneously, the head mentor and executive producer compiled feedback for the Integration Team on how well the curriculet worked. Thus, Integration Teams received feedback on the success of their design within about 3 weeks after they completed their work. At the end of our first year of curriculet production, evaluator Ted Kahn reported: ESCOT, the Math Forum and other ESCOT testbed partners’ early success in supporting this kind of national collaboration of multiple, geographically distributed participant communities – teachers, volunteer and commercial technology developers, educational technology and curriculum specialists, and mentors – in the service of launching and supporting a full year’s series of interactive problems and challenges offered to teachers and students over the Math Forum’s web site, is one that may well be unique – not only in federally funded educational R&D projects, but also, in many commercial high tech companies. In both years of curriculet production, an external evaluation team led by Wes Shumar debriefed each member of each Integration Team after the completion of their EPoW. This evaluation team interviewed each team

96


member over the phone using an interview protocol that focused on three factors: 1. Belonging: To what extent did the members of the group subscribe to common goals, understand their roles, and feel their contributions were desired? 2. Worldview: To what extent did the members of the group develop a common perspective on the learning experience they were designing and the role of technology in it? 3. Communication: How smooth or troublesome was communication within the group? Shumar’s evaluation team is reporting their detailed findings in a separate publication (Shumar, in press), hence we report here only their high level findings. Generally speaking, the evaluators found that the teams were able to work well together in designing their curriculets and were able to solve technical and pedagogical problems with little difficulty. Close working relationships emerged, and the team members valued their time together. Overall, the groups seemed to do a good job developing a shared set of understandings. Programmers were often sensitive to pedagogical issues and some teachers were interested in the programming issues. Most members of the integration teams believed it was a very worthwhile learning experience. However, the evaluators also noted some cases in which integration teams experienced problems. An unexpected problem on some teams was leadership. Occasionally, one of the group members took on the leadership role but worried about being too controlling. It was difficult for this person to sense how the others were feeling about their leadership, since the interactions were conducted over email and IM. Some groups thought leadership was lacking. A dominant issue in worldview was the tension between the technical sophistication of an interactive curriculet and a strong focus on student learning. Developers often pushed for more technical complexity and excitement; teachers often pushed for simpler, more focused curriculets. These findings resonate with related findings from the ‘‘work circle’’ model (Reiser et al., 2000). With regard to communication, teams that established a strong design and strong relationships at the summer workshop tended to do very well in continuing their work together via instant messaging and email. Some teams,


97

however, did not establish strong relationships or a strong initial design and found it hard to get focused without face-to-face interactions. Based on this evaluation, we believe that a key component of the integrationteam approach was our face-to-face integration-team workshops. Participants reported that the workshops were a particularly rewarding and energizing way to begin a new school year. They also felt that the meeting was critical for establishing and reestablishing the working relationships that carried teams through the development of curriculets at a distance. The lessons we learned about how to organize distributed production around reusable interoperable components are likely to become increasingly important as digital libraries of reusable objects emerge and educators seek ways to rapidly build educationally valuable lesson plans from raw resources. Another important aspect of integration teams was the balance of roles. We believe that teachers and the developers each offered important perspectives: teachers had unique knowledge of classroom settings, how children learn, and curriculum; and developers had unique knowledge of the intricacies of creating highly interactive software. Another role that we realized was critical was that of producer. Without the careful scheduling and occasional prodding by the producer, many of the curriculets might never have been finished. Yet another key role was the instructional technologist who helped bridge communications barriers between the teacher and the developer worlds. This role was often played by the same person who played producer. Finally, some roles that we experimented with in year 3 showed promise: graphic artist and student critic. We believe that curriculets that benefited from an artist had a more professional appearance and were attractive to students. When we involved student critics, we were pleased to have their ‘‘kid insight’’ into what makes an engaging curriculet. ESCOT integration team roles are similar to those adopted by other curriculum-design teams. For example, we compared notes with the Center for Learning Technologies in Urban Schools in a June 2001 meeting in Chicago. We found that their Work Circles (Reiser et al., 2002; Shrader, Williams, Lachance-Whitcomb, Finn, & Gomez, 2001) have a similar mix of developers, educators, instructional technologists, and producers. One notable difference was that work circles also often included policy makers such as school administrators. One explanation for the need for this additional role is that a work circle typically generated a multiday intervention that might require administrator approval, while a curriculet might occupy only a single class period.

98


CHALLENGES AHEAD Chief among our disappointments in the ESCOT project has been the problem of disseminating finished curriculets to students. The crux of the problem has been the failure of the ‘‘write once/run everywhere’’ promise of Java. All in all, we estimate that we spent over 50% of our development time determining how to make Java run on our target platforms. Despite our efforts to integrate cross-platform support and troubleshooting into our applets, they were still vulnerable to seemingly minor updates to browsers or Java installations. For instance, when Apple released a slightly new Java installation with caching, we suddenly had to add a new workaround to load images differently, depending on whether the image was in an Apple or a Windows image cache. We were also stymied by school computers with insufficient RAM to run Java in a browser, or with default settings (which were reset every night by a centrally administered script) that prevented Java use. On the other hand, there is no obvious component platform alternative to Java for educational tools. Java has very nice properties for a component testbed: Many educational developers are already using Java, and it provides a nice framework for packaging tools as reusable components (e.g., JavaBeans). Java has sufficient power to code complex tools such as spreadsheets, algebra expression parsers, and dynamic geometry engines. By way of contrast, consider an alternative such as Macromedia Flash. Flash can render impressive applets, but it: has only a rudimentary component framework to allow combining tools from different vendors and lacks the mathematics libraries and object orientation that would facilitate the creation of complex tools, such as an algebraic expression processor or a dynamic geometry engine. Visual Basic, another candidate, is not used by many research-based educational software developers and only runs on Windows, excluding a large percentage of the target audience in K-12 education. Squeak, a SmallTalk derivative, is catching on in education but as yet has few developers compared with Java. It is not clear that ESCOT could have done any better by choosing a different language than Java. The lack of an appropriate, high-quality platform for interactive component tools is a major obstacle to achieving the benefits of ESCOT’s approach.


99

Although we started this project with optimism, we have come to believe that the appropriate platform for education will not just ‘‘emerge’’ as industry continues to develop browsers. Further, the capabilities of the ‘‘public Internet’’ are increasingly driven by e-commerce. E-commerce does not require the kinds of complex interactive tools needed in education, and hence there are low incentives for enhancing the public Internet platform to meet educational needs. Until the platform problem is resolved, there is a limit to how successful any component-based project can be in providing educational solutions to a wide variety of schools. Another limitation of our work to date is that we have built integration teams with hand-selected master teachers and experienced programmers. Master teachers are busy people and in short supply. Likewise, the very best programmers tend to be attracted to application areas that are more profitable than education. Hence, there is a human-resource limit to how quickly ESCOT might scale. We are addressing this limit in a follow-on project called Training and Resources for Assembling Interactive Learning Systems (TRAILS, http://trails-project.org), in which we are experimenting with adapting our process to university students studying computer science and education.

CONCLUSION: MANAGING RECIPROCAL INFLUENCE In its summary of the state of the art in learning research, a noted panel of experts (Donovan et al., 1999) argued that much educational research fails because it works from a linear flow model of how research influences practice. A linear flow model (Fig. 4a) emphasizes orderly transitions of knowledge from scientific processes to implementation in classroom practices. The authors argue instead for a ‘‘reciprocity-of-influence’’ model. In this model, research and practice mutually condition the development of a cumulative knowledge base on teaching and learning. This mode (Fig. 4b) emphasizes a continuous, ongoing relationship of research and practice through many iterative cycles. In reviewing how and why ESCOT was able to meet its goals, we believe a slight variation of this distinction is useful for articulating a unifying lesson across the three factors already discussed. The variation places a cumulative software base of reusable components in the center alongside the cumulative knowledge base. In our view, this cumulative software base can best be managed, configured and used through processes that focus on coordinating reciprocal influences from R&D and classroom practice. To wit:

100


Fig. 4. (a) Linear flow model.

Fig. 4. (b) Reciprocity-of-influence model.


101

In Factor 1, we described how ESCOT selected the EPoW curriculet as its unit of production. One could characterize this choice as one that maximizes the opportunity for teaching practice and software development to interact frequently and iteratively. Curriculets are small enough that they can be built and deployed quickly; at the same time, they are small enough to be adopted and tested by teachers quickly. Further, the Problem of the Week environment optimizes quick learning from rapid field trials. In Factor 2, we described ESCOT’s strategy for managing software reuse and interoperability. Here we emphasized that ESCOT deliberately built its software component collection in light of a domain analysis of the needs in middle school mathematics education. Available software capabilities were reciprocally coordinated with needs evident in analyzing curriculum, standards, and use studies. Further, our product line approach sought the natural synergies of what was possible technically and what was desirable pedagogically. Most importantly, our emphasis on design patterns focused on developing a ‘‘knowledge base’’ of interoperability techniques that captured the technical challenges that emerge when pedagogical use is taken seriously. Hence, we actively managed the tools and knowledge in our core collection by paying constant attention to the reciprocal influence of technology and pedagogy. In Factor 3, we described how we structured integration teams. Here the attention to reciprocal influence is prominent: the fundamental purpose of the integration-team process was to mutually engage the wisdom of teaching practice and the knowledge of research-based educational technology development. Significant attention was paid to enabling the creation of an environment within teams in which mutual trust, respect, and understanding could thrive. Further, the process deliberately minimized the high-level design decisions made in advance of team formation; the summer workshop started with brainstorming that aimed to give teachers and developers true joint ownership of their planned curriculets. While most educational projects include teacher members, few make the leap of faith to allow distributed teams of teachers and developers to own their design problem. Our original ESCOT proposal focused strongly on the economic rationale for component software in education, building upon a long-standing component software vision (McIlroy, 1968) and the newer work of the Educational Object Economy to apply that vision in education (Roschelle et al., 1997). Ironically, at the end of the project, we find that the shakiest aspect of our vision is whether the economic conditions for educational component software will emerge. Given the high costs of coordination and the

102


tendency of funders to support projects and not cross-project interoperability work, it is not clear whether this kind of effort is economically sustainable. Further, it is far from clear whether supplying schools with a suitably powerful platform for use of component software is in the economic self-interest of those who could afford such an overhaul. Without such a platform, the costs of deploying component-based software to schools are insurmountable by relatively small economic entities such as the ESCOT testbed. Nonetheless, we see an emerging, equally powerful rationale for componentbased (or at least component-oriented) educational software development from the point of view of reciprocal influence of classroom practice and research. In this rationale, component-based software is a means of accumulating software building blocks that align with educational needs, while at the same time deferring fine-grained decisions. At one end of a spectrum of granularity of decision-making, ESCOT sought to collect general-purpose building blocks whose likely use can be foreseen by analyzing curricular needs. At the other end of the spectrum, ESCOT empowered small, distributed teams to make many fine-grained decisions about how technology can best be used to support a specific learning objective. Further, by structuring a unit of production such as the EPoW curriculet – which is amenable to rapid prototyping and testing in a realistic setting of use – ESCOT used component software to maximize the reciprocal influence that can be gained from rapid iteration. While purely economic or technical motivations may be insufficient to encourage researchbased educational software developers to adopt a component-oriented perspective, educational software developers and researchers should take notice of component software engineering techniques on the merits of their ability to foster reciprocal influence of classroom practice and research-based innovation in learning technology design. ACKNOWLEDGMENTS ESCOT was a very large, coordinated effort. We had seven major partners, and 34 organizations contributed software, volunteers, and/or contractors. Forty-four people made a substantial contribution. Nineteen teachers participated in our integration teams. Eight graduate students were supported at least 6 months each, and one student is writing a Ph.D. following directly from her ESCOT research. More than 50 external researchers attended at least one ESCOT-sponsored workshop. Thanks is due to each of them. We also thank our reviewers, Mimi Recker, Edward Gaible, and Shelley Goldman for helpful feedback. The work in this article was supported by the National Science Foundation (Award: REC9804930). The opinions presented are the authors and may not reflect those of the funding agency.


103

REFERENCES Becker, H., Ravitz, J.L., & Wong, Y. (1999). Teacher and teacher-directed student use of computers and software (3). Irvine, CA: Center for Research on Information Technology and Organizations, University of California. Birbilis, G., Koutlis, M., Kyrimis, K., Tsironis, G., & Vasilou, G. (2000, June 4–11, 2000). E-slate: A software architectural style for end-user programming. Paper presented at the Proceedings of the 22nd International Conference on Software Engineering, Limerick, Ireland. CEO Forum on Education & Technology. (2000). The power of digital learning: Integrating digital content (Year 3 Report). Washington DC: CEO Forum on Education and Technology. DiGiano, C., & Roschelle, J. (2000). Rapid-assembly componentware for education. Paper presented at the International Workshop on Advanced Learning Technologies, Palmerston North, New Zealand. DiGiano, C., Roschelle, J., & Chung, M. (2001). The need for a coordinated engineering discipline for the production of educational software. Paper presented at the 2001 International Conference on Advanced Learning Technologies, Madison, WI. DiGiano, C., Roschelle, J., & Chung, M. (in Preparation). The three year evolution of escot component interoperability. diSessa, A. (1997). Open toolsets: New ends and new means in learning mathematics and science with computers. Paper presented at the 21st Conference of the International Group for the Psychology of Mathematics Education 47–62, Lahti, Finland. diSessa, A. (2000). Changing minds: Computers, learning, and literacy. Cambridge, MA: MIT Press. DOD Software Reuse Initiative Program Management Office. (1995). Software reuse executive primer. Falls Church, VA: DOD Software Reuse Initiative Program Management Office. Donovan, M.S., Bransford, J.D., & Pellegrino, J. (Eds.). (1999). How people learn. Washington DC: National Academy Press. Feiler, J. (1996). Essential opendoc. Reading MA: Addison-Wesley. Fischer, G. (1987). Cognitive view of reuse and redesign. IEEE Software, 4(4), 60–72. Fisher, G., Redmiles, D., Williams, L., Puhr, G.J., Aoki, A., & Nakakoji, K. (1995). Beyond object-oriented technology: Where current approaches fall short. Human–Computer Interaction, 10(1), 79–119. Gamma, E., Helm, R., Johnson, R., & Vlissides, J. (1995). Design patterns. Reading, MA: Addison-Wesley. Goldenberg, P. (1995). Multiple representations: A vehicle for understanding understandings. In D.N. Perkins, J.L. Schwartz, M.M. West, & M.S. Wiske (Eds.), Software goes to school (pp. 155–171). New York, NY: Oxford University Press. Hopkins, J. (2000). Component primer. Communications of the ACM, 43(10), 27–30. IEEE Computer. (1997, October). Special issue on object oriented development and reuse. IEEE Computer. Jacobson, I., Griss, M., & Jonsson, P. (1997). Software reuse: Architecture, process, and organization for business success. New York: ACM Press. Kaput, J. (1992). Technology and mathematics education. In D. Grouws (Ed.), A handbook of research on mathematics teaching and learning (pp. 515–556). New York: Macmillan.

104


Kozma, R.B., Russell, J., Jones, T., Marx, N., & Davis, J. (1996). The use of multiple, linked representations to facilitate science understanding. In S. Vosniadou, E. De Corte, R. Glaser, & H. Mandl (Eds.), International perspectives on the design of technologysupported learning environments (pp. 41–60). Mahwah, New Jersey: Lawrence Erlbaum Associates. Laleuf, J.R., & Spalter, A.M. (2001). A component repository for learning objects: A progress report. Paper presented at the ACM/IEEE-CS Joint Conference on Digital Libraries, Roanoke, VA. McIlroy, M.D. (1968). Mass produced software components. In P. Naur & B. Randell (Eds.), Software engineering (pp. 138–155). Garmisch, Germany: NATO Scientific Affairs Division. Myers, W. (1997). Software reuse: Ostriches beware. IEEE Computer, 30(4), 119–120. National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: National Council of Teachers of Mathematics. Poulin, J.S. (1999). Reuse: Been there, done that. Communications of the ACM, 42(5), 98–100. Reiser, B.J., Spillane, J.P., Steinmuller, F., Sorsa, D., Carney, K., & Kyza, E. (2000). Investigating the mutual adaption process in teachers’ design of technology-infused curricula. Paper presented at the International Conference of the Learning Sciences, Ann Arbor, MI. Reiser, B.J., Spillane, J.P., Steinmuller, F., Sorsa, D., Carney, K., & Kyza, E. (2002). Investigating the mutual adaption process in teachers’ design of technology-infused curricula. Paper presented at the International Conference of the Learning Sciences, Ann Arbor, MI. Repenning, A., Ionannidou, A., Payton, M., Ye, W., & Roschelle, J. (2001). Using components for rapid distributed software deployment. IEEE Software, 18(2), 38–45. Rine, D.C. (1997). Supporting reuse with object technology. IEEE Computer, 30(10), 43–45. Roschelle, J., DiGiano, C., Chung, M., Repenning, A., Tager, S., & Treinen, M. (2000, December). Reusability and interoperability of tools for mathematics learning: Lessons from the escot project. Paper presented at the International ICSC Congress on Intelligent Systems and Applications: Symposium on Interactive and Collaborative Computing, Wollongong, Australia. Roschelle, J., DiGiano, C., Koutlis, M., Repenning, A., Jackiw, N., & Suthers, D. (1999). Developing educational software components. IEEE Computer, 32(9), 5–58. Roschelle, J., Hand, V., & DiGiano, C. (2001). Towards a coherent on-line collection of tools for math learning. Journal of Online Mathematics and its Applications, 1(2). Roschelle, J., Henderson, B., Spohrer, J., & Lilly, J. (1997). Banking on educational software: A wired economy unfolds. Technos, 6(4), 25–28. Roschelle, J., & Jackiw, N. (2000). Technology design as educational research: Interweaving imagination, inquiry & impact. In A. Kelly & R. Lesh (Eds.), Research design in mathematics & science education (pp. 777–797). Mahwah, NJ: Lawrence Erlbaum Associates. Roschelle, J., & Kaput, J. (1996). Educational software architecture and systemic impact: The promise of component software. Journal of Educational Computing Research, 14(3), 217–228. Roschelle, J., Kaput, J., Stroup, W., & Kahn, T. (1998). Scaleable integration of educational software: Exploring the promise of component architectures. Journal of Interactive Media in Education. Retrieved January 8, 2003, from the World Wide Web: http://wwwjime.open.ac.uk/98/6/


105

Shrader, G., Williams, K., Lachance-Whitcomb, J., Finn, L.-E., & Gomez, L. (2001). Participatory design of science curricula: The case for research for practice. Paper presented at the Annual Meeting of the American Educational Research Association, Seattle, WA. Shumar, W. (2003). The role of community and belonging in online learning: The ESCOT Project at The Math Forum. In M.A. Mardis (Ed.), Developing Digital Libraries for k-12 Education. Syracuse University, Syracuse, NY: ERIC Clearing house on Information and Technology, pp. 174–187. Spohrer, J., Sumner, T., & Shum, S.B. (1998). Educational authoring tools and the educational object economy: Introduction to this special issue from the east/west group. Journal for Interactive Media in Education, 98(10). www, jime.open.ac.uk/98/10 Standish Group International. (1994). The chaos report. Retrieved, from the World Wide Web: http://www.standishgroup.com/sample_research/chaos_1994_1.php Wattenberg, F. (1998). A national digital library for science, mathematics, engineering, and technology education.

106


APPENDIX 1: INTELLECTUAL PROPERTY IN THE ESCOT TESTBED In a planning phase prior to the ESCOT proposal, we met with developers in three invitational meetings to determine under what conditions they might be willing to contribute their components to a testbed. From those meetings, we learned that building trust in the developer community was a critical issue, and no matter what a license said, developers were unlikely to share precious source code to untrusted parties. Further, commercial developers were very cautious about releasing any intellectual property that might undermine their current business model. To address these concerns while meeting our goals, ESCOT developed a novel licensing structure. First, we defined ESCOT as a members-only testbed. Educational software developers could only gain access to the ESCOT code repository by signing a membership agreement (or contractually in the case of the formal project partners). Second, we defined the testbed as for ‘‘research and evaluation’’ purposes only. Making money from ESCOT code was prohibited, unless a further agreement was reached among the owners of the various components comprising an interactive curriculet. Further, large-scale distribution on CDROM or via a public Web site was prohibited. Third, we defined a pair of licenses under which code might exist in the testbed. The reasoning of the pair of licenses was as follows: On one hand, some developers wanted to restrict access to their prized code to the testbed; on the other hand, some developers were worried that if they built dependencies in their upon interoperability protocols in the testbed, they would be unable to later release a commercial product. Hence we adopted a ‘‘restricted’’ license for the code making up a component or tool, enabling the owner to retain control of uses outside the testbed. The complement to this license was the ‘‘open’’ license for the code making up protocols connecting components, enabling any testbed member to use the code without fear of being later restricted. (The full text of the volunteer agreement and the two licenses is available on our Web site, http://www.escot.org). This unique intellectual-property structure worked wonderfully for our purposes. In addition to making our anchor tenant partners comfortable, we were able to get a wide variety of commercial software donated to our project for free (due to the restriction to research and evaluation purposes), including a spreadsheet, a Web browser, and Logo interpreter.


107

We were not limited in our ability to carry out research or evaluation (which was our project goal). In fact, in the last year of ESCOT, when we decided to test our curriculets through a public Web site, we obtained our testbed members’ consent to a very simple technique for determining when a release of software was going beyond ‘‘evaluation’’ and becoming ‘‘large scale distribution.’’ We agreed to make the logs of access to the Web site available to any testbed member, and to allow them to pull their intellectual property from the testbed if they were concerned. In practice, no one utilized this privilege, because we only published specific curriculets and not the underlying tools, so the value of their property was preserved. In addition to enabling the congenial, joint production of curriculets, we believe the ESCOT IP structure was a catalyst for new relationships between testbed members. The IP agreements allowed members to learn details about other testbed members’ contributions that may otherwise have been proprietary. This often revealed powerful ways that each party could leverage the other’s work. As a consequence, at least three new business relationships formed among testbed members, some of which concern the production of commercial products.