Scenario-Based Methods for Evaluating Collaborative Systems

Computer Supported Cooperative Work (2009) 18:331–356 DOI 10.1007/s10606-009-9095-x

© Springer 2009

Scenario-Based Methods for Evaluating Collaborative Systems Steven R. Haynes1, Sandeep Purao1 & Amie L. Skattebo2 1 College of Information Sciences & Technology, The Pennsylvania State University, University Park, PA, USA (E-mail: [email protected]; E-mail: [email protected]); 2Department of Psychology, The Pennsylvania State University, University Park, PA, USA (E-mail: [email protected])

Abstract. Evaluating collaborative systems remains a significant challenge. Most evaluation methods approach the problem from one of two extremes: focused evaluation of specific system features, or broad ethnographic investigations of system use in context. In this paper, we develop and demonstrate a middle ground for evaluation: explicit reflections on scenarios of system use coupled with analysis of the consequences of these use scenarios, represented as claims. Extending prior work in scenario-based design and claims analysis, we develop a framework for a multiperspective, multi-level evaluation of collaborative systems called SWIMs: scenario walkthrough and inspection methods. This approach is centered on the capture, aggregation, and analysis of users’ reflections on system support for specific scenarios. We argue that this approach not only provides the contextual sensitivity and use centricity of ethnographic techniques, but also sufficient structure for method replication, which is common to more feature-based evaluation techniques. We demonstrate with an extensive field study how SWIMs can be used for summative assessment of system performance and organizational contributions as well as formative assessment to guide system and feature re-design. Results from the field study provide preliminary indications of the method’s effectiveness and suggest directions for future research. Key words: evaluation methods, collaborative systems, scenario-based approaches, claims analysis, field studies

1. Introduction Effective evaluation of collaborative systems remains an elusive goal for researchers and practitioners in computer-supported cooperative work (Ackerman 2000; Grudin 1994). Unlike a manufacturing control system, for example, where tangible contributions can be measured by objective metrics such as reduced cost, increased output, or reduced defects, measuring the contribution from collaborative systems requires an understanding of how dynamic work practices are improved through system use. This requires drawing out and reflecting on user experiences and perceptions that have an inherently tacit dimension. What to evaluate, how, and why, are all questions still in need of further investigation.

332

Steven R. Haynes et al.

Much of the prior research on evaluation of collaborative systems clusters around two general approaches: those characterized by assessments of specific system features, and those characterized by ethnographic, context-sensitive techniques (John 1996; Nielsen and Mack 1994). Neither approach can adequately account for the complex, multi-level consequences of collaborative systems, nor do they reliably provide the formative input (Scriven 1996) necessary for subsequent design and re-design cycles. Further, results produced by these approaches generally do not connect contributions from a collaborative system to organizational goals and priorities (Rogers and Bellotti 1997; Seddon et al. 2002). The objective of this paper, therefore, is to argue for and develop a method in the middle ground: a scenario-based evaluation framework attuned to eliciting system contributions. The method is derived from and extends the theoretical foundations provided by scenario-based design and claims analysis (Carroll 2000; Carroll and Rosson 1992). In this paper, we describe SWIMs (Scenario Walkthrough and Inspection Methods), an evaluation approach that elicits and captures the consequences of interactions between users and a collaborative system within specific scenarios, and that uses techniques to aggregate and analyze user claims across organizational hierarchies and functional units. Further, the method helps make explicit how micro-features of collaborative systems relate to evaluative claims for generating formative as well as summative evaluation results. In the following section, we review some existing approaches to evaluation, including a review and argument for the appropriateness of scenario-based approaches for evaluating collaborative systems. Section 3 describes the SWIMs approach, built on the theoretical foundations provided by scenarios and claims analysis. A real-world application of SWIMs for evaluating a Marine Corps Integrated Digital Environment (IDE), is reported in Section 4. This instantiation of our approach provides the necessary details on operationalizing the method to support its replication by future researchers and practitioners. The results from this case study, reported in Section 5, demonstrate how SWIMs spans multiple levels and units of analysis to capture contextually-sensitive, user-centric perspectives. We conclude, in Section 6, with reflections on the effectiveness of SWIMs and implications for future research. 2. Prior work on the evaluation of collaborative systems Collaborative systems present unique evaluation challenges because (a) they represent feature-rich environments, (b) their use is distributed over space and time, and (c) their contributions are perceived differently by people with varying backgrounds, goals, and priorities (Grudin 1988). The range of approaches demonstrated in prior work on evaluation can be grouped into two broad categories. The first involves focused analysis of specific features or capabilities of a collaborative system using either experts (e.g., designers, cognitive

Scenario-Based Methods

333

psychologists, usability engineers, etc.) or users as informants (Steves et al. 2001). These methods frequently involve sensitizing experts with criteria or heuristics (Nielsen 1994). For example, the groupware walkthrough method (Pinelle and Gutwin 2002), an adaptation of cognitive walkthroughs (Polson et al. 1992), uses scenarios decomposed into tasks and sub-tasks as the basis for analyzing collaborative affordances. Such approaches are sometimes favored because they are inexpensive relative to laboratory or field methods (Drury 1999), but are limited because they do not account for the context that characterizes collaborative system use in situ. (Damianos et al. 1999). Other feature-based evaluation approaches include questionnaires and experiments. These have shown to be effective for identifying and measuring specific system capabilities (Blanford and Rugg 2002; Orlikowski and Gash 1994; Steves et al. 2001; Twidale et al. 1994), but have been criticized for their lack of ecological validity (Sokolov 1999) and generalizability across varied domains of use (Gray and Salzman 1998). The second broad category of approaches to collaborative system evaluation involves field studies of deployed systems. These techniques include breakdown analysis, which involves the collection and investigation of group interactions and transcripts that highlight critical, generally negative incidents (Ramage 1999). While ethnographic and other field methods such as these enable capturing information about the context of use, they typically require an extended observation and analysis time frame (Grudin 1994), and their results may still not provide for a comprehensive evaluation. Conducting analysis within a specific theoretical framework can help focus evaluation activities (Hughes et al. 1994) but may also bias results towards that particular frame of reference. In addition, results from field studies are often inconclusive and difficult to translate into specific prescriptions because they identify problems, but do not directly point to potential solutions or prescriptions for design. Although some methods have been proposed to integrate evaluation activities with design (Dewan 2001), few reports from the field have demonstrated how different evaluation techniques can be effective as part of a formative assessment. One suggestion from prior work is combining multiple techniques to leverage their individual strengths (Ross et al. 1995). One such proposal includes prototype development coupled with field observations and focus groups to determine users’ needs, followed by post-deployment surveys to identify socalled ‘hot spots’ in a given implementation (Sokolov 1999). Another proposal recommends that any evaluation include ‘illuminative’ descriptions from the viewpoints of different user roles including line workers, managers, and customers (Blythin et al. 1997). The different perspectives gained from multimethod evaluation may be better equipped to address the inherent complexity of collaborative systems. A significant gap in many published accounts of collaborative system evaluation is the lack of detail needed to replicate and adapt their use both in

334


the research context and in real-world settings. We found that many evaluation methods were described with a sound conceptual basis, but without the procedural details needed to understand how activities and measures are employed in the field. In many cases this lack of detail is a function of article and conference paper length restrictions, in others we suspect it is a consequence of not having been used outside the laboratory. We believe sufficient details of how to operationalize a method are required to support progressive evaluation research and evaluation as practiced. We argue that scenario-based techniques (Carroll 2000; Carroll and Rosson 1992) may provide for collaborative systems evaluation research with both a sound theoretical foundation and operationalizable specificity. Scenariobased techniques focus on using concrete and grounded units of analysis (scenarios) as opportunities for explicit reflection on users’ experiences with their systems. Scenario-based techniques account for multiple user roles as well as a range of contextual influences. Scenarios include narratives that describe details of a user’s interaction with a system, an identified actor represented as a prototypical role, a setting describing the context in which the scenario takes place, and the goals motivating the scenario. A scenario specification provides a self-contained package linking requirements to one or more aspects of an evolving design (Bardram 2000). Scenarios were first proposed for understanding and managing the complexity of nuclear strategy and deterrence (Kahn 1962; Kahn 1967), where they helped facilitate reflection on this rapidly changing problem space at multiple levels of analysis. Scenario-based approaches have since been applied to the design of information systems, where they bridge the gulf between the abstractness of engineering rules and the inescapable situatedness of work (Carroll 2000; Carroll and Rosson 1992). They have been shown to be effective in a variety of applications including: human-computer interaction (Carroll 1995; Carroll et al. 1993; Clements et al. 2001), expert evaluation (Mukkamalla et al. 2002), discount usability evaluation (Blandford 2002; Blanford and Rugg 2002; Nielsen 1995), diary studies (Nielsen 1995), visualization (Winckler et al. 2004), software architecture evaluation (De Bruin and Van Vliet 2001; Ionita et al. 2002; Pang et al. 2005), evaluation of a document access control system (Stiemerling and Cremers 1998), comparison of specific behaviors of configuration management systems (Ge and Whitehead 2006), as well as training following from evaluation (Bernstein and Vorburger 2006). One study used scenarios in controlled settings for evaluating specific collaborative system features (Damianos et al. 2000; Damianos et al. 1999). As the scope of these prior applications suggest, a key advantage of scenarios is their scalability and flexibility to account for work practices distributed over space, people, and time. Another advantage is their ability to account for context (Carroll 2000; Stiemerling and Cremers 1998; Sutcliffe 2002); that is, scenariobased evaluation can focus not only on particular features of the system, but also


335

on the contextual interactions between the user, task, system features, and working environment. At the same time, scenarios provide specific focal points for evaluation, and therefore have the potential for reducing the effort and expense associated with ethnographic and other more immersive methods. Finally, scenarios provide a common, tractable language for designers, users, and evaluators (Carroll 2000), which is necessary to support both formative and summative evaluation. Drawing on these observations, we develop a scenario-based approach for the evaluation of collaborative systems. We describe the approach as a flexible framework that can be applied in a variety of collaborative system evaluation contexts. The specific methods presented in the next section and its application as described in Section 4 are aimed at supplying sufficient detail to allow both practitioners and evaluation researchers to adopt and refine the approach.

3. Swims: a scenario-based evaluation approach We propose the use of scenario walkthrough and inspection methods (SWIMs) as a way to use scenarios to elicit specific reflections on users’ experiences with collaborative systems. The SWIMs method makes use of claims analysis to help understand the consequences of these experiences. Aggregating the consequences of system use across scenarios and user roles allows SWIMs to characterize technical, psychological, organizational, social, and economic outcomes from system use and to identify potential system design enhancements. The premise underlying SWIMs is that scenarios both help elicit reflections on users’ actual and envisioned interactions with a system, and help elicit the benefits, costs, and risks that accrue from use of a system in relation to a particular goal in a particular context. The SWIMs approach includes four key stages: 1. Building a priori contextual appreciation 2. Eliciting current and envisioned scenarios from individual users 3. Validating scenarios with individuals and groups 4. Analyzing the consequences of scenarios in context with claims analysis Each of these four key stages is described in the sections that follow.

3.1. Stage 1: building a priori contextual appreciation Because SWIMs focuses on interactions between the user and the system in specific contexts of use, it is important for evaluators to form an a priori appreciation and understanding of this organizational context before entering the field. This a priori understanding includes knowledge of users, their tasks, and task goals, and the system capabilities designed to support these tasks. This contextual understanding is important because it allows evaluators to actively interpret what they hear from system users and other informants during scenario

336


interviews, and it supports formulating good probe and follow-up questions in response to what these interviewees say. Information about user roles, the tasks and goals of different user groups, and specific system capabilities may be collected through discussions with organizational management, system developers and integrators, and other stakeholders, as well as by reviewing documentation such as organizational charts, desk procedures, and system manuals. Understanding the perspectives of different user groups and roles is also important to help ensure comprehensive and representative scenario and claims sampling and validation in later stages of SWIMs evaluation. This information should be captured in a way that enables later analysis and validation of a scenario catalog’s completeness and representativeness, and should articulate and link scenario components such as task goals and user roles to help guide participant sampling. 3.2. Stage 2: scenario elicitation Scenario elicitation involves prompting user reflections on key questions about the capabilities provided by a collaborative system, the organizational contributions derived from these capabilities, and identifying and overcoming barriers to increasing these contributions. SWIMs operationalizes scenario elicitation with semi-structured interviews because they both provide the structure necessary to reduce drift and ambiguity in informant discussions while continuing to offer the flexibility required to capture contextual detail. Semi-structured interviews also afford some reliability across researchers by providing a common format and protocol to guide interactions with study participants (Campion et al. 1988). Although it is possible to collect scenarios using other techniques such as focus groups (Stiemerling and Cremers 1998) or system design documents (Bass et al. 1998), we argue for the use of semi-structured interviews because they can focus on individual interactions with the system (use centricity) while allowing interviewers to probe for details regarding why system support for a scenario is either positive or negative, as well as what might be done to enhance this support (contextual sensitivity). Further, individual, semi-structured interviews are not subject to social influence factors that might bias results gathered in group settings (Podsakoff et al. 2003). The general form of an interview guide to elicit scenarios and claims appears in Table 1 below. Making effective use of a semi-structured interview guide such as this draws heavily on the a priori contextual appreciation that evaluators gain in the first stage of the method. This contextual knowledge helps evaluators probe, expand, and clarify answers and comments given by participants in the interview process, and also helps develop a sampling strategy for identifying key participants. SWIMs suggests a stratified sampling approach that includes multiple participants from each identified stakeholder group (Oppenheim 1992). It is important to include not only direct users of the system under evaluation, but also consumers


337

Table 1. General form of a semi-structured interview guide to elicit scenarios and claims. Questions related to current use scenarios a) Describe a common system use scenario, including the context within which this takes place i) What is your role? ii) What is the goal of this scenario (task goal)? iii) What features of the system are used during the scenario? iv) What are the benefits, costs, or risks of system use for accomplishing the goal of this scenario? v) What information management needs of the scenario are not supported by the system? vi) How does the system support collaboration across different roles during the scenario? vii) How could the scenario be better supported by the system and/or organization? viii) How does this scenario contribute to the organization’s mission? (iterate as time allows) Questions related to prospective or envisioned use scenarios b) Describe a scenario that you would like the system to support in the future i) What information needs are not currently being met by the system? ii) How could these unmet information needs be addressed? (iterate as time allows)

of the information the system provides, such as management and stakeholders external to the organization. The stratified sampling strategy not only ensures that the sample is representative of users from different organizational levels and functions, but also leads to coverage of a broad range of system uses and features. Similar to the strategy employed in iterative grounded theory development (Glaser and Strauss 1967), SWIMs suggests interleaving the collection of scenarios and claims with a taxonomy-guided analysis of the data gathered. This may require coding and analyzing textual data to extract concepts, such as roles, tasks, goals, features, and claims. To help ensure that information extracted in this way accurately reflects users’ own understanding of their activities and priorities, the next stage of SWIMs is scenario validation. 3.3. Stage 3: scenario validation Scenario validation is important because initial analysis of data collected through interviews imposes a structure that may be overly influenced by evaluators’ a priori understanding of the system and its use context. An explicit validation stage allows evaluators to member-check with users to ensure that the evolving scenario catalog and claims analysis capture users’ own understanding of the problem space, and, importantly, that this characterization is reasonably comprehensive. For the SWIMs approach, we suggest using focus groups to operationalize this stage of scenario validation. During focus group sessions, an evolving catalog of scenarios representative of system interactions and related activities (Stiemerling and Cremers 1998) is presented for group discussion and

338


critique. Other techniques, such as triangulation on single scenarios across multiple informants (Yin 2003) may also be used, as this can help elicit ways in which scenarios interact to reflect collaborative work practices and organizational processes. For instance, the short-term task goal of one scenario may precede the accomplishment of other scenarios and goals, some of which may be performed by people using different features of the system. SWIMs adapts a scenario value chains model (Chewar et al. 2005) to conceptually link otherwise temporally and spatially decoupled scenarios, viewing some as upstream and some as downstream relative to overall information flow. Upstream scenarios often require investment of effort by users, whereas users in downstream scenarios often reap the benefits of these efforts. The goal of the validation stage is a catalog of scenarios that represents the consensus of the user and stakeholder community. A validated and substantially complete scenario catalog is a prerequisite for the last stage of SWIMs: analysis of claims. 3.4. Stage 4: claims analysis Claims reflect the consequences of scenarios, the results of interactions between users, system features, and task goals, or the likely consequences of such interactions in the case of envisioned scenarios (Carroll 2000; Carroll and Rosson 1992). Claims capture what is ‘right’ (positive claims) and what is ‘wrong’ (negative claims) with respect to system support for a given scenario. Claims analysis, therefore, involves aggregating and interpreting claims. Using SWIMs involves creating claims taxonomies to fulfill the evaluation objectives of assessing contributions accrued from the system implementation and providing formative assessments to help design and re-design the collaborative system. Because many organizational contributions are not directly accessible for measurement, and because the catalog of scenarios covers both current and envisioned scenarios, SWIMs distinguishes four categories of contributions: (a) measurable, (b) tangible, (c) intangible, and (d) unrealized. Figure 1 below shows how earlier stages of SWIMs support the claims analysis stage and the development of the claims taxonomy. The first category, measurable contributions, represents user perceptions of an organizational contribution that is concrete, for example, dollars saved per month due to system support for a specific scenario and its tasks. The second category, tangible contributions, reflects user perceptions of a specific system benefit that can be easily identified and agreed upon (if not easily measured) for example, supporting better decision making. The third category, intangible contributions, refers to users’ perceptions of a benefit that cannot be easily tied to one or more organizational goals, for example, access to historical documents. The final category, unrealized contributions, is important to formative evaluation because it identifies potential contributions identified by users that have not been realized because of some obstacle, for example, the lack of a specified system feature to


339

Figure 1. The claims taxonomy in SWIMs.

support an important organizational task. Next we discuss the first three of these claims types, specifically those that assess system contributions, before moving on to the fourth and final category of formative claims. 3.4.1. Claims analysis for assessing contributions Development of SWIMs was explicitly motivated by the need to link outcomes from system use to specific system capabilities and features (Grudin 2004). SWIMs accomplishes this by tracing evaluative claims to system features via scenarios (as shown in Figure 1 above) and using a claims taxonomy to focus on how the collaborative system supports higher-level organizational goals (Rogers and Bellotti 1997). The specific claims analysis techniques included in SWIMs, therefore, include: (a) aggregating claims for a scenario across multiple users, (b) comparing and contrasting claims from different user roles for a single scenario, and (c) analyzing claims across scenarios linked through information dependencies. The first technique aggregates micro-level claims from individual users to derive conclusions about how system support for a scenario results in contributions to an organization. This method of claims analysis attempts to deliver on a key premise motivating SWIMs: the linking of specific system features and capabilities to claims regarding the consequences of system use. Specifically, aggregation allows bridging from the psychological perceptions articulated by individual users as claims to organizational level outcomes

340


specified in terms of tangible, measurable contributions tied to organizational goals. The second technique, comparing and contrasting claims from different users, ensures that positive as well as negative claims from multiple user roles are appropriately juxtaposed as part of the assessment. With this technique, multilevel evaluation is extended to address the second SWIMs objective: multiperspective evaluation. Instead of a contrived or general evaluation from multiple viewpoints, this analysis technique supports the explicit and specific recognition of different perspectives from different roles through claims articulated by users in each role. It ensures that the evaluation results for each scenario are characterized in terms of the differential value a system may deliver to multiple user roles with their different goals and priorities. The third technique, analyzing claims across scenarios, leverages implicit and explicit linkages, captured as scenario value chains, for further analysis and higher-order interpretation. For example, a particular scenario, analyzed in isolation using the first two techniques, may not appear to contribute to organizational goals. An upstream scenario in the scenario value chain (Chewar et al. 2005) may, however, be critical to supporting other downstream scenarios that do make a positive contribution. Without explicitly linking claims from related scenarios, users of upstream scenarios may not recognize or appreciate how their work with a system contributes to downstream scenarios that are conducted asynchronously. This technique thus recognizes that evaluators are likely to find positive contribution claims (e.g., measurable, tangible, intangible claims) associated with what we call value out scenarios, and negative claims associated with value in scenarios. This claims analysis technique therefore reveals linkages across multiple scenarios, delivering on the goal of identifying system use consequences and evaluation outcomes that are distributed across space and time. The discussion so far has relied on elaborating the rationale underlying the three techniques. A number of further analytical possibilities can be identified for each depending on the specific characteristics of the scenario and claims data gathered, such as the number of users, the level of claims specificity and detail, and any missing or sparsely populated claims sets for particular scenarios. The techniques as described above, however, still do not achieve the goal of eliciting and identifying formative evaluation results to guide action, such as system redesign. The next section describes how this evaluation objective is addressed. 3.4.2. Claims analysis for formative evaluation Formative evaluation involves treating claims analysis as a method for identifying opportunities to enhance system capabilities in future design cycles. SWIMs addresses formative evaluation by focusing on unrealized contributions as classified in the claims taxonomy developed earlier (see Figure 1). This


341

analysis technique suggests how a collaborative system can be improved to increase its contribution to organizational goals. Specifically, we suggest that investigating the causes of unrealized contributions provides a powerful means for generating design prescriptions. These causes–SWIMs calls these obstacles to realizing positive claims–can be categorized as: (a) specific feature enhancements that can help realize contributions (e.g. ability to annotate online documents), (b) general concerns about the implemented system that should be addressed before the benefits can be realized (e.g. speed of response concerns), (c) specific organizational concerns that impede the realization of benefits (e.g. lack of organizational incentives to use the system for a task) , or (d) more general concerns related to collaborative work (e.g. face-to-face conversations across a cubicle wall may be favored over the chat session capability). The first two categories of obstacles provide evaluation outcomes to suggest feature enhancements and to inform system re-design. The third category identifies factors in the organizational environment that may hinder the realization of system contributions. The final category includes remaining factors, often social factors, identified as obstacles to successful implementation. A second technique used by SWIMs for formative evaluation involves eliciting and analyzing envisioned scenarios. Envisioned scenarios represent current or future work activities that are not yet supported by the collaborative system but are suggested by evaluation participants as a means to increase the contribution of a system to an organization’s goals (MacLean and McKerlie 1995). Envisioned scenarios are useful for orienting and guiding new design discussions because they represent concrete needs tied to specific goals and activities, which are in turn linked to new requirements and corresponding features that may be implemented in future design cycles. They provide thumbnail design specifications for future enhancements to the system that are grounded in users’ actual work contexts. We believe that the SWIMs approach overcomes a number of concerns identified in prior work on evaluating collaborative systems including: linking concrete system features to more abstract organizational goals, accounting for the multiple perspectives inherent in the evaluation of collaborative systems, and bridging multiple levels of analysis. Employing user reflections as the primary inputs to the evaluation process, and drawing on the theoretical foundation of scenario-based design, the claims taxonomies proposed for use in SWIMs support identifying, aggregating, and analyzing the consequences of collaborative systems use. In the next section we describe use of the SWIMs method to evaluate a working instance of a complex and large-scale collaborative system. 4. Applying swims in the field We used the SWIMs approach for a field evaluation of a complex and large-scale collaborative system, the Integrated Digital Environment or IDE, implemented

342


and used at the offices of the United States Marine Corps Program Manager for Light Armored Vehicles (PM LAV). In this section, we describe the evaluation and demonstrate some of the techniques we used for operationalizing SWIMs. 4.1. Applying stage 1: building a priori contextual appreciation Our goal was to evaluate the IDE in terms of its support for product lifecycle management for the LAV fleet. The evaluation began by meeting with members of the PM LAV organization to understand the various stakeholder groups and the core tasks of the organization’s different functional units. We learned that the PM LAV has responsibility for procurement and ongoing lifecycle management for a fleet of about 800 vehicles in a variety of configurations deployed worldwide and tasked with a range of different missions. Product lifecycle management involves monitoring vehicle health and field performance, analyzing opportunities to enhance vehicle performance, overseeing the vehicle maintenance process, and designing and implementing engineering upgrades. About 70 engineers, logisticians, scientists, active duty Marines, and other staff use the IDE for email, videoconferencing, workflow, scheduling, document management, project management, and specific functions such as vehicle readiness reporting related to the organization’s core tasks. To better understand both the context of the PM LAV domain and the kinds of technologies available to support its work, we carried out a feature assessment/ comparison and gap-fit analysis, which provided pre-empirical knowledge to help guide data collection in the field. The results of this initial analysis also provided significant material for the development of SWIMs because it showed how a feature-based assessment failed to link the capabilities of the system to organizational goals and priorities. We engaged in other activities to understand the IDE and its operating context, including the analysis of organizational documents such as early design specifications and user manuals. We also obtained access to the system ourselves and used it to better understand the IDE’s functionality, features, user interface, and how these performed in the context of common tasks. 4.2. Applying stage 2: scenario elicitation To elicit scenarios and claims from IDE users, we designed an interview template based on the example presented in Table 1 above. Using the interview guide, we conducted semi-structured interviews with 26 PM LAV personnel during two site visits. To ensure that scenarios reflected multiple perspectives, interview participants were recruited from various roles and positions throughout the organization, including scientists, engineers, logisticians, and active-duty Marines. All participants had significant experience working in the PM LAV office,


343

and all had worked directly with the IDE. The interviews were conducted with one participant and one interviewer to reduce potential confirmation bias (Asche 1955). Two research assistants coded the interview transcripts. They were largely blind to the purpose of the study but were provided the SWIMs framework as a coding guide (see Figure 1). The Atlas/ti software was used for coding. The example in Figure 2 below illustrates the data coding process. In spite of our efforts to focus the interviews, we found that complete articulation of a scenario and associated claims (such as that illustrated in Figure 2) by the interviewee was less common than discussion that moved between scenario fragments. For example, an interviewee might mention using the IDE’s Document Management System (DMS) early in the interview, move on to discussing features in other parts of the system, and then return to discuss benefits of the DMS later in the interview. These experiences point to the inherent connectedness of scenarios and capabilities in the collaborative systems context, raising difficulties for analysis, but also motivating new claims analysis techniques for managing this complexity. Having learned the users’ vocabulary in the contextual appreciation stage (Stage 1) of SWIMs also proved to be valuable. Given the proclivity for using acronyms in the PM LAV office (also common in other defense and government domains), understanding the

I’m a [Technician*]…

Role: Technician

I use the Portal for … my leave request… [N]ow instead of filling out one sheet of paper and it’s a leave request and giving it to whoever’s authorized to grant me leave and have him sign it, now I log onto my Portal, type in my password a couple times, [open a workflow], click on a bunch of buttons, have it open up the electronic file, drag and drop all the right names for a leave request, send it out. It goes out, comes back approved… [and] will populate somebody’s calendar somewhere. So it shows me as being gone during that time period. I still have to type that same sheet of paper and chase it with hard copies. So now I’ve doubled my effort and time for something electronically that’s supposed to streamline a process.

Task Goal: To receive approval for leave request

* Modified to protect the participant’s identity

Figure 2. An example of coding scenarios and claims.

Feature: Workflow

Positive Claim: Tracks employee whereabouts Negative Claim: Inefficient process

344


vocabulary proved invaluable in our attempts to make the best use of our time with field informants. The 26 interviews yielded approximately 300 pages of transcribed text. Coding and analysis of this data identified 43 unique scenarios (27 actual and 16 envisioned), and 464 claims, with a range of 0 to 22 claims per scenario. On average, a scenario was identified and discussed by four interviewees (minimum 1, maximum 20), suggesting there was a good deal of agreement across interviewees on common interactions with the system. An assessment of intercoder reliability conducted on a subset of six transcripts suggested moderate to high levels of agreement (Cohen 1960) between the coders, ranging from 84% to 100% agreement for different elements in the SWIMs framework. Instances in which coders did not initially agree were negotiated to achieve consensus prior to the claims analysis.

4.3. Applying stage 3: scenario validation The evaluation team later visited the field study site to conduct three focus group sessions to validate the scenario catalog and initial claims taxonomies derived from the initial interviews. Each focus group included study participants and representatives from different levels of the organization across different functional units. The catalog was presented to the focus group in the form of a simple list of scenarios, each connected to several claims that could be accessed using a web browser. The evaluation team seeded discussion at the focus group meeting by asking the group to indicate whether any scenarios were misrepresented or missing from the catalog. Based on their reactions, participants were probed to identify new claims about the organizational contributions gained from IDE support for identified scenarios. Overall, the focus groups resulted in only minor changes to the scenario catalog. The scenarios and claims gathered during stage 2 were validated through these focus groups. The final stage of in-depth claims analysis started with a catalog of 43 scenarios and more than 200 related claims, with an average of slightly over five claims per scenario. 4.4. Applying stage 4: claims analysis As a precursor to analyzing the claims with the taxonomies and techniques suggested by SWIMs, the coded claims were compared against the SWIMs claims analysis taxonomy (see Figure 1). One outcome from this analysis was that the scenarios identified by the interviewees were found to relate to only 19 of the 74 discrete features available in the IDE. Without the focus group validation from SWIMs Stage 3, this result may have been interpreted as indicating a potentially incomplete scenario elicitation. Given that we believed the scenario catalog to be validated, this result instead pointed to potential IDE feature bloat,


345

the inclusion of features that did not support common activities of the implementing organization (McGrenere 2000). 4.4.1. Applying claims analysis for assessing IDE contributions To assess IDE contributions, coded claims were categorized using the three categories of (a) measurable, (b) tangible, and (c) intangible contributions. Table 2 shows a distribution of the 212 claims across the three categories. As expected, the results show that measurable and tangible contributions are difficult to identify by users of complex collaborative systems. This occurred despite extensive use of probe questions during scenario elicitation designed to focus on measurable benefits, and despite encouraging participants during scenario validation to articulate claims with specific examples of measurable and tangible claims. Further analysis of the claims based on users’ roles suggests other interesting aspects of the claims. Table 3 below, for example, shows the distribution of claims types across user roles. The claims in Table 3 indicate that personnel in managerial roles identified a significantly higher proportion of positive claims. This result suggests that managers may accrue more benefits from the system than users in other roles, or that managers are simply more aware or better able to articulate the benefits of IDE support for collaborative work. We conducted an additional analysis using scenario value chains to identify value-in and value-out scenarios. The analysis revealed that users in different roles tended to participate disproportionately in these two kinds of scenarios. As expected, positive claims were more often associated with value-out scenarios, and negative claims were more often associated with value-in scenarios, as shown in Figure 3 below. The analysis showed that technical and administrative personnel were more likely to participate in value-in scenarios, which provided one explanation for the relatively low number of positive contributions reported by users in these roles compared to those reported by users in management roles. In contrast, the number of positive claims for value-out scenarios was greater, even when accounting for negative claims from individuals in managerial and administrative roles. We found representations such as in Figure 3 to be useful for (a) analyzing and communicating the occurrence of different types of claims for different roles and Table 2. Analysis of claims following the claims taxonomy (n=26). Claim Type

Frequency

%

Measurable Contributions Tangible Contributions Intangible Contributions Total Contribution Claims

11 54 147 212

6 26 68 100

346


Table 3. Analysis of contribution claims following the claims taxonomy and user roles (n=26).

Managerial Personnel (n=9) Technical Personnel (n=14) Administrative Personnel (n=3)

Measurable Contributions

Tangible Benefits

Intangible Benefits

4 6 1

23 27 4

75 60 12

(b) illustrating the disparity between different role types and the value they perceive from system use. The three examples given in this section represent the three techniques suggested by SWIMs. The first (Table 2) demonstrates how claims may be aggregated to bridge different levels of analysis in evaluation. The second (Table 3) demonstrates how multiple perspectives can be accommodated in a single evaluation effort. The third (Figure 3) shows how a scenario value chain can be used to reveal some of the social dynamics that attend collaborative systems. We constructed a number of additional representations of the claims data and were impressed with how useful and flexible the claims data were in helping to understand the IDE and its role in the organization.

Value In Scenarios

Value Out Scenarios

Figure 3. An example scenario value chain and associated claims.


347

We used these analyses and representations to formulate aggregate statements about the contribution of the IDE to organizational outcomes, and how these contributions are realized as the IDE supports work practices separated across space, time, and multiple user roles. 4.4.2. Applying claims analysis for formative evaluation To inform further cycles of design and re-design, the unrealized claims category was analyzed following the taxonomy suggested by SWIMs. Table 4 shows an example of the summaries created from analyzing these claims, along with a more detailed analysis of claims related to specific feature enhancements. The evaluation team constructed summaries of unrealized claims with associated features and user roles, as well as the number of occurrences for each claim, such as the ones shown in Table 5. The summaries allowed PM LAV management to identify areas where feature enhancements or other interventions were most needed so that constrained resources could be focused on the most important future design efforts and organizational interventions. Summaries of envisioned scenarios were also constructed, following the second formative evaluation technique suggested by SWIMs. These summaries include the number of participants who described the envisioned scenario (frequency), and the number of positive claims associated with each, as illustrated in Table 6 below. Analyses such as these can be readily used to guide specific enhancements to the existing system. For example, it is clear that the scenario assess vehicle readiness levels was perceived to be the most important (i.e., receiving the highest frequency of claims) as well as potentially most valuable (i.e., connected to a large number of positive claims). With the claims data available, it is also possible to identify the user roles most interested in support for envisioned scenarios so that they can be recruited as participants in future design cycles. Table 4. Example of categorizing unrealized contributions claims. Claim Type

An Example Claim

Specific Feature Enhancement to the IDE General Concerns about the IDE

The workflow feature in the IDE should provide error messages The need for multiple authentications when logging on and using different capabilities in the IDE Many users do not used the IDE to its full potential because management does not promote or mandate IDE use The PM LAV personnel enjoy and prefer to communicate face to face instead of via electronic communication

Organizational Concerns at the PM LAV related to IDE General Concerns Related to Collaborative Work

348


Table 5. Frequencies of unrealized contributions claims related to feature enhancements (n=26). Feature Enhancement Type

Managerial Personnel (n=9) Technical Personnel (n=14) Administrative Personnel (n=3) Total

Communication

Process support

Knowledge management

8 13 0 21

38 26 10 74

11 15 5 31

Both of the formative evaluation techniques suggested by SWIMs provide useful guidance for improving future versions of the IDE. The first, summarized in Tables 4 and 5, shows how causes of unrealized contributions can be elicited as claims, analyzed in aggregate form, and used as the basis for re-design and other interventions. The second, summarized in Table 6, highlights concrete user requirements tied to specific organizational goals and activities. These, in turn, are expressly linked by scenarios to thumbnail design specifications grounded in the work context. This field application of SWIMs to evaluate the IDE provides a first demonstration of the method’s applicability and effectiveness. It also provides the opportunity to reflect on our experiences using SWIMs. In the next section we share some of these reflections.

Table 6. An example summary of envisioned scenarios (n=26). Envisioned Scenario

Frequency

Positive Claims

Assess vehicle readiness levels Hold on-line meeting Up-load LAV data from field operations Track time & attendance Process travel vouchers Check inventory for parts available Identify work processes Access readiness data Auto-reminder for meetings/ task deadlines Track procurement status Track contract work problems Access contract history Make contract modifications Solicit contracts from vendors Access documents remotely Collaborate with partners outside PMLAV Diagnose field equipment Edit files & ECPs

7 4 4 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1

5 5 2

4 1 4 3

1


349

5. Discussion Results from the field study demonstrate some of the benefits of SWIMs and also reveal some identifiable limitations and concerns. The SWIMs approach to evaluation of complex, collaborative systems is intended to sensitize and focus evaluators on the uses and context of these systems. Towards this end, SWIMs draws on the theoretical foundations provided by scenario-based design, uses scenarios and claims as core units of analysis, and helps identify organizational contributions accrued from collaborative systems adoption and use. As the field study illustrates, the scenario and claims analysis techniques we propose as part of SWIMs are inherently multi-level and multi-perspective. The SWIMs approach also helps to identify obstacles to realizing contributions, both technical and social/organizational, which in turn supports formative evaluation. That is, it provides understanding of not only how well a given system is performing, but also of how its support for organizational goals and priorities can be improved through re-design and other interventions. The structure of the method also facilitates replication, thereby leveraging relatively expensive and time-consuming field evaluation activities for subsequent evaluations.

5.1. Assessing the appropriateness of use centricity in evaluation The field study showed that the use of scenarios and claims as a focusing mechanism added use-centricity to the evaluation process. By asking users to limit their discussion of a system to specific scenarios of use, we were able to obtain evaluation data grounded in what users actually do, in their day-to-day experiences with the system. Although this focus on scenarios helped to produce evaluation outcomes that may not have been identified using a feature-focused evaluation technique, the results did not always contribute to clear and complete explanations of system performance. For example, only 19 of the IDE’s 74 features were discussed in the scenario interviews. This result highlights the power of SWIMs as a user- and use-centric evaluation method. In later discussions with system users, we obtained support for our interpretation of this result as suggestive of feature-bloat: a substantial proportion of the functionality available in the IDE either was never put to use or was unsuitable to the purposes for which it was intended. Though it is possible that ethnographic approaches would have resulted in similar findings, we believe that the linking of scenarios, user roles, and system features to the consequences of system use (claims) provides a level of technology specificity and formative direction not always achieved from using ethnographic methods alone. An issue of major concern was the existence of claims disconnected entirely from specific scenarios. In the field study, we found that 223 of the 464 claims were not explicitly associated with any particular scenario. One possible explanation for this is that some aspect of the interview guide or interviewer

350


technique allowed participants to drift from explicit discussion of a focal scenario. A second explanation is that users were able to reflect upon and articulate consequences of use without explicit reference to concrete scenarios. During subsequent analyses, the claims connected with scenarios proved more specific and informative just because they were grounded in activities recognizably supported by specific IDE features. Because they were connected to a scenario and related system feature(s), they also contributed to formative evaluation. Still, the relative prevalence of disconnected claims suggests that individual users may often think about system benefits and drawbacks as transcending specific scenarios. To address this issue, it may be desirable to complement SWIMs, with its focus on scenarios and claims, with a parallel assessment of the specific features implemented in the system, for example, using the cognitive walkthrough method (Polson et al. 1992). Results from a complementary analysis such as this could then be triangulated against results obtained from scenario-based interviews. This adaptation of the approach may be particularly effective when used in group walkthroughs, where users can then compare and contrast their individual experiences with those of other users in the community. It is conceivable that such comparisons may help identify system benefits and drawbacks that go beyond specific features and that instead speak directly to the social and contextual factors inherent in collaborative systems. A related finding from the field study was the limited awareness users have about their roles in the overall process of work within their organization. Although the scenario value chains technique in SWIMs attempted to identify inter-scenario linkages and dependencies, this analysis was done ex post facto and was therefore unavailable during scenario and claims elicitation. Information about how an individual’s tasks relate to larger organizational processes made available earlier in the evaluation cycle would allow evaluators to more easily construct scenario value chains, and to consider scenario relationships during claims elicitation. One potential work around is to apply SWIMs iteratively to allow early identification of scenario value chains and later reflection on claims to produce more comprehensive and holistic evaluation results. 5.2. Assessing appropriateness of contextual sensitivity for evaluation Contextual sensitivity is an avowed goal for collaborative systems evaluation (Beyer and Holtzblatt 1998; Blandford 2002; Blanford and Rugg 2002; Blythin et al. 1997). Some researchers have argued, for instance, that the usefulness of accounts of work practices and the role that technology plays in them is dictated by how recognizable the accounts are to those who actually work with the technology (Button 2000). One strength of scenario-based evaluation methods is that these accounts are made comprehensible to users because they are grounded in actual instances of system use. The use of scenarios and claims may thereby


351

provide an effective middle ground between methods that are (a) purely featurebased and often detailed with technical language of expert evaluators, and those that are (b) purely focused on lived experiences, often characterized by ethnographic observation that may not be easily linked to specific system features and capabilities, and their use. An important corollary to the contextual sensitivity gained through scenariobased evaluation is the cost associated with operationalizing the approach. Our field application of SWIMs demonstrated that gathering data directly from users in the field was time-consuming, as was transcribing, coding, and analyzing the data. Here as well, we are encouraged by comparisons against ethnographic approaches, which are likely to require at least equal effort and may not produce the level of specificity that SWIMs is able to provide in its evaluation results. Nevertheless, it is necessary to acknowledge that the application of SWIMs can be expensive and on cost alone may not compare well with more focused, feature-based assessments. Evaluation teams should carefully weigh the benefits of the contextual detail gathered with SWIMs against the attendant costs of field evaluations. An ancillary benefit of SWIMs was enhancing stakeholders’ acceptance of the evaluation outcomes that resulted from an open process (e.g. focus groups) and that pointed to future cycles of re-design. The field application for PM LAV was, thus, judged to be valuable despite the apparent costs. Evaluation results including scenario value chains and potential causes of unrealized contributions (particularly organizational and social obstacles) reflect the benefits of this contextual sensitivity. Though controlled comparisons of SWIMs against alternative evaluation approaches is likely to prove difficult, future applications of the method may help support the conjecture that the benefits of this contextual richness outweigh the costs. 5.3. Assessing effectiveness for measuring contributions Although we were able to obtain some quantitative results in the field study of SWIMs, we also recognize that purely quantitative assessment of systems benefits will continue to present difficulties in evaluation of collaborative systems. The results showed that despite extensive use of probe questions during interviews and focus groups, it is extremely difficult for system users to identify and articulate measurable claims, for example, minutes or hours saved as a result of system support for a particular activity. Although we were sometimes able to move the discussion towards articulating specific, measurable benefits, movement along this dimension was typically arduous and results were only modest. Users in all roles, technical, administrative and managerial, experienced the same difficulty articulating specific measurable claims. There are several implications from this finding. First, it is possible that the benefits of collaborative systems are, by nature, largely intangible. This would suggest that it is not the evaluation approach that needs to be modified, but rather, our understanding of how to

352


measure social and psychological benefits that indirectly affect other, more tangible outcomes. Even so, the small number of measurable claims elicited provides some insight into the return-on-investment gained from the IDE implementation. If the objective of evaluation is considered to generate a “portfolio” of claims, some of which may be measurable, others tangible but not measurable, and yet others, intangible, SWIMs succeeded in delivering this result. A second implication of this finding is that it may be possible to use scenario value-chain analyses to link measurable claims, tangible benefits, and intangible outcomes across scenarios to demonstrate how the latter may amplify or constrain the former. Future research to explore techniques for analyzing scenario value-chains and additional field applications may help refine the method and provide further evidence for its potential usefulness. 5.4. Assessing support for multi-level and multi-perspective evaluation Our field study of SWIMs showed that claims analysis techniques produced outcomes to support aggregating claims at different levels of analysis and across different user roles, thereby reflecting multiple perspectives and affording multilevel evaluation. The aggregation of claims allowed potential assessment of return-on-investment, although capturing measurable claims presented significant challenges, as discussed in the previous section. The field study also showed that SWIMs may help overcome limitations of some current evaluation techniques, in particular, those that do not explicitly account for the complexity of cooperative tasks (Damianos et al. 1999). Scenarios and claims analysis anchor evaluation activity in a simple and common unit of analysis. We found that scenarios remained relatively constant over time, making it easier to move between discussions of system features, the cognitive dimension of system use, and group/ organization/social issues, even as the evaluation team traversed different levels of analyses. Also, claims analysis focused on unrealized contributions, envisioned scenarios, and feature enhancements provide direct support for decisions about prioritizing future design and re-design efforts. 6. Conclusion We have described SWIMs as an approach to evaluating complex, collaborative systems and have demonstrated its application in a field study of a working system. SWIMs represents a middle ground between two extremes of evaluation methods: focused but context-insensitive assessments of specific system features, and contextually sensitive ethnographic studies that can be difficult to translate into actionable outcomes. The SWIMs method suggests using concrete scenarios as the focal unit of analysis, and describes techniques for eliciting and analyzing evaluative claims associated with those scenarios. The field study showed that scenarios and claims analysis provide a way to evaluate collaborative systems by


353

identifying the contributions these systems provide to their host organizations. In doing so, we build on prior work showing that scenarios provide tractable and ecologically valid units of analysis for evaluation of complex, collaborative systems. We acknowledge that the field application of SWIMs, though conducted in an authentic setting over several months and spanning multiple levels within the organization, only reflects a single application of the method. Clearly, a single field study does not validate an approach as complex as that required to evaluate real-world collaborative systems. That said, the method instantiation we have described, the details of the field study, and reflections on both the positive and negative results suggest some potential for scenario-based evaluation as an important method for usable, replicable, in situ assessments of collaborative systems. Acknowledgements We would like to thank the office of the Program Manager, Light-Armored Vehicles and the United States Marine Corps for their support of this research. Comments and suggestions by the anonymous reviewers and the associate editor greatly improved this paper.

References Ackerman, M. (2000). The intellectual challenge of CSCW: the gap between social requirements and technical flexibility. Human-Computer Interaction, 15(2&3), 179–203. Asche, S. E. (1955). Opinions and social pressures. Scientific American, 193(5), 31–35. Bardram, J. E. (2000). Scenario-based design of cooperative systems: redesigning an hospital information system in Denmark. Group Decision and Negotiation, 9, 237–250. Bass, L., Clements, P., & Kazman, R. (1998). Software architecture in practice. Reading, MA: Addison-Wesley. Bernstein, A., & Vorburger, P. (2006). A scenario-based approach for direct interruptability prediction on wearable devices. Journal of Pervasive Computing and Communication, 2(2). Beyer, H., & Holtzblatt, K. (1998). Contextual design: defining customer-centered systems. San Francisco, Calif.: Morgan Kaufmann Publishers. Blandford, A. (2002). A case study on integrating contextual information with analytical usability evaluation. International Journal of Human-Computer Studies, 57, 75–99. Blanford, A., & Rugg, G. (2002). A case study on integrating contextual information with analytical usability evaluation. International Journal of Human-Computer Studies, 57, 75–99. Blythin, S., Hughes, J. A., Kristoffersen, S., Tom, R., & Rouncefield, M. (1997). Recognising “success” and “failure”: evaluating groupware in a commercial context. Proceedings of the international ACM SIGGROUP conference on Supporting group work: the integration challenge: the integration challenge, Phoenix, Arizona, United States, pp. 39–46. Button, G. (2000). The ethnographic tradition and design. Design Studies, 21(4), 319–332. Campion, M., Pursell, E., & Brown, B. (1988). Structured interviewing: raising the psychometric properties of the employment interview. Personnel Psychology, 41, 25–42.

354


Carroll, J. M. (1995). Scenario-based design: Envisioning work and technology in system development. New York: Wiley. Carroll, J. M. (2000). Making use: Scenario-based design of human-computer interactions. Cambridge, MA: MIT Press. Carroll, J. M., Koenemann-Belliveau, J., Rosson, M. B., & Singley, M. K. (1993). People and computers VIII. In J. L. Alty, D. Diaper & S. Guest (Eds.), Critical incidents and critical themes in empirical usability evaluation (pp. 279–292). Cambridge, UK: Cambridge University Press. Carroll, J. M., & Rosson, M. B. (1992). Getting around the task-artifact cycle: how to make claims and design by scenario. ACM Transactions on Information Systems, 10(2), 181–212. Chewar, C. M., McCrickard, D. S., & Carroll, J. M. (2005). Analyzing the social capital value chain in community network interfaces. Internet Research, 15(3), 262–280. Clements, P., Kazman, R., & Klein, M. (2001). Evaluating software architectures: Methods and case studies. Boston: Addison-Wesley. Cohen, J. (1960). A coefficient for agreement for nominal scales. Education and Psychological Measurement, 20, 37–46. Damianos, L., Drury, J., Fanderclai, T., Hirschman, L., Kurtz, J., & Oshika, B. (2000). ScenarioBased Evaluation of Loosely-Integrated Collaborative Systems (pp. 127–128). The Hague: Conference on Computer-Human Interaction (CHI). Damianos, L., Hirschman, L., Kozierok, R., Kurtz, J., Greenberg, A., Walls, K., Laskowski, S., & Scholtz, J. (1999). Evaluation for collaborative systems. ACM Computing Surveys, 31(3es). De Bruin, H., & Van Vliet, H. (2001). Scenario-based generation and evaluation of software architectures. Proceedings of the third international conference on generative and componentbased software engineering; Lecture Notes in Computer Science, pp. 128 – 13 Dewan, P. (2001). An integrated approach to designing and evaluating collaborative applications and infrastructures. Computer Supported Cooperative Work, 10(1), 75–111. Drury, J. (1999). Methodologies for evaluation of collaborative systems workshop. SIGGROUP Bull, 20(2), 50–51. Ge, G., & Whitehead, J. J. (2006). Scenario-based evaluation for configuration management systems (Technical Report): School of Engineering University of California, Santa Cruz. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theory: Strategies for qualitative research. Chicago: Aldine de Gruyter. Gray, W. D., & Salzman, M. C. (1998). Damaged merchandise? A review of experiments that compare usability evaluation methods. Human-Computer Interaction, 13, 203–261. Grudin, J. (1988). Why CSCW applications fail: Problems in the design and evaluation of organizational interfaces. Proceedings of the 1988 ACM Conference on computer-supported cooperative work, Portland, Oregon, pp. 85-93. Grudin, J. (1994). Groupware and social dynamics: eight challenges for developers. Communications of the ACM, 37(1), 92–105. Grudin, J. (2004). Return on investment and organizational adoption. Proceedings of the 2004 ACM conference on Computer supported cooperative work, Chicago, Illinois, pp. 324 - 327. Hughes, J., King, V., Rodden, T., & Andersen, H. (1994). Moving out from the control room: ethnography in system design, Proceedings of the 1994 ACM conference on computer supported cooperative work. Chapel Hill, North Carolina, United States: ACM Press. Ionita, M. T., Hammer, D. K., & Obbink, H. (2002). Scenario-based software architecture evaluation methods: An overview. Orlando, FL: Workshop on Methods and Techniques for Software Architecture Review and Assessment at the International Conference on Software Engineering. John, B. E. (1996). Evaluating usability evaluations. ACM Computing Surveys, 28(4). Kahn, H. (1962). Thinking about the unthinkable. New York: Avon. Kahn, H. (1967). The Year 2000 a framework for speculation on the next thirty-three years. In H. Kahn & A. J. Wiener (Eds.), The use of scenarios (pp. 262–264). New York: Macmillan.


355

MacLean, A., & McKerlie, D. (1995). Scenario-based design: Envisioning work and technology in system development. In J. M. Carroll (Ed.), Design space analysis and use-representations (pp. 183–207). New York: John Wiley. McGrenere, J. (2000). Bloat: The objective and subject dimensions (pp. 337–338). The Hague: CHI '00 extended abstracts on Human factors in computing systems. Mukkamalla, R., Britton, M., & Sundaram, P. (2002). Scenario-based specifications and evaluation of architectures for health monitoring of aerospace structures. Proceedings of the 21st Digital Avionics Systems Conference, pp. 1–12. Nielsen, J. (1994). Usability engineering: Morgan Kaufmann. Nielsen, J. (1995). Scenario-based design: Envisioning work and technology in system development. In J. M. Carroll (Ed.), Scenarios in discount usability engineering (pp. 59–83). New York: John Wiley. Nielsen, J., & Mack, R. L. (1994). Usability inspection methods. New York, NY: John Wiley & Sons. Oppenheim, A. N. (1992). Questionnaire design, interviewing and attitude measurement (2nd ed.). London: Pinter Publishers. Orlikowski, W., & Gash, D. (1994). Technological frames: making sense of information technology in organizations. ACM Transactions on Information Systems, 12(2), 174–207. Pang, N. L. S., Sanxing, C., Schauder, D., & Klein, R. R. (2005). A hybrid approach in the evaluation of usability for multimedia objects: case study of the media assets management platform for an advertainment production project toward Beijing Olympics 2008. Proceedings of the 3rd International Conference on Information Technology and Applications (ICITA), pp. 82–87. Pinelle, D., & Gutwin, C. (2002). Groupware walkthrough: adding context to groupware usability evaluation. Proceedings of the SIGCHI conference on human factors in computing systems: Changing our world, changing ourselves, CHI '02, Minneapolis, Minnesota, USA, pp. 455–462. Podsakoff, P. M., MacKenzie, S. B., Lee, J., & Podsakoff, N. P. (2003). Common method biases in behavioral research: a critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. Polson, P. G., Lewis, C., Rieman, J., & Wharton, C. (1992). Cognitive walkthroughs: a method for theory- based evaluation of user interfaces. International Journal of Man-Machine Studies, 36, 741–773. Ramage, M. (1999). The learning way: Evaluating cooperative systems. Lancaster, UK: Lancaster University. Rogers, Y., & Bellotti, V. (1997). Grounding blue-sky research: how can ethnography help? Interactions, 4(3), 58–63. Ross, S., Ramage, M., & Rogers, Y. (1995). PETRA: Participatory Evaluation Through Redesign and Analysis. Interacting with Computers, 7(4), 335–360. Scriven, M. (1996). Types of evaluation and types of evaluator. Evaluation Practice, 17(2), 151– 162. Seddon, P. B., Gaeser, V., & Willcocks, L. P. (2002). Measuring organizational IS effectiveness: An overview and update of senior management perspectives. The DATA BASE for Advances in Information Systems, 33(2), 11–28. Sokolov, J. (1999). Methodologies for evaluation of collaborative systems. SIGGROUP Bulletin, 20 (2), 44–46. Steves, M. P., Morse, E., Gutwin, C., & Greenberg, S. (2001). A comparison of usage evaluation and inspection methods for assessing groupware usability. In Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, pp. 125–134. USA: ACM Press. Stiemerling, O., & Cremers, A. B. (1998). The use of cooperation scenarios in the design and evaluation of a CSCW system. IEEE Transactions on Software Engineering, 24(12), 1171–1181.

356


Sutcliffe, A. (2002). The domain theory: Patterns for knowledge and software reuse. Mahwah, NJ: Lawrence Erlbaum. Twidale, M., Randall, D., & Bentley, R. (1994). Situated evaluation for cooperative systems. Proceedings of the 1994 ACM conference on computer supported cooperative work, Chapel Hill, North Carolina, pp. 441-452. Winckler, M. A., Palanque, P., & Freitas, C. M. D. S. (2004). Tasks and scenario-based evaluation of information visualization techniques. Proceedings of the 3rd Annual ACM international conference on task models and diagrams, Prague, Czech Republic, pp. 165–172. Yin, R. K. (2003). Case study research: Design and methods (3rd ed.). Thousand Oaks: Sage.