Ethnography, Evaluation, and Design as Integrated Strategies: A Case Study from WES Michael Khoo Department of Anthropology, CB 233 University of Colorado, Boulder, CO 80309-0233, U.S.A.
[email protected] Abstract. The Water in the Earth System (WES) collection is a collection of the Digital Library for Earth SystemEducation. As WES relies on its user community to generate metadataresources; identification of robust user community features, and ofpotential user community problems, is thereforeimportant. This paper describes (a) how ethnography is being used to studythe WES community; (b) how technological frames theory and technology usemediation theory is being used to analyse this data; and (c) how researchoutcomes are being used to generate recommendations for supporting future WES development.
1 Introduction: DLESE and WES The Digital Library for Earth System Education (DLESE: www.dlese.org) is aNational Science Foundation project that provides online access to earthsystem resources for a wide range of users, from schoolchildren touniversity professors. DLESE's development strategy emphasises the involvement of end-users in the design process.The aim is not just user-centred design, but sustainable user involvementin ongoing library operations; collections development, for instance, will rely upon libraryusers to contribute the metadata necessary to grow the collection. Userinvolvement is encouraged and supported in various ways; one strategyinvolves the development and support of user communities whose members meet in electronically mediated forums,and occasionally face-to-face. The Water in the Earth System (WES) community isa prototype user community that has been given institutional identity andcoherence through the specification of a collection scope (water), and a constituency (primarily highschool educators). As a prototype, WES is 'road testing' a number offeatures, such as community-led collections development, that will underpinthe DLESE collection. The current WES community is not, however, thought to be representative of future WEScommunities. Future WES users will likely be less 'digital library'literate, and will articulate more diverse needs, than the presentcommunity [44]. On a practical note, the financial cost of meetings between participants currently flown around the countrywill not scale with WES's growth, and electronically mediated communitydevelopment will play an increasingly important role. Future WES community growth will therefore be more distributed, andintegrate more people from more diverse backgrounds who are less familiarwith computational tools, than at present. As the participation of thiscommunity is central to WES's P. Constantopoulos and I.T. Sølvberg (Eds.): ECDL 2001, LNCS 2163, pp. 263-274, 2001. Springer-Verlag Berlin Heidelberg 2001
264
Michael Khoo
growth, a major developmental question for WES lies in the identification of emergent features of this community suitable for strategicsupport once initial WES institutional scaffolding is (inevitably) scaledback. Ethnographic tracking of WES's growth offers an opportunity to tryand identify emergent features of the WES community within a longitudinal framework.
2 Science, Technology, and Ethnography Ethnographic investigations of science and technologyrun the risk of falling between the stools of the various disciplines theyspeak to. Inquiries within anthropology,for instance, are infrequent, partly because of the perceived theoreticaldifficulty of constituting technical practice as a 'place' with anattendant 'culture' explicable through anthropological theory [10], partlybecause such accounts have been criticised for remaining on the level of theoretical speculation rather thanengaging with underlying technological realities [27]. Conversely,inquiries outside of anthropology can lack the contextual richness that theconsideration of human behaviour as cultural activity can provide. Some accounts [e.g. 1, 2, 5, 15, 16, 38, 39] havehowever transcended these boundaries, indicating that the locus ofinvestigation of science and technology lies between disciplines. Thissuggestion is supported by the contributionsof heterogeneous fields of inquiry such as STS and theories such asActor-Network Theory [e.g. 9, 24]. Applying these observations to the question ofthe study of technology design and use, we again find that much researchelides cultural concerns, focusing instead on more traditional human factor and usability questions. Thisapproach overlooks what will be argued below is an important element ofuser-centred design: understanding the culturally located perceptions oftechnological artifacts, what the artifact 'means' to designers and the users. This paper considers therefore not just thedigital library, but what a digital library can mean for designers andusers. In investigating these meanings, I am concerned particularly withthe difference between designers' and users' understandings of digital libraries, and theconsequences that these differences may have for library design anddevelopment, including (in this case) the development of sustainable usercommunities. Taking its interdisciplinary cue from science and technology studies, the investigation acknowledges a numberof disciplines, including (to greater or lesser degrees) anthropology,communication studies, computer science, and history. To address potentialproblems arising from the presence of divergent disciplinary methodologies, the research emphasises inductiverather than deductive methodologies and theoretical frameworks, such as:abductive inference [20]; actornetwork theory [9]; adaptive structurationtheory [11]; applied anthropology[e.g. 4, 22, 29, 30]; grounded theory [14]; technology context schemes[19]; technological frames theory [36]; and technology structuring theory[1, 2]. The placement of these approaches in proximityis not intended to imply the existence of theoretical congruences; rather they are presented as a 'toolkit' to be pragmaticallyselected from according to circumstance. It is however possible to discerna common intention to capture the complexities of human activity, throughmethods that often are inductive and iterative. Grounded theory, for instance,
Ethnography, Evaluation, and Design as Integrated Strategies
265
looks for discontinuities inthe data, asks "What theory does this situation remind the researcher of?"and encourages the researcher to widen their theoretical remit to bring newperspectives to bear upon the data. Similarly, abductive inference—a strategy of working with themost economical guess of causes—allows for the development offurther inferences to be adopted if they prove more economical thanexisting ones. In the following section, I will show how oneethnographic response to the question of "What might future requirements ofthe WES community be?" involves similar inductive cycles of inquiry.
3
An Ethnographic Research Cycle
The research carried out so far may be characterisedby although not confined to an iterative cycle of documentation andobservation, analysis, and outcome generation. 3.1 Documentation and Observation First, the design process is being documented. DLESE and WES publicationsand ephemera (e-mails, meeting agendas, Powerpoint presentations, web sitepages, etc.) have been collected and archived. Unstructured interviews andother informal interactions arebeing recorded in order to obtain participants' understandings ofparticular situations. Away from the field site, archival and library workis researching the history of libraries and library technology. Ethnography holds, however, that social process can not be discerned solelyfrom artifacts, or from participants' recollections of what they think theyhave done [17, 42]. What people think they do, and what they actually do,are often different things, and ethnography emphasises observing participants insitu. DLESE and WES meetings have thus been ethnographicallyobserved, with field notes recording both what is said, and also how it issaid—at what points meetings become contentious, off-topic, argumentative, aswell as points of general consensus and agreement. A complementary strategy is that of microethnography, the minute analysison computer of digitised video clips of meetings, etc. Microethnographyexamines human interaction in order to identify repetitive patterns ofverbal and corporal behaviour that are held to embody communicative practices indicative of individuals' desiresto accomplish certain tasks (for instance the promotion of an agenda at ameeting) [e.g. 12, 28, 32]. 3.2 Analysis Second, the collected materials are being analysed. The theoreticalframework currently guiding this evaluation is derived from Weedman'sincentive theory of usercentred design [46], and the work of Orlikowskiand colleagues on technological frames theory and technology use mediation [33, 34, 35, 36, 37]. Weedman's case study describes the design ofcomputational tools for a global change research project (Sequoia 2000[40]). She identified a number of factors outside of the technological artifactitself that impacted the user-centred design process. These included therole of politics and power relationships in the institutions
266
Michael Khoo
concerned, andthe fit of the technology with organisational processes. A third factor identified was that of the meaning ofthe artifact. Weedman found that the computer scientists saw the project asa revolutionary research challenge, while the geoscientists saw it as atool that would incrementally enhance their current research activities (making them faster, more efficient,etc.). The computer scientists were commited to extensive prototyping,which the geoscientists resisted, as it detracted from their research. Onthe other hand, the geoscientists wanted extensive debugging of the tools, which the computer scientistsopposed as it detracted from their research. At different stages in the project, therefore, geoscientists andcomputer scientists viewed the relevance of the project to their research in different ways, ways that produced varying levels of incentive to committo the project at any one time. These differences, Weedman holds, couldhave (although not necessarily would have) been alleviated by the presencein the project of 'boundary spanners,' individuals experienced in both geoscience and computer science whocould translate between the two communities. The technological frames theory of Orlikowskiand colleagues analyses the shared frames of reference underlyingindividual and collectiveperceptions of technology, the "particular assumptions, expectations, andknowledge of the technology, which then serve to shape subsequent actionstoward it." In any situation of technology use, there is potential formultiple technological frames to be present. The relationships between these frames can be either 'congruent'and 'incongruent.' Congruent frames imply similar expectations of atechnology. Incongruent frames imply "important differences inexpectations, assumptions, or knowledge about some key aspects of technology ... We expect that where incongruent technologicalframes exist, organizations are likely to experience difficulties andconflicts around developing, implementing and using technologies" [37]. Ihave suggested elsewhere that in addition to congruent and incongruent frames, a third level of technologicalframe interaction exists termed, after Kuhn [23], 'incommensurate.' Whileonly a small proportion of a community might espouse incommensurateperspectives, the impact of that small group on the overall communicative efficiency might be out of proportion totheir size within the group. Studies of the telephone call centres which dispatch emergency services,for instance, have found that callers who treat the service as a 'promptdelivery' service, akin to ordering a pizza, can become frustrated whentechnological and adminstrative constraints (including the need for dispatchers to collect and enter into adatabase certain information, and funding and staff shortages that resultin call triage) result in their concerns being addressed in an apparently tardyfashion [45]. These frustrations can generate arguments between callers anddispatchers that in rare cases have proved fatal [47]. As with Weedman's 'boundary spanners,'differences between technological frames may be bridged by mediators. In acase study of the introduction of a computer conferencing system in aninstitution, Okamura et al found that "the use of a computer conferencing system ... wassignificantly shaped by a set ofintervening actors-mediators ... These mediators adapted the tchnology toits initial context and shaped user interaction with it; over time, theycontinued to modify the technology and influence use patterns to respond tochanging circumstances" [33]. In the case of incongruent technological frames, therefore, mediators canperform on an ongoing basis 'translation' between variant technologicalframes, bridging the gaps in understanding between technology users.
Ethnography, Evaluation, and Design as Integrated Strategies
267
3.3 Outcome Generation Third, ethnographic findings derived from thetheoretical analysis of data are used to evaluate and inform WES's development in a number of areas. This is occuring in a number ofareas, including: requirements for technical support; definitions of thefuture institutional relationships between the WES community and WES andDLESE project centres; and requirements for growth towards community self-sufficiency, such as induction andoutreach services for new community members.
4
Case Study of the WES Community
It must be emphasised that each of the above strategies forms part of an iterativeanalytical cycle that privileges no particular stage. All stages of theresearch inform, and are informed by, all other stages. To illustrate thecycle in practice, I will describe the stages in an ethnographic study of a WES meeting, as follows: 1. Description of research heuristicestablished in previous research cycle 2. Observation 3. Analysis 4. Evaluations and recommendations 5. Modifications to research heuristic 6. Application of modified heuristic to new research cycle 4.1 Description of Research HeuristicEstablished in Previous Research Cycle I begin with a description of the research heuristic developedin prior analyses. Early observations of DLESE meetings indicated the possible presence ofdivergent technological frames in DLESE discourse. The frames weretentatively outlined, evidence for them was then sought in analyses ofDLESE documentation, and a refined analysis was drawn up. The technological frames identified consisted of two alternate definitions of 'digital library': firstly as a librarythat is digitised, and secondly as a digital object with library-likefunctions [21]. This analysis was reapplied in a number of contexts, andfound to be an parsimonious explanation for a number of observable phenomena. The first frame treats DLESE and WES aslibraries that are being digitised, essentially 'bricks-and-mortar'libraries made better and faster through the application of technology. Forinstance, in describing how library materials will be peer-reviewed and 'rigidly' catalogued,' DLESEpublications are referencing 'traditional' library practices. The secondframe treats DLESE as a digital object with library-like functions. In thisframe, DLESE is a 'portal,' a searchengine,' a 'network,' and a holder of 'digital artifacts.' Directlyreferencing Vannevar Bush's description of the Memex [8], the frameemphasises speed, seamlessness, hypertext, and fungible documents. A seriesof vignettes in DLESE's 'Community Report' describe how teachers and professors with approaching deadlines forcurriculum development turn to DLESE not just for resources but for ways tostructure diverse resources in systematic and meaningful ways. Acomputationally enhanced libary experience is seen as one way in
268
Michael Khoo
which technology can render the vast amounts of earthscience data already accumulated more meaningful [26]. These two frames arecontrasted in Table 1. I suggest that these two frames correspond toWeedman's identified communities of interest in the Sequoia 2000 project, with the geoscientists being more alignedwith the left-hand column above, and the computer engineers being morealigned with the right-hand column. Table 1. Technological frames andDLESE [21] DLESE as a Library that is Digitised
DLESE as a Digital Artifact with Library-likeCharacteristics
Metaphor: The bricks and mortar library
Metaphor: The technology enabled community
Centralised, enclosed building Local holdings Repository Catalogue Stable documents
Decentralised, open network Distributed resources Portal Search engines Fungible/customisable documents
The presence of two definitions of 'digital library' in one institutionalcontext can affect the institutional efficiency of that context. In oneexample, confusion over the nature of electronic holdings impacteddiscussions of DLESE's collection review policies. In these discussions, academic publications and web sites areperceived as qualitatively different resources, with different reviewrequirements. Textual analysis of the e-mail list for DLESE's collections developmentgroup has indicated that discussion of collection review policy lacked aconsistent definition of what review should consist of, with a tensionbetween (a) the accession of stable resources assessed through expert peer review, and (b) the accession ofonline resources reused in various pedagogical contexts by thecommunity. In another example, observation of meetingsnoted the presence of confusion amongst some community members (often newer community members) over the fact that DLESE and WES do not have'holdings' of their own (as bricks-and-mortar libraries might), but instead'hold' metadata records that point in the direction of resources elsewhereon the Internet (a function more in keeping with a digital object such as a portal site). Users' commentsreflected a feeling that while a bricks-andmortar library's scope isrepresented in its catalogue and in the resources on its shelves and in itsstacks, and a web site's scope canbe represented through its search and discovery interfaces, precisely howDLESE's scope could be understood was unclear. In unstructured interviewscarried out at these meetings some community members, especially those newto DLESE or WES, expressed the desire for a way to get them 'up-to-speed' with what exactly DLESE or WESwere about. These examples of alternate technologicalframings—meanings—of what a digital library is are the sortof differences potentially suitable for boundary spanning or mediation. Although detailed analysis of these roles in DLESE and WES has not
Ethnography, Evaluation, and Design as Integrated Strategies
269
yetbeen carried out, observation suggests that experienced community membersmay play an important role in this. These community members are able to seewhere WES (as a library) has come from, and also where WES (as a library) is going, and, importantly, areable to convey this information on an ongoing basis to other members of theWES community. They can mediate in discussions between experienced projectmembers and newcomers in whichmisunderstandings are occuring. While it may not be obvious either to theexperienced member or to the newcomer that they might mean different thingswhen they talk about a digital library, to the mediator these differencesare clearer. Thus they often intervene in the form of translation moves that can take the form of "Let metry and explain this ...," "I think what X is trying to say is ...," and soon. Awareness of the role that mediators play in DLESE and WES communitymeetings led to attention being payed to the points in DLESE and WES meetings where mediation appeared to beoccuring. 4.2 Observation at the WES Meeting These analyses and theoretical speculations regarding technological framesand mediative practices, developed during previous ethnographic work, weresubsequently brought to bear on an analysis of a WES Working Group meeting.In January 2001, a number of WES members, including teachers and pedagogists, were flown into Boulder,Colorado, for a two day meeting. The second day of the meeting was observedby the author. 4.3 Analysis of the WES Meeting Analysis of the WES meeting evidenced (a) the existence of competingtechnological frames along the lines outlined above, (b) the potential ofthese competing frames to impact the direction of the meeting, and (c) theexistence of various meeting strategies that were used to generate consensus amongst meeting members. One example of technological frame incongruenceat the meeting has also (as has been pointed out) been observed in DLESE.It consists of a basic confusion over whether or not WES holds resources,or holds metadata records that point to resources held elsewhere. While the first morning session had awide-ranging agenda, most of the talk remained focused on what exactly adigital library resource was. Members voiced opinions of the advantages anddisadvantages of both 'traditional libraries,' and the Internet and hypertext. Traditional libraries were thought tobe limited in their holdings, but to hold items of a guaranteed 'quality.' Conversely, while the value of being able to link to any resourceinstantly was appreciated, members were wary of the undifferentiated massof information perceived to exist 'out there' on the Internet. The debateoften revolved around the question of whether it was better to link to a wide range of resources (WES as digitalartifact with library-like characteristics), or only to those that had beenrigorously reviewed (WES as library that is digitised). As illustrativeexamples of the discussion, one member asked, in a question that juxtaposed both sets of technological framesdiscussed above, "But are we going to have a jumble of categories that canbe searched, or a series of collections just like a normal library? Will webe searching by call number?Or will we be searching by [other standards of interest to high schoolteachers, such as] National Science Standards?" Another member stated: "Idon't want just another web site that I have to
270
Michael Khoo
search through in order tofind a resource that's suitable." Athird member stated: "With a normal library you go and look at thecategories. So are we building an organised library, or a library organisedthrough searches?" The question appeared to remain unresolved. The secondmorning session, scheduled to discuss fundraising, had been in progress for about ten minutes before a communitymember ventured, "Finding resources is easy. The problem is the reviewprocess," a comment that effectively sidetracked the discussion back into adiscussion of two alternate reviewmodels for the rest of the morning. At this point, despite being providedwith a handout of a flowchart developed at earlier DLESE meetings thatrepresented a prototypical review procedure for DLESE, talk stilloscillated between whether peer review criteria should be narrow (the expert, traditional library model) or broad (thecommunity model). Some of the differences in perception have beensummed in the following Table 2. While WES members appreciated thetechnological possibilities of WES, as teachersand pedagogists, they had to be sure that the advantages were not justtheoretical but real (again, this is a reading of the meeting that in manyways reflects Weedman's analysis of the feelings of geoscientists in theSequoia 2000 project). Table 2. The WES Discussion WES as Library that is Digitised
WES as Digital Artifact Library-likeCharacteristics
with
Technologically unsophisticated but Technologically sophisticated but proven unproven Organisedlibrary Jumble of resources Organised by call number Organised by searches and pedagogicalrequirement Guaranteed quality Just a list oflinks Peer reviewer as expert gatekeeper Peer reviewer as community colleague The afternoon session moved from open discussion, to a brainstormingsession focused around a flip chart and a chalkboard, guided by the meetingfacilitator. In the brainstorming session, members constructed a 'conceptmap' that served as the focus of discussion. (A concept map is a graphical representation of an information field that consists of annotated nodes and lines, in which nodes correspond toconcepts within a particular subject area or domain, and lines delineatethe relationships between nodes, i.e., between concepts.) The meeting's useof the concept map involved the inscription on the chalk board of spatial arrangements of key concepts ofthe water cycle, an activity that generated large lists of points. Thesewere copied down by participants, several of whom volunteered to turn theselists into documents for later circulation. The discussion during the construction of the concept map wasfocused and productive; participants refered both to this session and tothe previous day's meeting, when concept mapping was also used, as usefuland constructive.
Ethnography, Evaluation, and Design as Integrated Strategies
271
4.4 Evaluations and Recommendations for WES This phase of the ethnographic cycle was concernedwith identification of opportunities for support for the WES community. Ashas been noted, during the meeting, participants made extensive use ofspatial modeling tools to make sense of the issues under discussion. A correlation was noted between theresolution of issues that were complex or time-consuming, and'brainstorming' sessions, with the issues addressed in this way includingthose affected by the presence of alternate technological frames, such as the design of a review policy for the WEScollection. As a corollary, it was felt that modeling tools and theirfacilitators fulfilled some of the requirements for boundary-spanning andtechnology use mediation. Questions that arose from this observation were, therefore, How might the use of such toolsbe made available to the wider WES community? In what circumstances mightthey prove useful? Specific suggestions and recommendations arising fromthe analysis were therefore fed back to WES community members for comment, and were subsequentlyincorporated into WES grant proposals for further computational tools andsupport. As the field of user-sustained digital libraries is a relativelynew one, it is envisaged that these findings and proposal recommendations will also have relevance for similarcommunity development support programs to be developed by DLESE. Besides being used to generate future funding, the findings are also beingapplied to a short- to mid-term project aimed at fulfilling a boundaryspanning or mediatory function within WES. The project will include thegeneration of interpretive materialsto present some of the findings so far, and the creation of on-lineresources available to community members that will discuss the differences between WES and DLESE considered as libraries that aredigitised, and digital artifacts with library-like characteristics. 4.5 Modifications to Research Heuristic It should be noted that this is not the point at which the analysis stops.The research strategy outlined above emphasises the role that ongoinginvestigation of emergent phenomena can play in gaining furtherunderstanding of a situation. The next stage, therefore, involves feeding back observations and analyses into the researchheuristic. From observation, it was realised that the concept maps of the WESbrainstorming sessions were being used as design tools. They surveyedmembers' perceptions of a knowledge field, and allowed those perceptions tobe folded back into the design process (of collection policy, review policy, etc.). They crystalised relationshipsinherent in the concept under discussion, and indicated where WES communitymembers placed their priorities. They also generated objects (the conceptmaps) that meeting members could then return to and re-discuss. Concept maps thereforeplayed a dual representational function: they represented the WEScollection to the WES community members present, but also represented theWES community needs to WES and DLESE. The useof concept maps and other brainstorming tools also appeared to beassociated with the emergence of mediator roles. Future research questionswill include, therefore: understanding what makes a community member becomea mediator; identifying the contexts in which successful mediation occurs; and identifying the tools that can helpmediators to be more
272
Michael Khoo
effective. In pursuing these questions, a number oftheoretical approaches suggest themselves, including 'situated learning,''boundary objects,' and 'boundary spanning.' In Lave and Wenger's description of situated learning, initial 'legitimateperipheral participation' by newcomers can lead to a 'centrifugal'induction of new members into a community of knowledge [25]. One approachfor WES might therefore beto ask how tools such as concept maps can mediate between new andexperienced members, such that new members feel they have a stake in thedesign process that encourages them to become more deeply involved. Starand colleague's concept of 'boundary objects' examines objects at the boundaries of variant communities of perceptionthat serve as orienting points for translational discourse [7, 43]. Again,one research direction might be to consider how mediators and the toolsthey use can function as boundary objects. A third approach, that of Weedman's analysis of incentives toparticipation in Sequoia 2000 [46], was suggested by an anonymous reviewerwriting in response to a first draft of this paper, and subsequentlyincorporate into the final draft. Note that in line with the methodologies outlined above, these newtheoretical approaches are being treated as neither correct nor incorrect,but as heuristics aimed at gaining broader understanding of the processesinvolved in distributed collaborative collections building. 4.6 Application of Modified Heuristic to New ResearchCycle The next (future) stage in the ethnographic cycle will involve thereapplication of the modified investigative toolkit to a new researchsituation. Currently, this is expected to be at a series of DLESE meetings,to be held in Arizona at the end of April 2001, and (again in Arizona) in August 2001.
5
Conclusion
The iterative ethnographic method outlined aims toproduce descriptive and generative outcomes that will support the development of a sustainable WES user community. The method will producearchival material that document the growth of WES, and provide data forfuture research. It will be used to produce policy recommendations for WESin the areas of infrastructure, tools, outreach, library policy, etc. In order to address both these requirements,the methodology has drawn from various inductive theories to develop aseries of iterative research cycles that aim to describe and reinforceemergent features of the community-led design process.
Acknowledgements I wish to thank Ed Geary of WES for inviting myparticipation, and the anonymous reviewer who suggested Weedman'spaper.
Ethnography, Evaluation, and Design as Integrated Strategies
273
References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28.
Barley, S.: Technology as anOccasion for Structuring: Evidence from Observations of CT Scanners and theSocial Order of Radiology Departments. Administrative Science Quarterly 31(1986) 78-108 Barley, S.: Images of Imaging:Notes on Doing Longitudinal Field Work. Organization Science 1(3) (August1990) 220-247 Barley, S., Orr, J. (Eds.): Between Craft and Science. TechnicalWork in U.S. Settings. Cornell University Press, Ithaca (1997) Barth, F.: A Personal View of Present Tasks and Priorities inCultural and Social Anthropology. In: Borofsky, R. (ed.) (1994) 349-361 Born, G.: Rationalizing Culture. University of California Press,Berkeley CA (1995) Borofsky, R.: Assessing Cultural Anthropology. McGraw-Hill, New York(1994) Bowker, G., Star, S.: Sorting ThingsOut. MIT Press, Cambridge MA (1999) Bucciarelli, L.: Designing Engineers. MIT Press, Cambridge MA (1996) Bush, V.: As We May Think. The Atlantic Monthly 176 (1945) 101-108. Callon, M.: Society in theMaking. In: Bijker, W., Hughes, T., Pinch, T. (eds.): The SocialConstruction of Technological Systems. MIT Press, Cambridge MA (1987)83-103 Clifford, J. Routes. HarvardUniversity Press, Cambridge MA (1997) DeSanctis, G., Poole, M.:Capturing the Complexity in Advanced Technology Use: Adaptive StructurationTheory. Organization Science 5(2) (1994) 121-147 Duranti, A.: Linguistic Anthropology. Cambridge University Press,Cambridge, 1997 Giddens, A.: Central Problems inSocial Theory. University of California Press, Berkeley CA (1979) Glaser, B., Strauss, A.: TheDiscovery of Grounded Theory. Aldine Publishing, Chicago IL (1967) Hakken, D,: Computing Myths, Class Realities. Westview Press,Boulder CO (1993) Hakken, D.: Computing and Social Change: New Technology and theWorkplace Transformation, 1980-1990. Annual Review of Anthropology 22(1993) 107-132 Hakken, D.: Cyborgs@Cyberspace. An Ethnographer Looks to theFuture. Routledge, New York London (1999) Hughes, J., King, V., Rodden, T., Andersen, H.: Moving Out of theControl Room: Ethnography in System Design. ACM CSCW (1994) 429-439 Jackson, M.: The Meaning of "Communication Technology": TheTechnology Context Scheme. Communication Yearbook 19 (1996) 229-267 Josephson, J. R., Josephson, S. G.: Abductive Inference.Computation, Philosophy, Technology. Cambridge University Press, Cambridge(1994) Khoo, M.: Community Design of DLESE's Collections Review Policy: ATechnological Frames Analysis. ACM/IEEE JCDL (2001) (In press) Kottack, C., Colson, E.: Multilevel Linkages: Longitudinal andComparative Studies. In: Borofsky, R. (ed.) (1994) 396-412 Kuhn, T: The Structure of Scientific Revolutions. 2nd edn. TheUniversity of Chicago Press, Chicago IL (1970) Latour, B.: Science in Action.Harvard University Press, Cambridge MA (1987) Lave, J., Wenger, E. SituatedLearning. Legitimate Peripheral Perception. Cambridge University Press,Cambridge (1991) Manduca, C., Mogk, D.: DLESE: A Community Plan. DLESE ProgramCenter, Boulder CO (2000) Martin, E.: Anthropology and the Cultural Study of Science.Science, Technology and Human Values 23(1) (1998) 24–44 Moerman, M.: Talking Culture: Ethnography and ConversationAnalysis. University of Philadelphia Press, Philadelphia PA (1988)
274
Michael Khoo
29. Moore, S.: Explaining the Present: Theoretical Dilemmas inProcessual Ethnography. American Ethnologist 14(4) (1987) 727-736 30. Moore, S.: The Ethnography of the Present and the Analysis ofProcess. In: Borofsky, R. (ed.) (1994) 362-376 31. Newman, M., Landay, J.: Sitemaps, Storyboards, and Specifications:A Sketch of Web Site Design Practices. ACM DIS (2000) 263-274 32. Nofsinger, R.: Everyday Conversation. Waveland Press, ProspectHeights IL (1999) 33. Okamura, K., Fujimoto, M.,Orlikowski, W., Yates, J.: Helping CSCW Applications Succeed: The Role ofMediators in the Context of Use. ACM CSCW (1994) 55-65 34. Orlikowski, W.: The Duality ofTechnology: Rethinking the Concept of Technology in Organizations.Organization Science 3(3) (1992) 398-427 35. Orlikowski, W.: Learning From Notes: Organizational Issues inGroupware Implementation. MIT Sloan School Working Paper #3428-92 (1992) 36. Orlikowski, W., Gash, C.:Technological Frames: Making Sense of Information Technology inOrganizations. ACM Transactions on Information Systems, 12 (April 1994),174-207 37. Orlikowski, W., Yates, J., Okamura, K., Fujimoto, M.: ShapingElectronic Communication: The Metastructuring of Technology in Use. MITSloan School Working Paper #3611-93 (1996) 38. Orr, J.: Talking About Machines.Cornell University Press, Ithaca (1996) 39. Rabinow, P.: French DNA: Troublein Purgatory. University of Chicago Press, Chicago IL (1999) 40. Sequoia 2000:http://s2k-ftp.cs.berkeley.edu:8000/ 41. Shapiro, D.: The Limits ofEthnography: Combining Social Sciences for CSCW. ACM CSCW (1994)417-427 42. Simonsen, J., Kensing, F.: Using Ethnography in Contextual Design.Communications of the ACM 40 (7) (July 1997) 82-88 43. Star, S., Griesmer, J.: Institutional Ecology, "Translations" andBoundary Objects. Social Studies of Science 19 (1989) 387-420 44. Sumner, T., Dawe, M.: Looking at Digital Library Usability from aReuse Perspective. ACM/IEEE JCDL (2001) (In press) 45. Tracy, K.: Interactional Trouble in Emergency Service Requests: AProblem of Frames. Research on Language and Social Interaction 30(4) (1997)315–343 46. Weedman, J.: The Structure ofIncentive: Design and Client Roles in Application-Oriented Research.Science, Technology, and Human Values 23(3) (1998) 315–345 47. Whalen, J., Zimmerman., D., andWhalen M.: When Words Fail: A Single Case Analysis. Social Problems 35(4)(1988) 335–362
