Narratives of Satisfying and Unsatisfying Experiences of Current Mobile Augmented Reality Applications Thomas Olsson Tampere University of Technology, Unit of Human-Centered Technology Korkeakoulunkatu 6, 33720 Tampere, Finland
[email protected]
Markus Salo University of Jyväskylä, Department of Computer Science and Information Systems Mattilanniemi 2, 40100 Jyväskylä, Finland
[email protected]
ABSTRACT
INTRODUCTION
Over the last few years, mobile applications demonstrating Augmented Reality (AR) – such as Layar, Junaio and Google Goggles – have been introduced to consumers. We conducted an online survey to explore the user experience (UX) of early stage mobile AR applications available in the market in spring 2011, covering both location-based AR browsers and image recognition AR applications for objectbased interaction. We identify various types of experiences such applications have evoked by qualitatively analyzing 84 users’ narratives of their most satisfying and unsatisfying experiences. The results highlight, for example, experiences of awareness of surroundings, empowerment, positive surprise, amazement and fascination from the novelty value, as well as some examples of immersion and social connectivity. The analysis indicates that the applications have not yet reached their potential in evoking a multifaceted user experience that is characteristic especially to AR. This work helps in understanding the experiential design potential in mobile AR and points out UX issues to further focus on in design.
According to the broad definitions of Augmented Reality (AR), it combines real and computer-generated digital information into the user’s view of the physical and interactive real world so that they appear as one environment [12,28]. AR integrates digital information with the tangible surrounding reality it relates to, for example by visually superimposing content on top of the view of the real world. Hence, AR could allow the user to access, manipulate and create location- and object-based information with an interaction paradigm that is intuitive and based on similar tangible interactions that occur in interacting with the physical world [22].
Author Keywords
Mobile augmented reality; mixed reality; location-based service; object-based interaction; end user application; user experience; evaluation; online survey; user narration. ACM Classification Keywords
H.5.2 [Information Interfaces and Presentation]: User Interfaces – User-centered design; H.5.1[Information Interfaces and Presentation]: Multimedia Information Systems – Artificial, augmented, and virtual realities. General Terms
Human Factors.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI’12, May 5–10, 2012, Austin, Texas, USA. Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00.
As result of active prior research and development of enabling technologies like computer vision and mobile computing and sensor technologies, we have lately seen mobile applications with AR features to penetrate to the day-to-day activities of consumers. With the recent introductions of mobile applications such as Layar1, Junaio2, Google Goggles3, ShopSavvy4, we are witnessing AR-related technologies to become exploitable for purposes of, for example, browsing location-based content, entertainment and gaming, as well as identifying products and objects while mobile. The current publicly available mobile AR-like applications can be roughly classified into two common approaches: AR browsers and image recognition-based AR applications. Both types rely heavily on the visual modality in providing digital information related to real-world objects and locations. AR browsers explore the augmented world through the camera view (“magic lens” [1]), usually visualizing points of interest based on the GPS location and magnetometer orientation. In addition to Layar and Junaio, good examples of AR browsers of today are Wikitude5 and Sekai Camera6. Image recognition-based AR connects surrounding objects, products, and other physical targets with related digital information with the help of visual recognition of QR codes, barcodes, or other graphic markers. Product-related 1
www.layar.com, 2www.junaio.com, www.google.com/mobile/goggles, 4 www.shopsavvy.mobi, 5www.wikitude.org, 6 www.sekaicamera.com 3
applications (e.g. ShopSavvy7, pic2shop8, and StickyBits9) represent a notable share of such AR applications but Google Goggles as a general image-based search application is probably the most used. Such object-related and marker-based applications can also be seen as a visual interface for physical browsing [18], and “Internet of Things” [3]. Despite the technical differences between these approaches, from the user’s perspective 1) the ways of interaction are very similar, 2) both augment the visual physical world in real time by utilizing the real world to access digital content related to locations or objects, and 3) both provide augmented content that is interactive. AR browsers provide an overall picture of the surrounding digital information, whereas image recognition AR allows the same but on a narrower, object-based scale. Considering the similarities from the user’s point of view, in this paper we consider the two technological approaches as one entity. The current AR applications can be disparaged as rather simple demonstrators of AR (e.g. not truly 3D content, imprecise alignment). Nevertheless, being the 1st generation of publicly available applications that utilize visual AR elements in the context of location- and object-based services, they provide a fruitful ground to conduct user research on a large scale. Our research explores the various elements of user experience of mobile AR by gathering subjective descriptions of end users’ most satisfying and unsatisfying experiences. To further justify our research, user experience research is only in its infancy when considering the recent AR applications and, in fact, AR related technologies in general [8,22]. THEORETICAL BACKGROUND AND RELATED WORK
Offering a stimulating and pleasurable user experience (UX) is becoming a central goal and design strategy in design of technology products and services. UX is regarded as a comprehensive concept describing the subjective experience resulted from the interaction with a technological product or service [13]. UX depends on contextual factors like social setting, cultural influences, and user’s other activities. The nature of the experience with technology can be, for example, pragmatic, emotional, symbolic, aesthetic, meaning-related or visceral [6,11]. Emotional elements are easily underutilized in design although emotions play a surprisingly great role in use of technology: they initiate behavior and motivate decisions; they allow focusing on interesting things as well as make people discard the unsatisfying [7]. In the following, we present a few UX-related frameworks relevant in contemplating the results of this study. Jordan [14] describes various types of pleasure in use of products: 1) physio-pleasure (related to body and senses), 2) psycho7 9
www.shopsavvy.mobi, 8www.pic2shop.com, www.stickybits.com
pleasure (related to the mind and emotions), 3) sociopleasure (related to relationships and status), and 4) ideopleasure (related to values and attitudes). Desmet [5] discusses the matter from a perspective of emotional reactions, identifying 5 categories: 1) surprise emotions & amazement, 2) instrumental emotions (e.g. disappointment, satisfaction), 3) aesthetic emotions related to intrinsic pleasantness (e.g. disgust, attracted to), 4) social emotions (e.g. indignation, admiration), and 5) interest emotions (boredom, fascination) arising from, e.g., challenge. Finally, Hassenzahl [11] has distinguished between two main perceptions of product quality: pragmatic and hedonic, which create the abstract consequences (i.e. experiences) in the user: appeal, pleasure, satisfaction. Pragmatic quality refers to the product’s ability to support achievement of behavioral goals (e.g. usefulness, usability, appropriateness). Hedonic quality is divided into three dimensions: 1) stimulation (e.g. enabling personal growth), 2) identification (e.g. expressing and building one’s identity through the product) and 3) evocation (provoking memories and emotions). Overall, all of these three frameworks describe the types of user experience, however from slightly different standpoints and levels of abstraction. Methodologically, evaluating the subjective emotional user experience can be carried out with, for example, interviews and subjective agreement statements, but also probing and diary methods, such as the critical incident technique (CIT) [10]. CIT allows gathering users’ perspectives as stories of the most significant experiences that have occurred during the use of a product. Storytelling in general has been regarded as an effective way to get a holistic view of the user experience [9,19]. For a user it is relatively easy to remember single remarkable experiences with the product without needing to engage in intensive cognitive reflection. Furthermore, storytelling methods also enable related contextual information like physical context, temporal context, task context, social context, and the technical and information context influencing the described event [15] to be recorded. Similar approaches of users reporting their momentary and episodic experiences remotely are utilized in day reconstruction method [16] and experience sampling method [21], from which we also draw inspiration. What comes to related work on user experience of AR, very little research exists. The survey paper by Dünser et al. [8] shows that only approx. 10% of AR-related papers published 1992-2007 included some sort of user evaluation. Moreover, even those had limited to evaluating early prototypes, mostly looking into perception and cognition issues and user task performance (see e.g. [20]). More importantly, to the best of our knowledge, academic publications about user research on the aforementioned applications are so far virtually non-existent (excluding our prior publication [24]). Finally, location-based service (LBS) is an important underlying theme of the current AR applications. The
design targets are partially similar as in the current AR applications: for example including finding something or someone, proximity-based notifications and location-based communication. Regarding LBS, user research has been rather numerous. In addition to evaluating demonstrators of LBS technology, there has been research on users’ needs and requirements [27], user acceptance and adoption [4], and user experience aspects [23]. As an example, Chang et al. [4] point out that adoption rates of location-aware services suffer from worry of security & privacy issues and quality of information. Other related concepts are objectbased interaction, Internet of Things and physical browsing [3]. For example, linking social content with physical products has been seen as a viable way for users to receive ratings or other user-created information about products and thus help in decision-making [17]. However, because of the extent of these fields, we leave further discussion out of the scope of this paper. OBJECTIVES AND METHODOLOGY
This paper addresses the question “What kinds of experiences have been present in interaction with current publicly available mobile AR-like applications?” We aim to gather qualitative insight on 1) the characteristics of experience that have taken place, 2) what features of the applications have had an influence on the experiences, and 3) under what kind of contextual factors have they occurred. The research question was approached with an extensive online survey for the users of current AR applications. In this paper we focus merely on the users’ narratives of their most satisfying and unsatisfying experiences. Other parts of the survey, e.g. perceived strengths and weaknesses of the most used applications, have been published in another paper [24]. Survey Design and Dissemination
In the most central part of the survey we asked users to freely describe 1) the most satisfying and 2) the most unsatisfying single experience with the AR application that they had used the most. The most satisfying experiences were expected to reveal aspects like what kind of experiences users have regarded as the most pleasing or interesting, what features of the applications are appreciated, and what is the overall potential of AR in creating a rich, delightful UX. The most unsatisfying experiences were expected to support this by shedding light on what kind of experiences users have found most inconvenient and highlighting the central hindrances in the applications that cause them. The most satisfying experiences being our primary interest, and to avoid respondent fatigue during the entire 15-minute survey, it was not compulsory to report the most unsatisfying experience. This was also suggested by several pilot respondents, both fellow researchers and end users. The verbatim instructions for the respondents are presented next (similar instructions for the most unsatisfying experience).
Think of the personally most satisfying and unsatisfying experiences or moments you have had with the application. Take a few minutes to be sure to come up with the most crucial experiences that you also remember well. If it is very hard for you to come up with the most unsatisfying one, it is ok to skip that. First, please consider the most satisfying one. We would like you to verbalize your experience as diversely as possible from your personal perspective. For example, you can consider the following things: 1) what happened and what was the overall situation in which the experience occurred, 2) how you felt about it or otherwise reacted to it, and 3) what aspects when using the application do you think caused your feelings or reactions.
To measure the various contextual elements in the described situation, a set of Likert-statements regarding the context was applied from Partala & Kallinen [26]. After the experience narratives the respondents were asked to respond to UX-related subjective agreement statements operationalized from the experience categorization by Desmet [5,6], as well as our earlier findings on users’ expectations [23] (statement #6 relating to intuitiveness and #7 to transparency). Such quantitative data from the statements was expected to support the qualitative insights from the experience narratives by looking into the diversity of experiences that have taken place overall. Here as well, the focus was on the respondents’ most used application. To ensure understandability of terminology and appropriate formulation of the questions, the survey was piloted first with 10 fellow researchers, all experts in empirical user research, and after that with 5 pilot respondents as representatives of end users. To reach the target group of AR application users, the survey call was spread through various web channels, such as specific AR related groups in Facebook and LinkedIn, well-known AR related blogs (Games Alfresco, Augmented Times, Augmented Planet), and a few email lists consisting of students and university staff members from the field of information technology. Furthermore, Layar, Wikitude, Junaio and Stickybits kindly announced the survey link through their Facebook and Twitter feeds, which all reach thousands of relevant persons. At the front page the concept of augmented reality was explained in layman’s terms and a requirement for participation was declared: the respondent had to have used AR application(s) to such a degree that one can point out and describe one’s experiences of them. The survey was informed to take about 15 minutes and that 50EUR Amazon.com gift cards would be raffled among the respondents. The survey was open for approximately a month during spring 2011. Narrative Analysis
Analysis of the experience narratives was based on datadriven, theory-independent coding (grounded theory approach). First, common themes were interpreted from the
data (open coding), iteratively constructing a hierarchy of categories in which each narrative could be classified. After this, each narrative was categorized into one or several categories. We argue that by not limiting the analysis on a specific UX framework, we could identify more relevant and a greater extent of different kinds of experiences. The analysis aimed at identifying three underlying elements from each narrative: 1) goal or activity the user was engaged in, 2) characteristics of the experience that occurred, and 3) features of the application that evoked or catalyzed the experience. As the narratives were usercreated, they often involved several different themes, which resulted in classifying each narrative to several categories (especially regarding the experience categories). The analysis was carried out with NVivo software, which provides a flexible way to construct, examine and revisit the themes and categorize the data. The categories were originally interpreted by one researcher but iterated in collaboration between two researchers. The same two researchers independently placed each narrative to the identified categories. Appropriate measures of inter-rater reliability are presented in the Results section. Respondents
The survey page was visited 2277 times, and we received 95 responses, of which 90 could be included in the analysis (some of them were flawed or evident acts of mischief). 74 of the respondents were male, 15 female and one preferred not to report their gender. The mean age was 31 years, ages ranging from 16 to 65 years (median 28). The respondents represented a total of 18 nationalities, for example American (7), Dutch (7), German (6), Austrian (4), and Finnish (27, probably explained by the research team being located in Finland). 13 respondents preferred not to report their nationality. 43 respondents had completed an undergraduate degree and 41 had a master’s degree or higher, meaning that in general the respondents were relatively highly educated. The respondents were asked 8 Likert statements about their general attitudes to technology and experiences with it, which generally showed a high level of technological orientation. For example, they agreed strongly with statements about being active in using mobile services and other new technology. In this sense, the overall profile of the respondents is in line with our target of gathering user perspectives from early adopters who are technologically adept enough to understand the principles and possibilities of AR. It is important to understand how the first adopters of a new technology value it and what are the issues that they see to stand in the way of making it more widespread. RESULTS
We first shortly describe background information about the used applications and contexts of use. The main contribution, i.e. the qualitative analysis of the most
satisfying and unsatisfying experiences, is first introduced by a brief analysis of the users’ activities and contextual factors related to the experiences. Finally, we shortly report the statistics about the subjective statements with regard to the respondents’ overall UX. Applications and Usage Statistics
Most respondents had been using several applications that match with our definition of an AR-like application (on average, 2.8 different applications per respondent). The question regarding the most used application highlights especially Layar (35 selections as the most used application), Google Goggles (22) and Junaio (16). Other responses included e.g. Stickybits (2), Nokia Point&Find, ShopSavvy and Pic2shop. All of the most used applications can be considered as representatives of an AR application that builds around location- or object-based information. Other, less used, applications represented a wide range: other utility-oriented AR browsers like SekaiCamera, Acrossair and FlightRadar24Pro, 3D augmentation (e.g. Argon), shopping-related (e.g. pic2shop, StickyBits, ShopSavvy), AR-gaming (e.g. Pandemica), and translation applications (e.g. WordLens). The majority had used their most used application for a rather long time already. 73% had obtained their most used application already “3 months ago or more”, 16% “1-3 months ago”, and only 4% “1-4 weeks ago” and 4% “Less than a week ago” (3% “hard to say”). Additionally, over half reported having used the application on at least a weekly basis after installing it: 19% “Daily”, 36% “Weekly”, 24% “Monthly”, and 18% “Less frequently than monthly” (3% hard to say). With regard to the contexts of use, almost all had used the application in densely populated areas such as city centrums and campus areas or industrial areas, but sparsely populated or rural areas had also been the context for roughly 25% of the respondents. Outdoors use was very frequent as expected (78% had used) but indoors use was also common, e.g. at home (62%), at office (38%), in shops (31%), or at events & exhibitions (7%). Expectedly, the gps-based applications were more used outdoors whereas the marker- and object-based mostly indoors. Activities and Contexts in the Experience Narratives
The users’ activities taking place in the reported most satisfying and unsatisfying experiences were identified as part of the qualitative analysis. They included many practical use cases typical to the AR applications in question, and the categories were also very similar in both the satisfying and unsatisfying experiences (Table 1). The ‘other’ category in Table 1 includes e.g. just playing around, testing 3D features, and showing off. All in all, the activities include both very goal-oriented and explorative use. The activities are further exemplified by the user quotes in the upcoming sections.
The user’s activity in the narrative Identifying objects and acquiring additional information about them Navigation and finding points of interest Entertainment and art Exploring the surroundings Acquiring specific location-related information Creating or utilizing own content Work- and study-related activities Shopping & price comparison Translation Other Not able to identify
16
16
13 10 8 6
7 0 7 0
5 3 3 3 6 16
0 6 0 0 6 26
Table 1. Users’ activities in the narratives of most satisfying ( ) and most unsatisfying ( ) experiences (# of cases).
The contextual factors related to the described experience were queried with six statements [26]. As shown in Table 2, the social context rarely had any influence on the experience, the reported experiences had occurred in both familiar and unfamiliar environments, and busyness was rarely present. The activity performed with the application was not considered especially critical in most cases. The need for the information had high variance (Std 2.1 in both), which consolidates the observation in the previous section about applications having been used both for a specific need and just for exploration and passing time. Interestingly, the distributions were highly similar in both types of experiences: in pairwise comparisons only the item about technical problems showed statistically significant difference between the most satisfying and unsatisfying experiences (p
