Follow Your Heart or Mind? Measuring ... - Semantic Scholar

Follow Your Heart or Mind? Affective Responses to Aesthetics

Follow Your Heart or Mind? Measuring Neurophysiological Responses and Subjective Judgments for Visual Aesthetics Indicate Submission Type: Emergent Research Forum Papers

Upasna Bhandari National University of Singapore [email protected]

Tillmann Neben University of Mannheim, Germany [email protected]

Wen Yong Chua National University of Singapore [email protected]

Klarissa T.T. Chang National University of Singapore [email protected]

Abstract This research-in-progress explores measurement of affective responses to visual stimuli by two methods: objective neurophysiological measures (galvanic skin conductance and facial electromyography) and subjective psychometric measures (self-report using survey). We aim to argue that neurophysiological measures can supplement traditional measures to capture effectively emotional responses to aesthetically manipulated visual design interfaces. Besides that, we also establish links between emotional responses: valence and arousal and aesthetics: classical and expressive to better understand the impact on design parameters on subconscious information processing. Keywords Neurophysiological, Psychometric, Valence, Arousal and Aesthetics

Introduction The number of mobile applications has been increasing predominantly in these recent years. Usage of mobile applications depends on what a user thinks and feels about the application (Bhandari et al. 2013; Vilnai-Yavetz et al. 2006;Tractinsky et al. 2007). Mobile applications with similar purposes such as Spotify or Rdio, are equipped with different interfaces attract different level of popularity and download. Aesthetics is a huge contributing factor with regards to the impact of the initial affective response to a product because it is the first impression to a user (Bhandari et al 2014). Studies are looking at how visual aesthetics can determine emotional impact (what the user feels) for a product (De Angeli et al. 2006; Hartmann 2006; Schenkman & Jönsson 2000; Bhandari 2014). Positive emotions are the major cause of positive aesthetic responses. Developers of existing mobile applications try to increase interface interactions by developing applications that are fully functional. However, that does not satisfy the pleasure and/or arousal dimension of end users. Therefore it is critical for developers to consider the role of emotions when developing a holistic system. Usage experience has been commonly evaluated using the Technology Acceptance Model (TAM) (Davis 1989; Davis et al. 1989). However this model has produced mixed results over the years. One thing that it has not been able to explain is how emotions impact its constructs like perceived usefulness (PU) and perceived ease of use (PeOU) (Sun & Zhang 2006). The reason is that models like TAM and extended TAM have an underlying assumption that higher order judgments are cognitive decisions. However there

Twenty-first Americas Conference on Information Systems, Puerto Rico, 2015

1


are some decisions that rely more on affective resources than cognitive resources (Beaudry & Pinsonneault 2010). Aesthetics form an important part of the impact of the initial affective response to a product. Studies have shown how visual aesthetics can determine emotional impact of a product and subsequent user judgments. Positive aesthetic responses cause positive interface interactions as aesthetics have been found to trigger positive emotions (De Angeli et al. 2006; Hartmann 2006; Schenkman & Jönsson 2000). Thus the role of emotions is critical for developing a holistic system that is functional as well as satisfies the pleasure (or, valence) and/or arousal dimension of end users. In this study, we measure the emotional impact of aesthetics objectively using neurophysiological methods (electrodermal activity EDA, facial electromyography EMG) and subjective responses using standard psychometric tools. Affective responses have been linked with subsequent user judgments like intention to use and/or intention to continue usage. This is true for retrospective evaluation of visual aesthetics as well. Since we measure emotions subjectively and objectively, we aim to determine whether objective or subjective emotional responses are better predictors of user judgments like aesthetic evaluation. To address the insufficiency of application design in the market, we argue that the over reliance and dependency on cognitive perspective for technology use can be neatly complemented by the emotional perspective. The study contributes to existing literature by bringing out the differences between measuring physiological affective responses and aesthetics and retrospective emotional evaluation. We explore how classical and expressive aesthetics are linked to various components of emotions and how these impact further user judgments. Practically, managers and designers benefit by knowing how emotions are effected by visual aesthetics and leads to higher order judgments, This further emphasizes that application development should focus on the experiential value that users derive from using the products.

Theoretical Background Emotions For core emotion, past research has proposed three different approaches for differentiating emotions: discrete, dimensional and appraisal models (Lazarus 1991; Scherer 2000). The dimensional perspective is based on the fact that emotion has two components namely, arousal and valence. Russel’s circumflex model of emotions is one of the most widely explored frameworks. Valence represents the pleasantness or unpleasantness dimension. Arousal on the other hand represents the degree of activation associated with the emotion. ‘Valence’ and ‘Arousal’ have been known to capture most of the variance in self-reported mood ratings (Suri & Gross 2012). They are the core components of emotional states and have been argued to be responsible for predicting choices. Emotions are different from moods because moods are long-lasting responses that may or may not be related to stimuli. Attitudes on the other hand are strong beliefs towards person, phenomenon or object. Emotions are thus short-lived reactions in the expressive, physiological and behavioral systems in response to stimuli. Emotions can be understood as a physiological arousal, a behavioral expression (affect) or a conscious expression of emotion (feeling). Our facial expression and vocal expression comes under behavioral expression. Research has shown our facial expression and vocal expression can be manipulated for various reasons e.g. social appropriateness. Moreover they are subjective in nature and interpretation can almost never be objective. Our focus for this study is to get as many objective measures for emotions as possible and thus we focus on physiological arousal as our independent variable for emotions. We do supplement this with traditional subjective measures. Specific to emotions, it has been shown that the higher the amount of pleasure, the more desirable the outcome will be (Elster 1998; Isen et al. 1978; Lang 1995). Arousal has been recognized as one of the components that are responsible for decision choice though slightly less than valence (Suri & Gross 2012). How much each one contributes to the final decision, there are two perspectives. One is that they contribute equally, fifty-fifty to the decision making process. However the other view is that one may take over the other, thus we need to be clear about which dimension is expected to play a more prominent role.


2


Besides core affect, artifacts, objects or stimulus also possesses emotional qualities. Artifacts, objects or stimulus have been studied in the information systems literatures in the form of affective quality, perceived affective quality (PAQ), affective judgment, and affective reaction etc. These studies have shown the subjective evaluation of user towards the affect eliciting qualities of an artifact (Russell 2003; Zhang & Li 2004). Although we recognize that this evaluation is influential and take measurements for emotions after the stimulus has been exposed, this is not the focus of the study.

Perceived Visual Aesthetics Aesthetics is a non-instrumental factor that has received lot of attention recently (Lavie & Tractinsky 2004; Lindgaard et al. 2006; Lindgaard 2007; Sonderegger & Sauer 2010; Van der Heijden 2003). Studies have linked it as a “response to a product” (Hassenzahl 2004), “visual appeal” (Lindgaard & Dudek 2003) and “beauty in appearance” (Lavie & Tractinsky 2004). It remains as a general agreement that aesthetically pleasing objects have a positive influence on product preference (Yamamoto & Lambert 1994). Strengthening the fact that when decisions are complex, consumers were found to prefer products with pleasing aesthetics, there is rising interest in looking at aesthetics from “pleasure providing” perspective (Schenkman & Jönsson 2000; Van der Heijden 2003). Studies have shown that emotional states like pleasure and pain play an important role in the formation of a consumer’s aesthetic judgment (Guyer 2008; Ginsburg 2008; Iseminger 2008). We focus on the classic and expressive aesthetic model proposed by (Lavie & Tractinsky 2004) due to its suitability for exploratory study like this and for clearer aesthetic manipulation.

Emotion Recognition Using Physiological Responses While conventional measures can be useful for various HCI related research, it would be ideal if objective data in terms of neurophysiologic responses could be collected for emotions. Thus, methods for unobtrusive measurement like heart functioning could be useful (Djmasbi et al. 2008; Loiacono & Djamasbi 2010). In recent years, progressions in sensor technologies have enabled more unobtrusive and comfortable measurement of the user's physiological changes than before. This created the possibility of embedding various sensors into different objects in the environment or to develop sensors that need not be in contact with the user. Such developments are essential for physiological measurements to become a feasible alternative. In Table 1, we discuss some of the widely used subjective as well objective measurements for various components of emotions. It would be e to see how these two approaches complement or supplement each other to make an informed decision on choice of method. Emotional Component

Subjective/Objective

References

Arousal

Subjective: Questionnaire

(Russell & Mehrabian 1977)

Subjective: Pictorial Scale

Self-Assessment Manikin (SAM), (Bradley & Lang 1994)

Objective: EDA

(Mahlke et al. 2006; Léger et al. 2010)

Subjective: Questionnaire

(Russell & Mehrabian 1977)


Self-Assessment Manikin (SAM), (Bradley & Lang 1994)

Objective: Facial EMG

(Sloan et al. 2002; Martinie et al. 2013)


Self-Assessment Manikin (Bradley & Lang 1994)

Objective: N/A

N/A

Pleasure/Valence

Dominance

Table 1.

Previous Studies Using Subjective/Objective Measurement to Capture Various Components of Emotions.


3


Hypothesis Development The two dimensions of aesthetics, classic and expressive, comprise of and cater to two distinct design aspects of a product. The classical aesthetics form the “visual clarity” dimension with dimensions like clean, clear, symmetric design etc. Expressive aesthetic on the other hand look at the creative side by looking at fascinating design elements, originality, sophistication etc. It has not been explored how different dimensions of design elements are related to various emotional dimensions. If visual aesthetics are believed to have influence over the affective responses, it would be interesting to see how physiological responses like EDA and facial EMG can capture visual aesthetics and how visual aesthetics are different in their predictive power from the subjective responses. A seemingly related question would then be how the affective responses impact the aesthetic evaluation of the stimuli. Regarding whether aesthetic evaluation is a cognitive or an affective process, (Tractinsky 2004) mentioned that although both can contribute significantly there are hints that aesthetic impressions are affective and formed immediately at a low level and thus precedes cognitive processes (Fernandez-Duque et al. 2003; Norman 2004; Pham et al. 2001; Zajonc & Markus 1982).

Figure 1:

Research Model

Both arousal and valence have been shown to contribute to the decision making process (Suri et al. 2013). Studies have mapped Russell’s arousal-valence dimension to classic-expressive aesthetics whereby classic aesthetics are related to valence based emotions and expressive aesthetics are linked to arousal based emotions (Tractinsky & Lowengart 2007). Besides that, facial reactions were shown to have relationship with the different levels of aesthetics in websites, specifically corrugator supercili (muscle above eyebrows) and zygomaticus (muscle in cheek) (Strebe 2011). An increased electrodermal activity has been associated with visually more stimulating work (Tschacher et al. 2012). Thus we hypothesize that: H1 (a) Facial muscle activation in corrugator supercili (negative valence) is negatively related to visual aesthetic evaluation. H1 (b) Facial muscle activation of zygomaticus (positive valence) is positively related to visual aesthetic evaluation. H1 (c) Electro dermal activity response is positively related to visual aesthetics evaluation. In this study we also propose that neurophysiological methodology and traditional subjective measurement/psychometric tools can be used in combination with each other to yield better results and measurements, thus we use electrodermal activity to measure arousal and facial electromyography to distinguish between positive and negative valence emotions along with subjective responses to these emotion based components. We believe that these objective data will be highly correlated with the retrospective evaluation of emotions. Hence, we hypothesize that: H2 (a) Facial muscle activation in corrugator supercili is positively related to corresponding subjective evaluation of negative valence in the behavioral experiment. H2 (b) Facial muscle activation in zygomaticus is positively related to corresponding subjective evaluation of positive valence in the behavioral experiment.


4


H2 (c) Electrodermal activity response is positively related to corresponding subjective evaluation of arousal in the behavioral experiment.

Research Methodology A behavioral study will be conducted as a lab experiment to collect the responses from the electrodermal activity (EDA) and facial electromyography (fEMG) to various aesthetically manipulated stimuli. Independent variable i.e., visual aesthetic is manipulated to get variations of a mobile application prototype i.e. high visual aesthetics and low visual aesthetics. High visual aesthetic mobile app prototype will have design guidelines like clean, clear and symmetrical design. Besides that, high visual aesthetic will also possess creativity, originality and have special effects. High visual aesthetic designs are guided by design guidelines from Lavie and Tractinsky (2004) who provided a model of Perceived Visual Aesthetics (Tractinsky 2004) and other existing studies (Tsai et al 2008, Park et al 2005, Ngo & Byrne 2001, Lauer 1985). This is done to isolate design guidelines that will provide designers with a clearer linkage between aesthetics and the emotional responses from users. Subjective data will be collected and compared with the physiological data obtained to test whether they correspond with each other. This addresses the issue of ecological validity or external validity (Dimoka et al. 2012).

Instrument Development To collect subjective responses to emotions, we use existing scales by Mehrabian and Russell (1974). The scale measures emotional components like pleasure and arousal. These are self-reported measures of felt emotions during the exposure to the stimuli to measure aesthetic evaluation we use scale forwarded by Lavie and Tracktinsky and adopt them slightly to suit the context of mobile applications.

Conclusions This study aims to examine users’ affective responses to visual aesthetics by a NeuroIS research in the context of mobile applications. By exploring in a context like mobile applications, this research investigates the effect of initial impact of visual design on how users process the initial assessment of the mobile application design in terms of facial muscle activation and skin conductance (electrodermal activity). Manipulating aesthetics gives us better understanding of how affective responses are captured in physiological responses and how it corresponds with the subsequent subjective evaluation. This can serve as a guide for better measurement choice when dealing with emotions caused by initial impact for a stimuli. Further processing of region of interest (ROI) related to aesthetics can guide which design guidelines lead to favourable emotional responses and is part of a future neuro study.

Acknowledgements (Optional) This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its International Research Centres in Singapore Funding Initiative and administered by the Interactive Digital Media Programme Office.


5


REFERENCES Beaudry, A. & Pinsonneault, A., 2010. The Other Side of Acceptance: Studying the Direct and Indirect Effects of Emotions on Information Technology Use. MIS quarterly. Bhandari, U. et al., 2013. Serendipitous Recommendation for Mobile Apps Using Item-Item Similarity Graph. Information Retrieval Technology. Springer Berlin Heidelberg, 2013. 440-451. Bhandari, U. and K. Chang, Role of Emotions and Aesthetics in ICT Usage for Underserved Communities: A NeuroIS Investigation. Proceedings of thirty fifth International Conference of Information Systems (ICIS) 2014. Bhandari, Upasna, et al. "Icon Types, Classical and Expressive Aesthetics, Pleasurable Interaction and Satisfaction with the Process of Semi-literate Users." SIGHCI (2014). Bradley, M.M. & Lang, P.J., 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental …. Davis, F. et al., Gmunden, Austria| June 5-7, 2014| www. NeuroIS. org. neurois.org Davis, F.D., 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS quarterly, 13(3), p.319. Davis, F.D., Bagozzi, R.P. & Warshaw, P.R., 1989. User acceptance of computer technology: a comparison of two theoretical models. Management science. De Angeli, A., Sutcliffe, A. & Hartmann, J., 2006. Interaction, usability and aesthetics: what influences users' preferences? … of the 6th conference on Designing …. Dimoka, A. et al., 2012. On the Use of Neuropyhsiological Tools in IS Research: Developing a Research Agenda for NeuroIS. MIS quarterly. Djamasbi, S., Tulu, B., Loiacono, E.T., Whitefleet-Smith, J. “Can a Reasonable Time Limit Improve the Effective Usage of a Computerized Decision Aid?” Communications of the AIS, October 2008 (23:22), pp. 393-408. Elster, J., 1998. Emotions and economic theory. Journal of economic literature. Fernandez-Duque, D., Grossi, G. & Thornton, I.M., 2003. Representation of change: Separate electrophysiological markers of attention, awareness, and implicit processing. Journal of Cognitive …. Ginsburg, F., 2008. Embedded aesthetics: Creating a discursive space for indigenous media. Critical Cultural Policy Studies: A Reader. Guyer, P., 2008. The psychology of Kant's aesthetics. Studies in History and Philosophy of Science Part A. Hartmann, J., 2006. Assessing the attractiveness of interactive systems. CHI'06 extended abstracts on Human factors in …. Hassenzahl, M., 2004. Emotions can be quite ephemeral; we cannot design them. interactions, 11(5). Iseminger, G., 2008. Aesthetic experience New York and London: …. Isen, A.M. et al., 1978. Affect, accessibility of material in memory, and behavior: A cognitive loop? Journal of personality and …. Lang, P.J., 1995. The emotion probe: Studies of motivation and attention. American psychologist. Lauer, D. A. 1985 Design basics. Belmont, CA, Wadsworth. Lavie, T. & Tractinsky, N., 2004. Assessing dimensions of perceived visual aesthetics of web sites. International Journal of Human-Computer Studies. Lazarus, R.S., 1991. Emotion and adaptation. Léger, P.M. et al., 2010. Psychophysiological Measures of Cognitive Absorption. Lindgaard, G., 2007. [CITATION][C]. Australian Journal of Emerging Technologies & Society. Lindgaard, G. & Dudek, C., 2003. What is this evasive beast we call user satisfaction? Interacting with computers.


6


Lindgaard, G., Fernandes, G. & Dudek, C., 2006. Attention web designers: You have 50 milliseconds to make a good first impression! Behaviour & information …. Loiacono, E. and Djamasbi, S. "Moods and Their Relevance to Systems Usage Models within Organizations: An Extended Framework," AIS Transactions on Human-Computer Interaction, June 2010 (2:2), pp. 55-72. Mahlke, S., Minge, M. & Thüring, M., 2006. Measuring multiple components of emotions in interactive contexts. CHI EA “06: CHI ”06 Extended Abstracts on Human Factors in Computing Systems. Martinie, M.A. et al., 2013. Evidence that dissonance arousal is initially undifferentiated and only later labeled as negative. Journal of Experimental …. Norman, D.A., 2004. Introduction to this special section on beauty, goodness, and usability. HumanComputer Interaction. Pham, M.T., Cohen, J.B. & Pracejus, J.W., 2001. Affect monitoring and the primacy of feelings in judgment. Journal of consumer …. Russell, J.A., 2003. Core affect and the psychological construction of emotion. Psychological review. Russell, J.A. & Mehrabian, A., 1977. Evidence for a three-factor theory of emotions. Journal of research in Personality. Schenkman, B.N. & Jönsson, F.U., 2000. Aesthetics and preferences of web pages. Behaviour & Information Technology. Scherer, K.R., 2000. Psychological models of emotion. The neuropsychology of emotion. Sloan, D.M. et al., 2002. Looking at facial expressions: Dysphoria and facial EMG. Biological Psychology. Sonderegger, A. & Sauer, J., 2010. The influence of design aesthetics in usability testing: Effects on user performance and perceived usability. Applied ergonomics. Strebe, R., 2011. Visual Aesthetics of Websites: The Visceral Level of Perception and Its Influence on User Behaviour. TPDL, pp.523–526. Sun, H. & Zhang, P., 2006. The role of moderating factors in user technology acceptance. International Journal of Human-Computer Studies. Suri, G. & Gross, J.J., 2012. Emotion regulation and successful aging. Trends in cognitive sciences. Suri, G., Sheppes, G. & Gross, J.J., 2013. Predicting affective choice. Journal of Experimental Psychology: …. Tractinsky, N., 2004. Toward the study of aesthetics in information technology. Tractinsky, N. & Lowengart, O., 2007. Academy of Marketing Science Review. Tsai, T., Chang, T. C., Chuang, M. C., & Wang, D. M. (2008). Exploration in emotion and visual information uncertainty of websites in culture relations. International Journal of Design. Park, S., Choi, D., and Kim, J. 2005 Visualizing E-Brand Personality: Exploratory Studies on Visual Attributes and E-Brand Personalities in Korea. International Journal of Human-Computer Interaction, 19, 1(2005), 7-34. Ngo, D.C.L., and Byrne, J.G. 2001. Application of an aesthetic evaluation model to data entry screens Computers In Human Behaviour 17, 2 (Mar. 2001), 149- 185. Tschacher, W., Greenwood, S. & Kirchberg, V., 2012. Physiological correlates of aesthetic perception of artworks in a museum. … Aesthetics. Van der Heijden, H., 2003. Factors influencing the usage of websites: the case of a generic portal in The Netherlands. Information & Management. Yamamoto, M. & Lambert, D.R., 1994. The impact of product aesthetics on the evaluation of industrial products. Journal of Product Innovation …. Zajonc, R.B. & Markus, H., 1982. Affective and cognitive factors in preferences. Journal of consumer research.


7


Zhang, P. & Li, N., 2004. Love at first sight or sustained effect? The role of perceived affective quality on users' cognitive reactions to information technology.


8