What does your chair know about your stress level?

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE: AFFECTIVE AND PERVASIVE COMPUTING FOR HEALTHCARE

1

What does your chair know about your stress level? Bert Arnrich, Cornelia Setz, Roberto La Marca, Gerhard Tr¨oster, Member, IEEE, and Ulrike Ehlert

Abstract—The inferred cost of work-related stress call for early prevention strategies. In this, we see a new opportunity for affective and pervasive computing by detecting early warning signs. This work goes one step towards this goal. A collective of 33 subjects underwent a laboratory stress intervention while a set of physiological signals was collected. In this work we investigate whether affective information related to stress can be found in the posture channel during office work. Following more recent work in this field we directly associate features which are derived from the pressure distribution on a chair with affective states. We found that nervous subjects reveal higher variance of movements under stress. Furthermore we show that a person-independent discrimination of stress from cognitive load is feasible when using only pressure data. A supervised variant of a self organizing map which is able to adapt to different patterns of stress responses reaches an overall accuracy of 73.75% with unknown subjects. Index Terms—stress, cognitive load, body pressure distribution, self organizing maps

I. I NTRODUCTION

I

N western countries work-related stress is of growing interest. Recently, the European Foundation for the Improvement of Living and Working Conditions called the attention on risks and consequences of work-related stress [1]. Based on national surveys available in the EU countries they found that stress is the second most common work-related health problem across the EU. Furthermore they present recent research findings which show that work-related stress is associated with several illnesses, such as cardiovascular diseases, musculoskeletal disorders, particularly back problems and neck-shoulder-arm-wrist-hand problems. These diseases lead to absence of work or even permanent disabilities resulting in high economic costs: 45 billion Euro in Germany, 7.5 billion Euro in the Netherlands and 4.9 billion Euro in Switzerland. From a biological point of view, a “stress reaction” occurs if an organism is in danger or injured. The resulting physiological changes were first described in 1936 by Hans Selye [2]. Stress affects mainly hypothalamus-pituitary-adrenal axis, the sympathetic nervous system, the parasympathetic nervous system, and the immune system. These different systems react on different time scales and influence each other. In the short term, the stress reaction helps the body to adapt to the stressor, e.g. by providing energy and by suppressing inflammations and infections. However, if the organism has no time to recover and the stress reaction proceeds over longer time, this can have adverse effects. Work-related stress occurs when there is a mismatch between job demands and the capabilities, B. Arnrich, C. Setz, and G. Tr¨oster are with the Electronics Laboratory, Swiss Federal Institute of Technology Zurich, Switzerland, e-mail: {barnrich, setz, troester}@ife.ee.ethz.ch. R. La Marca and U. Ehlert are with the Institute of Psychology, Clinical Psychology and Psychotherapy, University of Zurich, Switzerland, e-mail: {r.lamarca, u.ehlert}@psychologie.uzh.ch.

resources or needs of the worker [3]. If the worker is not able to recover, long term damage may result in the development of depression, gastric ulcers or increased sensitivity to infections. We see a new opportunity for affective and pervasive computing in a continuous monitoring of stress levels during office work. If a “Personal Stress Prevention Assistant” could advert us of stressful situations and responses during work and tell us how well we are recovering afterwards, it could help us to create a better “Work-Life-Balance”. Furthermore, such a “Personal Assistant” could enable early prevention, could significantly reduce stress-related economic costs and might also be used in psychotherapy. This work goes one step towards this goal. Since we are aiming at an application in the office environment, a suitable experiment which is close to a real-life office situation had to be found. Existing studies often use mental workload as stress-eliciting factor. However, there are other contributing factors such as social threat by superiors. We have therefore chosen a stress test which includes both factors. The Montreal Imaging Stress Task (MIST, [4]) is a standardized computerbased task consisting in a stress and a control condition: The stress condition combines mental arithmetic problems under time pressure with social-evaluative threat while the control condition consists in mental arithmetic with neither time pressure nor social evaluation which is similar to normal working on a computer. Because an office worker is always confronted with a certain amount of cognitive load, we try to discriminate a state of mental and psychosocial stress from a state of (mild) cognitive load rather than distinguishing between a rest (doing nothing) and a stress condition. For an “every day life application” it is important to use unobtrusive sensors. We therefore investigated to what extend a pressure mat mounted on a chair is appropriate to discriminate between stress and mild cognitive load. Our main hypothesis is that the posture channel contains affective information related to stress during office work. Following more recent work in this field we directly associate features which are derived from the pressure distribution measured on a chair with affective states. The features used in this work are derived from the spectra of the norm of the center of pressure. Since we aim at discriminating between stress and cognitive load automatically, the extracted feature vectors are fed into a classifier. In this work self organizing map classifiers have been employed to handle different patterns of subject’s stress responses. In the following we present related work in Section II. Next we describe the data collection during a human experiment in Section III. In Section IV we describe the methods used to extract features and discriminate between cognitive load and stress. The results are presented in Section V. Finally, we will draw conclusions and discuss our results in Section VI.

Copyright (c) 2009 IEEE. Personal use is permitted. For any other purposes, Permission must be obtained from the IEEE by emailing [email protected]. Authorized licensed use limited to: ETH BIBLIOTHEK ZURICH. Downloaded on January 12, 2010 at 09:29 from IEEE Xplore. Restrictions apply.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. 2

IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE: AFFECTIVE AND PERVASIVE COMPUTING FOR HEALTHCARE

II. R ELATED W ORK A. Pressure sensing for recognizing activities and postures Pressure sensing on the human body covers a wide field in pervasive computing. The activity of different muscles has been measured by force sensitive foil sensors [5], [6]. Upper arm positions have also been recorded by resistance changing foams [7], [8]. Sitting postures can serve as a measure for healthy sitting behavior or as a user interface. There exist a wide range of methods for detecting postures automatically. Pressure sensor mats have been used to detect the sitting posture in a chair [9], [10], [11], [12]. In addition, pressure mats have also been employed to identify a driver in a car [13], to measure driver’s fatigue based on driver’s postural changes [14] or as a measure for comfort [15], [16]. B. Pressure sensing in affective computing In [17], the authors describe techniques and tools to recognize affective states which are important in the interplay between emotions and learning. They suggest to use symbolic postures that convey a specific meaning about the actions of a user sitting in an office, e.g. leaning forward towards a computer screen might be a sign of attention. The objective in [18] was to classify emotions on the basis of facial expressions, gross body movements, and conversational cues. Body movements are measured by a pressure mat system. It is suggested to divide the sensing region into quadrants and to take the net force in each quadrant as features. In [19], it is stated that the relation between posture data and affective information remains unclear. The authors argue that in [20] it is concluded that body movements only carry information about the intensity of the emotion that is being experienced. In contrast, the results in [21] show that both body movements and positions carry information about four distinctive emotions and attitudes. Looking at some of the above mentioned methods which relate posture data with affective information, we can identify two main approaches: (i) recognize postures in a first step and associate them with affective states in a second step(e.g. [17], [19]), and (ii) directly associate features derived from the pressure distribution with affective states (e.g. [14], [18]). In this work we follow the more recent second approach. III. DATA COLLECTION A. Subjects A number of 33 male, healthy subjects (mean age: 24.06; mean BMI: 23.63) were recruited for the study. Inclusion criteria included male sex and an age range of 18 to 40 years and dexterity. Exclusion criteria included depression, self-reported acute and chronic somatic or psychiatric disorders, medication in the last two months, the consumption of psychoactive substances, and excessive consumption of alcohol or tobacco. The subjects received monetary compensation (80 CHF) for participating in two sessions of two hours. In order to eliminate possible habituation effects, half of the subjects were exposed to the stress condition in the first session and to the control

Fig. 1. Experiment procedure for stress and control condition. All subjects participated in both conditions during two sessions in random order.

condition in the second session while for the other half the sequence was vice versa. In this study we had to exclude 5 subjects because of erroneous pressure sensing during the experiment. This resulted in a total of 28 subjects. B. Human experiment The Montreal Imaging Stress Task (MIST; [4]) was developed in order to evaluate effects of psychosocial stress on physiology and brain activation by functional Magnetic Resonance Imaging (fMRI) or Positron Emission Tomography (PET). The MIST has shown to induce a moderate stress response [22], [23]. The test was slightly modified for a use outside of an fMRI environment by psychologists in agreement with the developers of the MIST. In our study, subjects were told the cover story that they are taking part in an experiment investigating the relationship between cognitive performance and physiological characteristics. Instead, the subjects were confronted with mental stress under time pressure and social evaluative threat during one session (the stress condition) and with mild cognitive load during another session (the control condition). Both conditions reflect well the situation of stress and normal cognitive load during office work. Fig. 1 illustrates the schedules for the stress and control conditions. 1) Stress Condition: During the stress condition the subjects performed mental arithmetic tasks on a computer. The program adapted the difficulty and the time limit such that each subject could only solve 45-50% of the arithmetic problems correctly. The subjects entered the answers on a keyboard. A time bar showed the remaining time for solving the task and “Timeout” was displayed when the time had passed. When subjects gave an answer before the end of the given time, the feedback “right” or “wrong” was displayed. A color bar showed a comparison between the individual performance and the performance of a simulated, representative comparison population. The subjects were told, that the test does not work if their performance does not reach the green range of the bar and that the study leader would monitor the whole experiment. This represents a social evaluative threat. Additionally after each block of mental arithmetics, the subjects received feedback by the experiment and the study leaders regarding their performance. Fig. 2 shows the screen display during the stress condition. The following experimental schedule was chosen:


This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. ARNRICH et al.: WHAT DOES YOUR CHAIR KNOW ABOUT YOUR STRESS LEVEL?

3

30 20

2 3

5

1 0 9

5

Comparision Population

10

10

20

30

Personal Performance

4

Time Bar

8 7

5 6

5

10

15

20

25

30

5

10

15

20

25

30

Rotary Dial

Fig. 2.

MIST screen during stress condition.

1) Instructions and signing of an informed consent. 2) Habituation and baseline period (20 minutes): The subjects can read magazines. 3) Cognitive stress I (4 minutes): Mental arithmetics under time pressure with performance evaluation. 4) First feedback phase inducing mild social stress: The experiment leader tells the subject that his performance was not good as expected and asks questions, e.g. “Is there a problem with the keyboard?”. 5) Cognitive stress II (4 minutes): Like cognitive stress I but with additional pressure not to fail again. 6) Second feedback phase inducing strong social stress: The study leader asks questions referring to personal problematic of the subjects, e.g. “Do you take drugs?”. Finally, the study leader requests the subjects to try again with more effort. 7) Combination of strong social and cognitive stress (4 minutes): Like cognitive stress II but additionally the study leader stays behind the subject and observes him. After the termination of the stress task, the subjects again reads magazines during a recovery period of one hour. 2) Control Condition: For the control condition, the schedule is analog to the one described above. However, during the mental arithmetic phases, there is no time limitation for solving the tasks and no social evaluation. The color and time bar are thus not displayed. During the feedback phases, the experiment leader ask friendly, neutral questions, e.g. “How did you come to participate in this study?”.

Fig. 3. Tekscan pressure mat images taken from the seat of the chair for the two sitting postures “leaning left” and “leaning right”. Based on all 1024 sensor elements the center of pressure, here shown as black circle, was computed.

map in order to discriminate the experimental conditions mild cognitive load and stress(IV-B). A. Feature extraction For each frame of the pressure mat we computed the center of pressure (CoP) based on the 1024 sensor elements. In Fig. 3 two exemplary frames including the CoP for two sitting postures are shown. Based on the x- and y-component of the CoP we derived the norm of the corresponding vector in order to attribute the CoP with a single value. Fig. 4 shows the norm of the CoP during the experiment (stages 3 to 7 in Fig. 1) for one subject during the control and stress condition. It can be seen that fast movements on the chair, which are represented by spikes in the norm of the CoF, occur more often during the stress condition. In order to characterize the movement properties of the CoP we have computed the corresponding spectra over the experiment stages 3 to 7. Fig. 5 shows the spectra of the norm of the CoP during the stress and the control condition for the same subject used in Fig. 4. It is clearly visible that the spectra differ considerably between the control and the stress condition. However, it has to be noted that these differences are not as pronounced for all the subjects. In order to obtain a feature vector for each subject, we devided the spectra in 20 frequency bands of equal width and computed the mean value of each band. These mean values will serve as input features for the classification tasks described in the following.

C. Recorded Signals and Synchronization During the human experiment several signals have been recorded including physiological signals, acceleration and sitting pressure. As introduced in Section I, this work focuses on the evaluation of the pressure distribution recorded by a pressure sensitive mat placed on the chair. The CONFORMat system [24], developed by Tekscan, was used to obtain the pressure distribution. The mat consists of 1024 sensor elements each having a relative accuracy of 10%. The mats have been calibrated before use with the Tekscan vacuum calibration system in the range of 0 to 3.3 N/cm2 . A sampling frequency of 25 Hz was choosen. IV. M ETHODS In the following, we first explain how we extract feature vectors from the pressure data (IV-A). Next, we describe how we feed the extracted feature vectors into a self organizing

B. Self Organizing Map In this work, we use self organizing maps (SOMs) to discriminate between cognitive load and stress. The reason behind this choice is that people react differently to stress and a method which is able to adapt to local cluster structures is therefore needed. In the following we will describe how a SOM adapts to local structures. Originally, the main purpose of SOMs was to visualize a high-dimensional signal space on a two-dimensional grid of nodes while preserving the topological relationships of the signal space on the two-dimensional display [25]. Starting with a concise introduction of the basic principle of SOMs we will later show how we can use this method for prediction. 1) SOM Basics: A SOM consists of a set of unconnected units which are spatially ordered in a two-dimensional grid. Each unit in the map is equipped with a weight vector


This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON INFORMATION TECHNOLOGY IN BIOMEDICINE: AFFECTIVE AND PERVASIVE COMPUTING FOR HEALTHCARE

1e+01 1e−05

1e−02

spectrum

23.5 23.0 22.5 21.5

1e−08

22.0

norm center of pressure

24.0

24.5

4

0

5000

10000

15000

20000

25000

0.0

0.1

0.2

0.3

0.4

0.5

0.4

0.5

frequency bandwidth = 1.15e−05

1e+00 spectrum

1e−02

23.0 22.5 21.0

1e−06

21.5

1e−04

22.0

norm center of pressure

23.5

1e+02

24.0

sample index

0

5000

10000

15000

20000

25000

sample index

Fig. 4. Illustration of the norm of the center of pressure for one subject during the experimental stages 3 to 7. Above the control condition is shown while below the stress condition is depicted. The color coding represents the experimental stages which were described in Section III-B1. Red: stage 3, Yellow: stage 4, Green: stage 5, Turquoise: stage 6, Blue: stage 7. We can identify that fast movements on the chair which are represented by spikes, occur more often during the stress condition.

which has the same dimension as the feature space. In order to map the feature space to the SOM we assume that we have a feature vector xstim = [ξ1 , ξ2 , . . . , ξn ]T ∈