Mobile System for Unobtrusive Monitoring of ... - CiteSeerX

6 downloads 0 Views 321KB Size Report
[10] William G. Iacono, David T. Lykken, Lori J. Peloquin, Ann E. Lumry, Roger H. Valen- tine, and Vicente B. Tuason. Electrodermal activity in euthymic unipolar ...

Mobile System for Unobtrusive Monitoring of Electrodermal Activity in Daily Life Franz Gravenhorst, Bernd Tessendorf, Amir Muaremi, Cornelia Kappeler-Setz, Bert Arnrich, and Gerhard Tröster Wearable Computing Lab., ETH Zurich Gloriastr. 35, 8092 Zurich, Switzerland {lastname}

Abstract Mental disorders increasingly generate severe negative personal and economic impact. There is a strong need for measurement devices to support therapists in treating and monitoring their patients’ mental health. Measuring the Electrodermal activity (EDA) at the patient’s foot is a promising approach. However, there are still no measurement systems available yet that allow for continuous long-term monitoring in daily life. In this paper we present a mobile EDA measurement system based on commercially available certified sensor units and a newly developed Android smartphone application. The Android application provides a GUI for real-time data visualization and supports instructed and automated annotation features. Self-assessment Manikin surveys enable logging the user’s perceived emotional state over time and synchronized with the sensor data. We performed a proof-of-concept study involving one healthy participant and interpret the resulting signals from a physiological point of view. We found the system to work reliably even in mobile settings without any technical problems or data loss. Keywords: Electrodermal activity, mobile EDA, Wireless, Shimmer, Android



Mental disorders are universal; they affect all countries and societies. Approximately 25% of the human population is affected during their life. This generates a substantial negative direct and indirect impact on economy and on quality of life [18]. Bipolar disorder, also known as manic depression, is in particular a widespread and severe diagnosis of a mental disorder. It is characterized by repeated episodes of elevated mood and episodes of depression. Therapists are interested in objective measures of relevant physiological and behavioral data, which ideally is recorded during the patient’s regular daily life. These measures would enable the therapist to observe patients continuously over long periods of time and to assess early warning signs or even to predict the occurrence of manic and depressive episodes. Currently,

therapists do not have any access to long-term objective measures of physiology and behavior from daily life. Being part of the European research project MONARCA, we develop and validate mobile technologies for multi-parametric, long term monitoring of physiological and behavioral information, which is potentially relevant to bipolar disorder. In the project we aim for integrating different technologies and approaches into an innovative system for management, treatment, and self-treatment of the disorder. The project addresses the goals of pervasive healthcare: making healthcare available anywhere, anytime and to anyone [1]. The so called MONARCA system [12] will consist of four sensing components: a sensor enabled smartphone, a wrist worn activity monitor, a stationary electroencephalography (EEG) system for intermittent measurements and a novel mobile “sock integrated” electrodermal activity (EDA) sensor. In this paper, we present a technical feasibility study of the mobile EDA measurement component. Physiological background. EDA measures the skin conductivity, which is proportional to the sweat secretion [5]. The total sweat secretion depends on the number of active sweat glands. Since sweat glands are exclusively innervated by the sympathetic nervous system [3], EDA is an indirect measure for psychophysiological activation. Additionally, EDA is known as a relevant indicator of the emotional state and the stress level of a person [15, 17]. Self-assessment. The major goal of EDA measurements in our project MONARCA is to correlate several physiological features with mood states and changes of bipolar patients. For the development of algorithms towards this goal as well as for later evaluation purposes, it is essential to collect ground truth mood data, which is synchronized with the objective EDA data. This ground truth data can be obtained by lab analysis of blood or saliva samples, by third-party ratings of professionals or by self-assessment questionnaires. We opted for the self-assessment questionnaires because they are unobtrusive and fit best to our mobile approach. As we want to measure frequently but at the same time do not want to impact the subject’s activities, we decided to implement Self-Assessment Manikin surveys [4]. Related work. Several studies have been conducted involving EDA measurements with persons, which suffer from mental disorders [19]. Some of the studies involve bipolar patients [10, 11]. However, the current state-of-the-art measurement devices are bulky and require trained personnel to be operated by. Thus, they are only applicable for stationary use in hospitals or laboratories, restrict the patients’ activities significantly, and cannot be used for continuous long-term measurements. A prototype of a mobile measurement device is presented in [8]. The authors attach an EDA device to one of the healthy subject’s hands and used proprietary software to log the data to a mobile phone aggregator. Although, the hand is a common used sensing location for EDA measurements, patients feel stigmatized to wear eye-catching measurement devices at this location in daily life situations. To ensure that users accept this kind of sensing in daily life, the sensors need to be comfortable and invisible. Therefore, we envision the integration of the EDA measurement electrodes into regular socks. From a physiological point of view, the feet are known to serve as a suitable measurement location for EDA [3, 9, 7]. In previous work [16] we compared EDA

measurements at the hand and the foot. Our results suggest that the foot recording location indeed is suitable for recordings in daily life even in the presence of moderate movement. In that feasibility study we used custom-designed hardware devices [14], and a notebook was required to record the data. The annotations had to be performed manually by the experiment leader. In order to obtain ethics approval to collect EDA data from bipolar patients, the measurement devices need to comply with the requirements described in Annex I of Directive 93/42/EEC (Conformité Européenne, CE certificate). For that reason we switched to a commercially available EDA sensor, which fulfills these requirements. To devise a setup for medical use with patients we implement the data recording and annotating tasks on a smartphone, which communicates wirelessly with one or more sensor devices. 1.1

Paper Scope and Contributions

In this work, we present an EDA measurement system, which meets the following requirements towards a pervasive healthcare system [1]: Mobility. Both the sensor and the logger device are small and lightweight (see Table 1). Unobtrusiveness. Electrodes are attached invisibly at the foot to possibly improve the user acceptance even for long-term recordings during their daily life [16]. Practicality. We use a smartphone as a logger unit and to transfer data immediately to therapists through data connections the phone provides out-of-the-box. Automatically or manually triggered immediate feedback to the patient based on the measured data can enhance self-treatment and monitoring systems [2]. Standardized and intuitive mood self-assessment data annotation. Ground truth information on the subject’s mental state is obtained by prompting the SelfAssessment Manikin survey [4] through the smartphone’s user interface. Certification. Deployment of out-of-the-box and CE certified sensor hardware allows an ethics-approved usage of the application with patients in clinical trials. 1.2

Paper Organization

We first provide an overview of our proposed EDA measurement system architecture and then describe each system component in detail. To confirm the technical feasibility of our system, we conducted a proof-of-concept study with a single user. We describe the experiment design and present our findings. Finally, we draw conclusions and give an outlook.


Mobile EDA Measurement and Annotation

Our proposed system comprises a sensor device and a smartphone application, which runs on an Android-based smartphone (HTC Desire S). The sensor unit conducts the EDA measurement and transmits the EDA data to the smartphone application using

the Bluetooth protocol. The smartphone application stores the data and provides feedback and instructions to the user. An overview of the system architecture is shown in Fig. 1. Its components are described in detail in the following sections.

Fig. 1. Three-layer architecture of the EDA measurement system: The sensor unit measures the EDA and transmits the EDA data to the smartphone application using the Bluetooth protocol. The smartphone application stores the data and provides feedback and instructions to the user.


Measurement Device

As a measurement system we opted for the commercially available Shimmer base unit extended with the galvanic skin response (GSR) extension module [13], because it is CE certified, which is a prerequisite for obtaining an ethical approval for clinical use of the system. Fig. 2 depicts the measurement device attached to the foot. The firmware is based on tinyOS, an open development platform [13]. Our firmware starts sampling and transmitting EDA data as soon as a Bluetooth connection has been established. A summary of the device specifications is listed in Table 1. We validated the conductance measurement values for plausibility with standard-grade resistors. The device features four measurement ranges which are switched automatically depending on the current measurement value.

EDA measurement device

Fig. 2. The Mobile EDA measurement device is attached to the user’s foot.

Table 1. Specifications of EDA measurement device [13]

Parameter Size Weight Current Draw Conductance Measurement Range Conductance Accuracy Sampling Frequency Communication CPU Accelerometer Energy Supply


Value 54x25x32 mm3 28 g 60 uA 0.2-100 uS +/- 10% 51.2 Hz Bluetooth, Class 2 8 MHz MSP430 triaxial, Freescale MMA7361 450 mAh battery

Smartphone Application

We opted for an Android-based software approach because Android is one of the most growing mobile OS [6] and it provides open source software development tools. We use a HTC Desire S smartphone as a platform for our application. The smartphone applications implements the following tasks as illustrated in Fig. 1: Connectivity. Available sensor nodes in range are identified, and then a Bluetooth connection is established to a selected sensor node to issue the data sampling. Storage. EDA data as well as system status messages are received and logged to the phone’s memory card. Visualization. The current data stream of EDA data is plotted graphically in real-time (see Fig. 3). Thus, the data quality can be observed visually during running experiments.

Fig. 3. Screenshot of the newly developed smartphone application. Here, a real-time visualization of the received data is shown.

Data loss. The current data stream is observed automatically and any data loss that exceeds a predefined threshold triggers a warning message and a vibration pattern to catch the user’s attention. Conversion. Received EDA data is decoded to raw ADC data and then converted to conductance values. Each characteristic of the four automatically switched measurement ranges is considered separately by non-linear calibration curves (4th order). Annotation. The graphical user interface (GUI) provides the possibility of setting custom data annotations, which are stored synchronized with the sensor data. Stimuli. The GUI supports the experiment leader by automatically issuing experiment instructions and stimuli to the subject. Interaction. Interaction is requested regularly from the subject by confirmation and reminder requests to assure a continuous and proper involvement during the experiment. Self-assessment. As we want to obtain regularly a ground truth measure of the user’s mental state but at the same time do not want to impact the subject’s activities, we decided to implement Self-Assessment Manikin surveys [4]. This is a graphical survey as depicted in Fig. 4, in which the user selects valence and arousal measures from two out of ten manikin. This survey does not require lengthy instructions and is completed in just a few seconds.

Fig. 4. Self-Assessment Manikin [4] survey to collect ground truth data of the subject’s mood and mood changes over time. It is implemented in the smartphone application and stored synchronized with sensor data recordings.


Proof-of-Concept Study

We conducted a proof-of-concept study with one male user to test our system and to be prepared for the upcoming large-scale studies with bipolar patients.

The experiment consists of four phases, in each of which the participant follows a predefined activity for approximately three minutes. Throughout all phases the EDA data from the sensor unit attached to the participant’s foot is continuously recorded: 1. After an adaptation phase, the first phase starts: The participant watches a documentary movie about animals in Alaska. 2. During the second phase he watches a thrilling action movie. 3. Then he plays a computer game (Minesweeper). 4. Finally, he goes for a walk. The phase transitions are instructed by the smartphone application automatically and confirmed by the user. During each phase the phone application prompts the user to hold his breath for three seconds every 20 seconds. This stimulates peaks in the EDA signal and helps to rate the quality of the measurements. These actions were also confirmed by the participant and generated event annotations in the recorded data set.

Fig. 5. The smartphone application instructs the participant to perform a breathing exercise. By confirming, the participant generates event annotations, which are synchronized with the measurement data stream.


Results and Discussion

The experiment lasted about 14 minutes in total. During this interval there were 41793 samples received which matches the expected value for a sampling rate of 51.2 Hz. There is no noticeable data loss during all phases, even when smartphone and sensor are moving. We did not experience any kind of issues with neither the sensor device nor the smartphone application. The data has been transferred to a PC. The resulting plot is shown in Fig. 6.

EDA start/stop phase start stimuli end stimuli


EDA [mS]

16 14 12 10 8 6 0


400 Time [s]



Fig. 6. EDA measurements throughout the experiment with event annotations (vertical lines): Point-dashed vertical lines mark the start and end of each of the four experiment phases. Start and end of the breathing stimuli are marked with solid and dashed vertical lines, respectively.

The breathing exercise introduces physiological stimuli, which are clearly visible by peaks in the EDA level, shortly after the breathing annotations are set. Highfrequency noise is introduced when the participant is walking; however, the EDA baseline is still unaffected.


Conclusion and Outlook

We presented a newly-developed wireless mobile EDA measurement system. It is based on commercially available and certified sensor hardware and a newly developed portable Android smartphone application which meets the requirements towards a pervasive healthcare system. The EDA sensor device is attached to the participant’s foot to enable invisible long-term measurements without restricting the patients in daily life activities. It wirelessly communicates with the smartphone using the Bluetooth protocol. We conducted a proof-of-concept study and experienced no data loss nor any kind of other technical issues with any of the components of our mobile EDA measurement system. Besides the acquired sensor data, the system supports user mood annotations with an established self-assessment procedure. From a technical and ethical point of view, the proposed setup is ready for deployment in clinical trials. From a physiological point of view, we found the measured EDA data to be plausible and in line with previous experiments.

Based on this work, we will conduct EDA experiments with bipolar patients to research the actual medical relevance of the data and the user acceptance and comfort when wearing the mobile EDA system for long-term measurements. We are planning to enhance the smartphone application with signal processing capabilities to extract relevant and meaningful high-level features from the raw data. Moreover, we will extend the communication channel between user and therapist through the smartphone application, allowing for continuous or on-demand EDA data transfer over the phone’s internet connection as well as therapist feedback back to the patient. We also plan to fuse the EDA data with data from additional modalities such as acceleration data. This could substitute manual data annotations and improve the emotion recognition by context-aware algorithms. Acknowledgements. This project is funded by the European project MONARCA in the 7th Framework Programme under contract Number: 248545. The Android application is based on code provided by Patrick F. Kugler, Digital Sports Group, Pattern Recognition Lab, University of Erlangen-Nuernberg, Germany. The authors gratefully thank all participants of the experiment and the pre-studies.

References [1] Bert Arnrich, Oscar Mayora, Jakob Bardram, and Gerhard Tröster. Pervasive healthcare - paving the way for a pervasive, user-centered and preventive healthcare model. Methods of Information in Medicine, 49:67–73, 2010. [2] Jakob E. Bardram, Mads Frost, Karoly Szanto, and Gabriela Marcu. The monarca selfassessment system: a persuasive personal monitoring system for bipolar patients. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, IHI ’12, pages 21–30, New York, NY, USA, 2012. ACM. [3] Wolfram Boucsein. Electrodermal Activity. Plenum Press, New York, 1992. [4] Margaret M. Bradley and Peter J. Lang. Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1):49 – 59, 1994. [5] Chester W. Darrow. The rationale for treating the change in galvanic skin response as a change in conductance. Psychophysiology, 1:31–38, 1964. [6] GoogleWatchBlog. Android: 850.000 Aktivierungen pro Tag, [7] Jennifer Healey. Gsr sock: A new e-textile sensor prototype. In 15th Annual International Symposium on Wearable Computers, 2011. [8] Jennifer Healey, Lama Nachman, Sushmita Subramanian, Junaith Shahabdeen, and Margaret Morris. Out of the lab and into the fray: Towards modeling emotion in everyday life. Pervasive Computing, pages 156–173, 2010. [9] Jennifer Healey and Rosalind Picard. Smartcar: detecting driver stress. In Pattern Recognition, 2000. Proceedings. 15th International Conference on, 2000. [10] William G. Iacono, David T. Lykken, Lori J. Peloquin, Ann E. Lumry, Roger H. Valentine, and Vicente B. Tuason. Electrodermal activity in euthymic unipolar and bipolar

affective disorders: A possible marker for depression. Archives of General Psychiatry, 40(5):557, 1983. [11] Hiroyasu Miwa, Shin-ichiro Sasahara, and Toshihiro Matsui. New mental health index based on physiological signals at transition between arousal and sleeping state. In Information Technology Applications in Biomedicine, 2007. ITAB 2007. 6th International Special Topic Conference on, pages 205–208. IEEE, 2007. [12] Alessandro Puiatti, Steven Mudda, Silvia Giordano, and Oscar Mayora. Smartphonecentred wearable sensors network for monitoring patients with bipolar disorder. In Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, pages 3644–3647. IEEE, 2011. [13] Shimmer Research. Shimmer - Wireless Sensor Platform for Wearable Applications, [14] Johannes Schumm, Marc Bächlin, Cornelia Setz, Bert Arnrich, Daniel Roggen, and Gerhard Tröster. Effect of movements on the electrodermal response after a startle event. Methods of Information in Medicine, 47(3):186–191, 2008. [15] Cornelia Setz, Bert Arnrich, Johannes Schumm, Roberto La Marca, Gerhard Tröster, and Ulrike Ehlert. Discriminating stress from cognitive load using a wearable eda device. IEEE Transactions on Information Technology in Biomedicine, 14(2):410–417, 2010. [16] Cornelia Setz, Franz Gravenhorst, Johannes Schumm, Bert Arnrich, and Gerhard Tröster. Towards long term monitoring of electrodermal activity in daily life. Personal and Ubiquitous Computing Journal, 2011. [17] Cornelia Setz, Johannes Schumm, Claudia Lorenz, Bert Arnrich, and Gerhard Tröster. Using ensemble classifier systems for handling missing data in emotion recognition from physiology: One step towards a practical system. In Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Conference on, pages 1 –8, 2009. [18] WHO. Official Records of the World Health Organization, volume 2, chapter Preamble to the Constitution of the World Health Organization as adopted by the International Health Conference, page 100. 1946. [19] Karl M. Williams, William G. Iacono, and Ronald A. Remick. Electrodermal activity among subtypes of depression. Biological Psychiatry, 20(2):158 – 162, 1985.

Suggest Documents