Workload Management Through Glanceable Feedback: The Role of Heart Rate Variability John Edison Muñoz Madeira-ITI, Universidade da Madeira (UMa) Funchal, Portugal
[email protected]
Fabio Pereira Madeira-ITI, Universidade da Madeira (UMa) Funchal, Portugal
[email protected]
Evangelos Karapanos Department of Communication and Internet Studies Cyprus University of Technology Limassol, Cyprus
[email protected] Abstract—The active monitoring of workload levels has been found to significantly reduce work-related stress. Heart rate and heart rate variability (HRV) measurements via photoplethysmography (PPG) sensors have shown a strong potential to accurately describe daily workload levels. However, due its complexity, HRV is commonly misunderstood and the associated measurements are rarely incorporated for workload monitoring in novel technological devices such as smartwatches and activity trackers. In this paper we explore the potential of consumer-grade smartwatches, equipped with PPG sensors, to assist in the active monitoring of workload during work hours. We develop a prototype that employs the SDNN index, a powerful HRV marker for cardiac resilience to differentiate between high and low workload levels along the work day, and presents feedback in glanceable form, by highlighting workload levels and physical activity over the past hour in 5-minutes blocks at the periphery of the smartwatch. A field study with 9 participants and 3 variations of our prototype attempts to quantify the impact of the HRV feedback over subjective and objective workload as well as users’ engagement with the smartwatch. Results showed workload levels as inferred from the PPG sensor to positively correlate with self-reported workload and HRV feedback to result to lower levels of workload as compared to a conventional activity tracker. Moreover, users engaged more frequently with the smartwatch when HRV feedback was presented, than when only physical activity feedback was provided. The results suggest that HRV as inferred from PPG sensors in wearables can effectively be used to monitor workload levels during work hours. Keywords—workload, photoplethismography (PPG), heart rate variability (HRV), smartwatch, glanceable, SDNN Index.
I. INTRODUCTION The relationship between work-related stress and the increasing of number of mental disorders has recently been emphasized by the European Foundation for the Improvement of Living and Working Conditions [1]. In 2015, the annual economic cost of work-related stress in the United States was estimated at $190bn [2]. One of the principal factors associated This work is supported by Projeto Estratégico - LA 9 - 2015-2016 and the ARDITI (Agência Regional para o Desenvolvimento da Investigação, Tecnologia e Inovação) institution.
Fig. 1. The three variations of our prototype employed in the respective conditions: (a) control condition, (b) AT, where blue bars indicate physical activity of 90-sec or more, and (c) HRV-AT, where green bars indicate appropriate workload,levels and orange bars indicate high workload levels. Each bar reflects a 5-min interval over the past hour.
with work-related stress is a mismatch between job load, the worker’s capabilities and self-control of the mental workload [3]. As a result, workload monitoring in workplaces may decrease the likelihood to develop multiple health problems such as chronic stress, anxiety disorders or mental fatigue and anger [4]. Novel approaches in mental workload monitoring include the use of information and communication technologies (ICT). In particular, with the development of more accurate and inexpensive physiological sensors, wearable computing presents a promising direction for mental workload monitoring technologies in workplaces. For instance, the Microsoft Band, Empatica and Angel wristbands all include specialized physiological sensing technology such as body temperature, electrodermal activity and heart rate (HR) sensors [5]. Using these technologies, we may unobtrusively track, visualize and prompt individuals to reflect upon their physiological human body responses [5]. For instance, stress and workload levels can be accurately monitored, thus increasing individuals’ selfawareness and capacity for self-regulation [6]. One of the most widely used physiological metrics for workload monitoring is the heart rate (HR), which can be
computed through photoplethysmographic (PPG) technology [7], which is a simple and non-invasive optical technique used to detect blood volume changes [8]. In particular, heart rate variability (HRV), a well-known physiological metric which analyzes the beat-to-beat fluctuations of the heart, assessing the regulation and resilience of cardiac activity along time [9], has been shown to be a good indicator of workload levels [10].
HRV, as measured from wearable devices, on monitoring mental workload in the everyday life of knowledge workers [20], detecting cognitive load and stress to improve performances and efficiency in human-computer interaction [21], as well as in a breath of medical applications such as clinical physiological monitoring and vascular and autonomic function assessment [8].
More than the unobtrusive sensing of individual’s physiological state, wearable-computing platforms present the promise of providing simple, glanceable feedback [12] at individuals wrists, thus leading to frequent monitoring as well as self-regulation of behaviors and emotions in the workplace. Research has shown for instance that people interact about 80120 times a day with their smartwatch [13] [14]. Even though about half of users’ interactions with smart watches are related to checking the time, presenting feedback, for instance with regard to physical activity, in the background of the watch has been found to account for as much as 80% of the variance in individuals behaviors [15]. All in all, prior research has shown the value of glanceable feedback in boosting the understanding of personal data by means of metaphorical representations [16], in supporting self-regulation in daily life through sustaining awareness around goal completion (e.g., meeting a daily goal of physical activity) [17] or supporting just-in-time behavior change through persistent and quickly-available information [12].
Typically, HRV analysis is made using Electrocardiography (ECG) signals by detecting the highest peaks (called R) and exploring multiple mathematical relations using the intervals between one peak and the other. The HRV analysis using PPG signals (PPGV) has shown high correlations with HRV parameters from ECG signals, indicating that it could be effectively used as an alternative measurement of HRV [22]. Generally, HRV analysis can be branched into two categories: time and frequency domain analysis. In the frequency domain, the ratio between low and high frequencies (LF/HF) has been discovered as the most accurate metrics to describe mental workload since it reflects the balance of autonomic nervous activities [9]. However, a frequency domain analysis is commonly computationally demanding [23], especially for wearable devices. On the other side, time domain analysis include parameters such as the SDNN (Standard Deviation of normal RR intervals) which is a global index of HRV and is used to measure temporal changes in heart activity. It is known that SDNN reflects overall fight and flight (sympathetic) responses and some rest and digest (parasympathetic) functions, thus it can reveal important heart resiliencies. Considering that changes in time intervals between adjacent heartbeats is considered as a positive adaptive skill [24], higher SDNN values indicates good workload levels representing an optimal heart regulation. Contrary, lower SDNN values are related with low levels of heartbeat adaptation which could be interpreted as high workload levels. SDNN has been used in past research to accurately describe workload levels in different contexts [25] [20] [26]. For our experiment, we used a variable called SDNN index, which is the standard deviation of all the RR intervals for each 5-min segment in long-term recordings.
Despite the growing body of work on the design of glanceable displays, information about how people engage with physiological feedback and the way in which such feedback affects their behaviors is still missing. The goal of this research is to analyze the impact glanceable feedback employing HRV data has on the subjective and objective workload users experience during a working day. We implemented an Android OS application1, using a PPG sensor to measure heart rate and infer workload levels from HRV parameters. A glanceable feedback display was implemented on the background of the smartwatch’s main screen, thus offering an abstract and persistent visualization of one’s workload levels over the past hour. We deployed the prototype with 9 users with the goal of studying how people engaged with the feedback as well as its impact, if any, on user’s experienced workload. To our knowledge, this is the first effort to untangle the role of HRV in technologies that enable individuals to reflect on their workload and regulate their behavior. We believe that a better understanding of the daily workload dynamics may increase the probability to carry out strategies to prevent high levels of stress, such as increasing the frequency pauses from work as well as sedentary activity or improve emotional self-control. II. WORKLOAD ESTIMATION FROM HRV Measures of mental workload can be classified as subjective measures, behavioral indicators, performance-based measures, or physiological metrics [18]. One of the advantages of physiological metrics is their continuous and uninterrupted recording of data in real time [19]. HRV is one of the most promising physiological analysis methods to describe mental workload. Prior research has highlighted the usefulness of 1
https://github.com/PhysioTools/HRV_Workload
III. METHODOLOGY A. Apparatus To study how users engage with HRV feedback and the impact this has on experienced workload, we implemented three variations (see fig. 1) of a prototype developed for the Android LG Watch R smartwatch. The first (HRV-PA) provided feedback both on heart rate variability as well as physical activity. The second only provided feedback on physical activity (PA), while the third reflected our control condition, which provided no feedback and acted only as a watch. Feedback was presented on the periphery of the smartwatch, along 5-minute blocks, which were colored according to individuals’ workload and physical activity levels during this 5-minute interval. Blue indicated a break from sedentary behavior (i.e., when the user was physically active for more than 90 seconds during the 5-minute interval). Green indicated appropriate workload levels (i.e., when the SDNN
index was higher than a pre-defined workload threshold). Orange indicated high workload (i.e. SDNN index being lower than the pre-defined workload threshold). A conventional hours:minutes:seconds visualization was always used in the center of the watch-face along with date information. The signal processing, interface implementation and data storage were carried out using Android Studio v1.5.1.
B. Signal Acquisition and Processing HR data was acquired from the PPG sensor of the smartwatch. The sampling frequency of the sensor is 10 Hz, however a resampling to 1 Hz was needed due to acquisition errors in the smartwatch. Data correction was further required in order to prevent signal noise due to sensor errors, motion artifacts or outlier data as follows. Firstly, HR values were filtered considering that normal HR values in healthy adults are between 60 to 100 BPM [28]. Moreover, outliers were identified using a simple rule via statistic descriptors as follows:
𝑂𝑢𝑡𝑙𝑖𝑒𝑟𝑠 = 𝑎𝑏𝑠 𝑋 − 𝑋 > (𝑧𝐹𝑎𝑐𝑡𝑜𝑟 ∗ 𝑋_𝑆𝐷) (1) where X are the HR measurements in 5 minutes, is the mean value of X, zFactor is the triggering variable and X_SD the standard deviation of X. We used a zFactor of 2 meaning that every single HR data which differs by more than 2 standard deviations from the previous one will be discarded. Finally we used a time window of 5 minutes to compute the SDNN index which was computed using the filtered signal. Physical activity was measured through Android’s native activity recognition API using the 3-axial accelerometer of the smartwatch. C. Workload Calibration Since there is high intersubject variability in HRV measurements [28], we designed a method to calibrate the workload estimation based on the SDNN index measurements for the HRV-AT watch-face. For that, we used the average of the SDNN index from two different sources. First, we used the protocol known as the N-back task to induce high workload levels during the first minutes of the control condition. The Nback is a well-known task to induce workload by playing a memory load game [29]. We used an online version of the Nback task (cognitivefun.net), where users have to observe a sequence of pictures on screen as they are displayed one by one. Users had to indicate repeated pictures over the prior N trials. We implemented two N-back tests: 2-back and immediately after 3-back. We used a protocol of 133 pictures appearing 0.5s with an inter-stimulus interval of 1.5s. A total of 10 minutes (5 minutes for each N-back task) were needed to complete the tasks and the HR was recorded.
The second source was based on self-reported workload. Users were asked to write down the 3 highest workload moments along the day. Finally, we computed a workload threshold using an average of both: SDNN index values during the 2-back and the 3-back tasks jointly with the SDNN index from the workload levels reported for the users during the day. D. Questionnaires: We employed four questionnaires to measure users’ perceived sedentarism levels, perceived workload, stage of behavioral change (in terms of physical activity) and watchface prototype experience. Perceived sedentarism was measured with the short version of the International Physical Activity Questionnaire (IPAQ) [30] which asks for the time spent in the last 7 days doing three specific types of activity: walking, moderate-intensity and vigorous-intensity physical activities. We also used a question with seven mutually exclusive options which described five stages of change named precontemplation, contemplation, decision, action and maintenance [31]. Perceived workload was measured with the Subjective Mental Effort Questionnaire (SMEQ) which consists of a single scale with nine labels from “Not at all hard to do” to “tremendously hard to do”, which relate to a numerical scale that runs from 0 to 150 (exceptional amount of effort) [32]. The SMEQ has been used before to effectively describe workload levels in different tasks with high levels of granularity [33]. Finally, we designed a questionnaire asking participants to rate on a 5-point likert scale their experience with the prototypes, and in particular a) how understandable the prototype was at a glance, b) how motivated they were to take a pause from work, c) the perceived usefulness of the prototype, d) its capacity to lead them to re-engage, e) their level of interest in the prototype, f) use again or recommend. E. Participants Nine adults (4 men, 5 women, ages 30.5 ± 4.8 years) participated in the study. All participants had no recent heart failures or blood pressure complications. 8 of the 9 participants had no past experience with smart watches. All users were knowledge workers from a local research institute. All participants gave informed consent prior to participation. F. Protocol Participants were invited to wear the LG Watch R smartwatch for three days (one condition per day). The control condition was always assigned to the first day, while the order of HRV-PA and PA was counterbalanced. Around 9:00 am of the first day we carried out the N-back tasks with users and gave them the smartwatch with the control watch-face. Around 5:00 pm, the smartwatch was removed and participants responded to the SMEQ, the question of the stages of change and the IPAQ questionnaires. In the remaining two days, we used the HRV-AT or the AT watch-faces interchangeably. For the HRV-AT condition, we used the workload threshold computed in order to color the
workload blocks in green or orange. Finally, in the HRV-AT and the AT conditions, we also requested for the SMEQ, the stage of change and the custom questionnaire at the end of the day. G. Data Collection To facilitate the data storage and management, we implemented an APP running in the background on a mobile phone with Android O.S., which was receiving data from the smartwatch, in particular regarding the SDNN index values, the physical activity and users’ engagement with the watch. Data were stored in the cloud to guarantee data safety. All data from one user were discarded due to server failure. IV. RESULTS A. Coherence between measured and reported workload In order to express SDNN index values in terms of workload levels, the data was normalized and inverted. The normalization was done around the highest SDNN index value of the complete dataset. Participants had lower levels of workload during the HRV-AT condition (M=0.24, SD=0.08) as compared with the AT condition (M=0.28, SD=0.10) but not with the control (M=0.14, SD=0.10). A repeated measures ANOVA revealed significant differences in the workload levels measured by the SDNN index, F(2.0, 14.0) = 3.79, p < .05. Figure 2 shows the measured workload for each condition. Moreover, a contrast analysis revealed that measured workload levels were significantly lower during control condition once compared with AT condition (p < .05) but no compared with HRV-AT condition.
Fig. 3. Perceived workload of users measured by the SMEQ.
We found considerable but not statistically significant correlations between the measured (SDNN index) and the perceived (SMEQ) workload levels for the control (r = 0.50, p = 0.19), AT (r = 0.45, p = 0.25) and HRV-AT (r = 0.24, p = 0.55) conditions. Although there is no statistical difference in the SMEQ data and the correlations mainly due to users’ variability and the small sample size, these findings provide grounds for further inquiry, given the possible correspondence between the workload measured by the SDNN index and the perceived workload of users gathered through the SMEQ. B. Influence of HRV-glanceable information on users’ engagement with the watch. An analysis of variance revealed a significant main effect of the type of watch-face (condition) on the number of times users glanced at the watch over the course of the day, F(2.0, 14.0) = 4.61. Users interacted with the watch more frequently in the HRV-AT condition (M=77, SD=30) where both HRV and physical activity than in the AT condition where feedback was provided only around individuals’ physical activity (M=58, SD=20, p ≤ 0.05), and the Control condition (M=49, SD=18, p ≤ 0.05) where no feedback was provided.
Fig. 2. Measured workload for each condition. The SDNN index values were normalized and inverted to represent workload levels.
Data from SMEQ questionnaire revealed a similar behavior along the three conditions (see figure 3). People reported lower levels of perceived workload in the HRV-AT condition (M=38.3, SD=22.6) once compared with the AT condition (M=44.4, SD=22.1) but not compared with the control (M=27.7, SD=14.7). Fig. 4. Number of glances represented by condition.
C. Questionnaires Users’ characterization with IPAQ reveals that 55% (5 out of 9) of the participants had low physical activity, 22 % had a moderate level and 22% a vigorous level. On average, users spent more than 10 hours in front of a screen per day. Furthermore, answers for the stage of change questionnaire revealed that, on average, participants were in the decision stage during the control condition (median = 3, SD=1.4) but after interacting with the AT watch-face (scoring = 4, SD=1.4) and the HRV-AT watch-faces, they moved to the action stage (median = 4, SD=1.4). This finding suggests that the use of exercise-oriented glanceable feedback can help people to become aware of their physical activity during working hours and attempt to engage with physical activity more frequently (e.g. short breaks, restrooms breaks, etc.). Additionally, most of the participants reported that some of the breaks carried out along the day were consciously produced via the observed feedback related with the workload levels in the HRV-AT condition, thus facilitating the cardiac self-regulation for an optimal workload control. Finally, the results from the custom questionnaire showed higher scores in all the domains for the HRV-AT watch-face once compared with the AT. Participants reported significantly higher scores in their ability to understand the feedback at a glance for the HRV-AT prototype (M=4.5, SD=0.5) than for the AT prototype (M=3.3, SD=1.0), t(8)=4.4, p
