EEG-based Emotion, Mental Workload and Stress ...

CogniMeter: EEG-based Emotion, Mental Workload and Stress Visual Monitoring Xiyuan Hou, Yisi Liu, Olga Sourina and Wolfgang Mueller-Wittig Fraunhofer IDM@NTU Nanyang Technological University Singapore houxy, liuys, eosourina, [email protected]

Abstract—Real-time EEG (Electroencephalogram)-based user’s emotion, mental workload and stress monitoring is a new direction in research and development of human-machine interfaces. It has attracted recently more attention from the research community and industry as wireless portable EEG devices became easily available on the market. EEG-based technology has been applied in anesthesiology, psychology, serious games or even in marketing. In this work, we describe available real-time algorithms of emotion recognition, mental workload, and stress recognition from EEG and propose a novel interface CogniMeter for the user’s mental state visual monitoring. The system can be used in real time to assess human current emotions, levels of mental workload and stress. Currently, it is applied to monitor the user’s emotional state, mental workload and stress in simulation scenarios or used as a tool to assess the subject’s mental state in human factor study experiments. Keywords—visual interface; EEG; stress; mental workload; emotions

I. INTRODUCTION As more and more wearable devices (Apple Watch, Fitbit, and Jowbone) are available in commercial market, it becomes popular to use wearable devices and mobile phones to monitor our daily physiological states based on number of walking steps, heart rate, calories consumption, and sleep pattern. All signals of these wearable devices come from the muscle activities or movement of the user. Thus, these devices have some limitations in monitoring people’s mental states. A brain computer interface is more suitable for such analysis since Electroencephalogram (EEG) signals are directly recorded from the brain. A number of EEG-based methods and the corresponding applications are designed and implemented in order to recognize the user’s mental states [1-3]. However, most of the current works neither process EEG data in real time nor provide intuitive visualization tools for experiment analysis and decision making. The EEG-based visualization of mental states has a potential to help researchers in their experiments. For offline data analyses, it can enable researchers to evaluate and improve human factor models based on the experiment results. For real-time data analyses, it can enable researchers to directly monitor the relationship between task variables and human mental states. In this paper, we propose a novel EEG-based CogniMeter system that integrates recognition algorithms to monitor emotion, mental workload and stress in real-time simultaneously. This paper is structured as follows: Section II introduces related works such as EEG visualizations tools and EEG-

based emotion, mental workload and stress recognition algorithms. Section III introduces the methodology used in the implemented algorithms and structure of the proposed visualization system. Section IV presents the details of the visualization tools to monitor the emotions and the meters to monitor the mental workload and stress from EEG. Section V gives the conclusion. II. RELATED WORK A. Visualization Tools There is a number of tools for analysis and visualization of EEG data such as EEGLab [4], Brainstorm [5], and ELAN [6]. The systems provide interactive graphic user interfaces which enable the users to interactively process high-density EEG data. For intuitive observation of activities in the brain, the amplitudes of EEG signals can be mapped to a 2-D or 3D model of the scalp according to the EEG channel positions. These visualization tools are able to process EEG data epochs using spectral analysis or independent component analysis [4]. In [7-8], a blobby model is implemented for the EEG signals visualization on a 3D head model. Because EEG signals are non-stationary and highdimensional, machine learning methods can be used for realtime analysis of brain activities with specific tasks [9]. In this way, the visualization tools can provide further interpretation of mental states from EEG data instead of showing only the amplitude of the signals. In [10], an EEG-based workload gauge is implemented and the workload level is monitored when the subjects are doing cognitive and operational Air Traffic Controller’s (ATC) tasks. It is critical to reliably measure the mental states and performance of the controllers/pilots when the mode of automation is changed or new tasks appears. During the simulation of ATC task, the workload gauge is updated with the recognized level of workload (three workload levels). Other gauges depict dynamic changes in EEG signals and show spectrum powers in the form of brain maps simultaneously. In [9], an EEG-based mental text entry system Hex-oSpell is proposed. By imagining left/right hand movement, the user can generate different brain patterns. These patterns are used to control visual text input. The visualization tools include a text dial, an arrow, and a bar. The orientation of the arrow and the length of the visual bar are controlled by the states of the human brain.

B. Emotion Emotions can be defined from a dimensional perspective where arousal, valence, and dominance dimensions are considered [11]. In the dimensional model, the arousal dimension ranges from not aroused to excite, the valence dimension ranges from negative to positive, and the dominance dimension ranges from being controlled to being in control. The dimensional model is preferable for emotion recognition because it allows locate discrete emotions in the dimensional space. Even the feeling that cannot be labeled with a certain word can be located in the dimensional model [12, 13]. Emotions can be induced by different kinds of stimuli such as audio, visual and combined ones [14]. Different algorithms have been proposed for EEG-based emotion recognition. [3] extracted power features from EEG data and used Support Vector Machine (SVM) as a classifier. 82.37% accuracy for distinguishing of four emotions was achieved by using 32 channels. Short Time Fourier Transform (STFT) as a feature extraction method and SVM as a classifier were applied in [15] and a mean accuracy of 62.07% was obtained with 16 channels. However, all these researches are off-line emotion recognition. Additionally, as the EEG signal is nonlinear and chaotic, traditional features may not be able to capture the nonlinear property of EEG. The fractal dimension (FD) can reflect the changes of the EEG signal during different mental tasks in real-time [1618]. In [19], FD is used in real-time EEG-based emotion recognition. C. Mental Workload Mental workload is described as a noticeable relationship between the human cognitive capacity and an effort required to process a particular function [20]. There are mainly three broad categories for workload definition including physiological workload, subjective workload, and cognitive workload [21]. In this research, we are interested in cognitive workload which indicates the capability of a person to complete a task with some amount of the mental effort. The performance in the task ascertains the cognitive workload [22]. In [9], mental workload is evaluated in online EEG monitoring during the security surveillance task. Comparing the mental workload index with the error rate for the subjects, the correlation coefficient is approximately 0.7, which indicates that when the workload increases people tend to make more errors. The significant correlation between workload and theta band power has been proved in [23-25]. In experiment [23], it is shown that the EEG theta band power increases when the workload is induced by mental arithmetic task. In [24], the driver’s mental workload is significantly correlates with both theta and alpha power. In different driving tasks, the frontal theta activity shows significant increases when working memory load increases. In another experiment studying the workload and fatigue in aircraft pilots [25], increased EEG theta band power and decreased alpha band power are observed in high mental workload comparing with low mental workload. Additionally, in [25] it is shown that when the pilots have high mental workload and mental fatigue, their EEG theta band power as well as the delta and alpha bands power increases.

D. Stress Stress is a human state which can be caused by a number of reasons, including high mental workload, emotions, or environmental influences [26]. The stress can be measured and assessed from physiological variables including EEG [1, 2, 27-29], blood pressure [30], heart rate variability [28, 31], skin conductance level [32], and electromyography [33]. EEG can be used to detect human stress levels. In work [30], the experiment shows that the stress is positively correlated with beta EEG power at the anterior temporal lobe. In [2], higher order spectra features are used for stress recognition. The SVM with RBF kernel is chosen as a classifier and the accuracy calculated with 5-fold cross validation for recognition of two stress states is 79.2%. In [27], the features such as Gaussian mixtures of EEG spectrogram, fractal dimension and magnitude square coherence estimation are used in stress recognition algorithm. The classification of two levels of mental stress is done by k-Nearest Neighbor (kNN) and SVM classifiers, and the best accuracy is 90%. However, neither [2] nor [27] uses standard stressor to induce stress in the experiments. In [1], a Stroop colour-word test is used to induce stress. The discrete cosine transform is applied to reduce the data size and extract features from the frequency domain. Classification is implemented with artificial neural network, linear discriminant analysis and kNN. The best classification result for two stress states is 72% with k-NN. In [29], the band power of theta, alpha, and beta are used as features in logistic regression and are fed into the k-NN classifier. The results show a median accuracy of 73.96% for the recognition of relaxed and stressed states. III. METHODOLOGY In our real-time EEG-based mental states monitoring system, the emotion is recognized using machine learning method, the workload is identified by theta power, and stress is inferred from a combination of the recognized emotion and workload. A. EEG Device The Emotiv Headset [34] is used to capture the users’ EEG signals wirelessly with the USB receiver. It has 14 channels locating at AF3, F7, F3, FC5, T7, P7, O1, O2, P8, T8, FC6, F4, F8, and AF4 as shown in Fig. 1. B. Feature Extraction The power spectrum feature is used in the real-time workload level recognition while the FD and statistical features are used in real-time EEG-based emotional recognition [19]. All features are extracted using a 4 seconds sliding window with 3 seconds overlapping. 1) Power Feature Power spectrum is one of the most commonly used features for EEG-based mental states recognition. The power spectrum over a time interval is obtained by the fast fourier transform. The EEG power spectrum is subdivided into bandwidths known as delta (0.5-4 Hz), theta (4-8 Hz), alpha (8-12 Hz), beta (12-30 Hz), and gamma (above 30 Hz). As

introduced in Section II C, the theta band power has been proved to be positively correlated with mental workload in [23-25]. In our study, the average of power spectrum density of theta band is used in the mental workload recognition. The composed feature vector is FVtheta = Ptheta .

Then the Fractal Dimension dim H can be obtained by logarithmic plotting between different t (ranging from 1 to tmax ) and its associated L (t ) [35]. dim H =

ln L (t )

. − ln t Thus, the FD feature vector composed is

(4)

FVFD (Higuchi) = [dim H ] .

(5)

3) Statistical Feature Statistical features are simple and widely used in EEGbased brain state recognition. For example, they are used in EEG-based emotion recognition algorithms [38, 39]. Six statistical features including mean μ , standard deviation σ , mean of absolute values of the first differences δ , mean of absolute values of the first differences of X

X

X

normalized signals δ , mean of absolute values of the second differences γ , and mean of the second differences of

Fig. 1. Location of 14 electrodes of Emotiv EEG device.

X

X

2) Fractal Dimension Feature FD measures the complexity and irregularity of time series [35]. It can be used as an index for characterizing the complexities of EEG signals. For a regular signal, the fractal dimension value is low. If the signal becomes irregular, the fractal dimension value increases accordingly. Wang et al. [36] proposed to use Higuchi fractal dimension to recognize different arithmetic mental tasks from EEG. It is also used in EEG-based serious games to identify attention level [37]. In this paper, the Higuchi algorithm [35] is used to calculate FD feature from EEG data in emotion recognition algorithm proposed in [19]. The idea of Higuchi algorithm is as follows. Let X (1) , X ( 2 ) ,… , X ( N ) be a finite set of time series samples. Then, the newly constructed time series is

⎛ ⎡ N − m ⎤ ⎞ . (1) ⋅t⎟ ⎣ t ⎥⎦ ⎠ ⎝ where m = 1, 2,..., t is the initial time and t is the interval time [35]. X tm : X ( m ) , X ( m + t ) , … , X ⎜ m + ⎢

t sets of Lm ( t ) are calculated by ⎧ ⎡ N−m⎤ ⎫ ⎞ ⎪⎪⎛ ⎢⎣ t ⎥⎦ ⎪⎪ − N 1 Lm ( t ) = ⎨⎜ ∑ X ( m + it ) − X ( m + ( i −1) ⋅ t ) ⎟ / t . (2) ⎜ ⎟⎟ ⎡ N − m⎤ ⎬ ⎪⎜ i=1 ⎪ ⋅ t ⎠⎢ ⎪⎩⎝ ⎣ t ⎥⎦ ⎪⎭

L ( t ) denotes the average value of Lm ( t ) , and one relationship exists L ( t ) ∝ t − dim . H

(3)

the normalized signals γ are extracted from EEG for emotion recognition. The composed feature vector of statistical features is FVstatistical = [ μ , σ , δ , δ , γ , γ ] . X

X

X

X

X

X

X

C. Data Processing 1) Emotion Recognition As mentioned above, the emotion recognition algorithm proposed in work [19] is used in this paper. By combining the FD and statistical as the feature and using SVM as the classifier, it can recognize up to 8 emotions with the accuracy of 52.67%. If only positive and negative emotions are needed to be recognized, the best accuracy can be 91.07% [39]. As the proposed emotion recognition algorithm is a subject-dependent one, before real-time emotion recognition, a classifier model has to be trained. Different types of stimuli can be used to evoke certain emotions such as sound clips from International Affective Digital Sounds database [40] or pictures from International Affective Picture System database [41]. A pipeline of data processing in the offline emotion training session is shown in Fig. 2. The raw EEG data recorded from four electrodes (AF3, F4, FC5 and F7) are labeled with different emotions by the selfassessment questionnaires. Then, the EEG data stream is filtered by a 2-42 Hz bandpass filter. The statistical and FD features are extracted from the filtered EEG data using a sliding window with the size of 512 (4 seconds) with 384 samples (3 seconds) overlapping. The feature vector FV for emotion classification is defined as follows: FV = [FV1, FV2, FV3, FV4] ’.

(6)

where 1 denotes FC5 channel, 2 denotes F4 channel, 3 denotes F7 channel, 4 denotes AF3 channel, and FVi is the feature vector per channel. Here, the FVi is composed by the combinations of FD and statistical features and denoted as

FVcombination_1. Normalization is applied to the FD and statistical features across the four channels.

FVconbination _ 1 = [ μ X , σ X , δ X , δ X , γ X , γ X , dimH ] ’.

(7)

Then, the extracted features are fed into the SVM classifier to train a classifier model for a particular subject. The polynomial kernel is used with penalty parameter C = 10 , degree d = 3 , gamma g = 1 , coefficient r = 1 . The trained model is saved and used in the real-time emotion recognition phase.

indicates the most excited state. For valence ratings, 1 indicates the most negative state and 9 indicates the most positive state. For dominance ratings, 1 indicates the feeling of fully being controlled and 9 indicates the feeling of full control. For mental workload ratings, 1 indicates the lowest workload and 9 indicates the highest workload. For stress level ratings, 1 indicates the lowest stress level (relaxed) and 9 indicates the highest stress level. The correlation coefficient is calculated among all ratings. As it is shown in Table I, the most significant correlation is found between stress and mental workload (positive correlation), and between stress and valence (negative correlation), which means when subjects have higher mental workload and more negative emotion, they tend to experience more stress. Thus in this work, we use the combinations of valence and mental workload to recognize stress. TABLE I. CORRELATION COEFFICIENT BETWEEN STRESS AND AROUSAL, VALENCE, DOMINANCE, AND MENTAL WORKLOAD. Stress

Fig. 2. The pipeline of offline classifier training.

2) Mental Workload Recognition As reviewed in Section II C, in work [23-25], mental workload is proved to be significantly correlated with theta and alpha band power. Especially, theta band power increases in high memory load and high mental workload. Thus, we use theta band power calculated from channel P8 for real-time mental workload monitoring. The theta band power is calculated from a sliding window with the size of 512 (4 seconds) with 384 samples (3 seconds) overlapping. For each subject, we set two thresholds to discriminate three levels of workload. The FVtheta is compared with the predefined thresholds and then the current level of mental workload is recognized. 3) Stress Recognition In order to recognize stress level, we carried out an experiment to study the correlation between workload, emotion and stress. 7 subjects were recruited and their age ranges from 21 to 28. All subjects reported that they had no history of mental diseases. The experiment consisted of 5 segments and the Stroop Color Word Test [42] is adapted to elicit different levels of stress. In segments 1 and 2, the subjects just relaxed and looked at the screen. In segment 3, the Stroop test was given, and the subjects were asked to identify the ink color of the words on the computer screen, given the words that match with its font color. In segment 4 and 5, the font color of the word and the word differs. The subjects were told to identify the ink color of the word. However, time limitation was added in segment 5 which is supposed to elicit a higher level of stress than in segment 4. Upon completion of each segment, a questionnaire was given to the subjects to rate arousal, valence, dominance, workload and stress. Each rating has a 1-9 scale for the subjects to select. For arousal ratings, 1 indicates the calmest state and 9

Arousal Valence Dominance Mental Workload TABLE II.

r 0.2735 -0.5666 -0.3835 0.9335

p 0.1119 0.0004 0.0230