Subject-independent human activity recognition using ...

172

Int. J. Ad Hoc and Ubiquitous Computing, Vol. 20, No. 3, 2015

Subject-independent human activity recognition using Smartphone accelerometer with cloud support Muhammad Arshad Awan and Zheng Guangbin Department of Computer Science, Yonsei University, Seoul 120-749, South Korea Email: [email protected] Email: [email protected]

Hie-Cheol Kim Department of Computer and Communication Engineering, Daegu University, 712-714, South Korea Email: [email protected]

Shin-Dug Kim* Department of Computer Science, Yonsei University, Seoul 120-749, South Korea Email: [email protected] *Corresponding author Abstract: Human activity recognition is an important task in providing contextual user information. In this study, we present a methodology to achieve human activity recognition using a Smartphone accelerometer independent of a subject compared with other user-dependent solutions. The proposed system is composed of four components; a data collector, a data storage cloud, a workstation module and an activity recogniser. The data collector extracts a unique set of defined features from raw data and sends them to the data storage cloud. The workstation module receives the training data from the cloud and generates classification models. The activity recogniser determines the user’s current activity based on up-to-date available classifier from the cloud. A prototype is implemented on an android platform to recognise a set of basic daily living activities by placing the Smartphone in different positions to the user and evaluated for offline and online testing to show the scalability and effectiveness. Keywords: human activity recognition; subject-independent; mobile cloud computing; context-awareness; ubiquitous computing. Reference to this paper should be made as follows: Awan, M.A., Guangbin, Z., Kim, H-C. and Kim, S-D. (2015) ‘Subject-independent human activity recognition using Smartphone accelerometer with cloud support’, Int. J. Ad Hoc and Ubiquitous Computing, Vol. 20, No. 3, pp.172–185. Biographical notes: Muhammad Arshad Awan received his BS in Mathematics and Physics as major from the University of the Punjab, Lahore, Pakistan and the MSc and MS in Computer Science from Quaid-i-Azam University, Islamabad, Pakistan and Allama Iqbal Open University, Islamabad, Pakistan, respectively. He is currently pursuing the PhD in Computer Science at Yonsei University, Seoul, South Korea. His research interests include human activity recognition in WSN, ubiquitous computing, mobile and cloud computing. Zheng Guangbin received his BS in Logistics Engineering from Shandong University, China in 2005. He had worked for Shanghai Wooruil Spine Hospital as the Information Department Director from 2007 to 2012. He is currently pursuing the Master’s degree in Computer Science Department at Yonsei University, Seoul, South Korea. His research interests include context engine, context aware applications and ubiquitous computing. Hie-Cheol Kim received his BS in Electronic Engineering from Yousei University, Korea and his MS and PhD in Computer Engineering from the University of Southern California in 1991 and 1996, respectively. From 1983 to 1988, he was a Researcher in Samsung Electronics, Ltd., Korea.

Copyright © 2015 Inderscience Enterprises Ltd.

Subject-independent human activity recognition using Smartphone accelerometer with cloud support

173

Since 1997, he has been on a Faculty of the Department of Computer and Communication Engineering, Daegu University, Korea, where he is currently a Professor. His current research interests include wireless sensor networks and embedded operation systems. Shin-Dug Kim received his BS in Electronic Engineering from Yonsei University, Seoul, South Korea, in 1982 and the MS in Electrical Engineering from the University of Oklahoma, Norman, in 1987. In 1991, he received his PhD from the School of Computer and Electrical Engineering at Purdue University, West Lafayette, IN, USA. He is currently a Professor of Computer Science at Yonsei University. His research interests include advanced computer architectures, intelligent cache/memory systems, embedded systems and ubiquitous computing. He is a member of the IEEE Computer Society.

1

Introduction

Human activity recognition has become a dynamic field of research over the last few years. Context awareness is an essential part of ubiquitous and pervasive computing, and activity recognition is the key component in context management, thus activity recognition is an important and challenging field of research that can support various new applications. With advances in wireless sensor network technology, the task of context awareness has become achievable to some extent, but requires further research to design and develop more generic models that fit into many applications. With advances in wireless sensors and sensor network, pervasive computing and artificial intelligence have contributed significantly to overcome the challenges of daily life (Cook et al., 2009). By using these technologies, we can design and develop systems that meet users’ requirements according to their needs and current situation. Different sensing technologies are currently used to automatically collect and classify user activities for various application domains by utilising sensory data. The target areas include: medical applications (monitoring and diagnosis), home monitoring and assisted living (child and elderly care, monitoring and emergency help, and assistance with cognitive disorders) and sports and leisure applications (daily sport activities, martial arts) (Avci et al., 2010). Previously, human activities were first recognised using a vision-based approach and then inertial sensing techniques. Recently, Smartphones equipped with numerous sensors, including an accelerometer, GPS and microphone can detect human activities. In the last decade, Smartphones have become very common and convenient devices with high usability. Currently, more and more sensors are available, especially those related to the ambient environment and physiological activity. Furthermore, the processing capacity of current Smartphones is equal to PC’s and can perform most of the tasks previously undertaken on PC, but Smartphones maintain easy portability. All of these including popularity, processing capacity, high usability and multiple network compatibility make Smartphones perfect devices for determining user behaviour or actions. Because Smartphones have become part of everyday life, their effective use can improve the quality of our life. By using the accelerometer built into the Smartphone, individuals can keep track of their activities during the day

such as walking, jogging or cycling. This information can help users to change their habits, for example, walking or cycling to school instead of taking the bus (Kmiecik, 2013). The work involved in recognising human activities is a combination of signal processing and classification. When a user performs a physical activity such as walking or jogging while using the Smartphone, the horizontal and vertical readings from the accelerometer can be recorded, known as gathering sampled data. Then, a unique set of features can be extracted, and lastly, learning algorithms are applied for classification processing. Once the known patterns are stored, the application attempts to match the new data to the known patterns to identify the current activity. Human activity recognition using a Smartphone’s accelerometer requires training data to build customised classification models that can be used to identify user activities involved in the training process, which limits the implementation scope of these systems. Building classification models that can be used to identify user activities regardless of training and test data are challenging due to a large variability in accelerometer signals and because of various factors, including movement patterns of individuals that are different due to age, gender, social factors, etc. In addition, the position/orientation of the phone while performing activities has a major impact on the classification process. To overcome these issues, we used cloud features for large training and test data storage that can be used to build classification models on an incremental basis and reduce the overhead of obtaining large training data for classification modelling that can be fitted to many users irrespective of their involvement in the training phase, known as subject-independent activity recognition. The issue of different phone positions was addressed by using various common positions while gathering data and made them part of the classification model to improve recognition accuracy. The proposed system was implemented on an android platform that included the support of cloud data storage and a workstation module to generate classification models. A set of 11 basic activities of daily living (ADL); lying down, sitting, standing, walking, jogging, ascending stairs, descending stairs, ascending in an elevator, descending in an elevator, ascending on an escalator and descending on an escalator, were recognised independent of the subject using our system by placing the phone into different positions while performing the activities. We performed

174

M.A. Awan et al.

a comparative study on a selected set of classification algorithms, extracted features and Smartphone positions of the user to determine their impact on accuracy of activity recognition. Finally, we conducted an offline and online evaluation of the system to recognise the activities of different subjects and achieved an average accuracy above 90% in recognising all the activities independent of the subject. The results proved the scalability and effectiveness of our proposed methodology. The remainder of the paper is organised as follows: Section 2 discusses some related work. The design and features of the proposed methodology are presented in Section 3. In Section 4, the implementation detail of the proposed system is described. Experimentation and evaluation of the system are discussed in Section 5. Finally, we conclude our paper with some further remarks in Section 6.

2

Related work

Activity recognition as a basic component in context acquisition has become an important research topic by providing personalised support in many general applications, especially in healthcare. The data obtained from sensors is manipulated through machine learning algorithms to infer the user’s current activity. The activity recognition system can be categorised into two broad categories (Guan et al., 2011): video sensor-based activity recognition and physical sensor-based activity recognition. Physical sensor-based activity recognition can be further divided into the wearable sensor-based activity recognition and object usage-based activity recognition. According to this categorisation, the Smartphone embedded sensors can fit into wearable sensors’ group and similarly the approach may be termed as wearable sensor-based activity recognition, which was examined in the present study. A number of studies have been conducted to recognise human activities in various application fields. A methodology (Jafari et al., 2007) was proposed to identify a patient’s fall, among other movement activities using a triaxial accelerometer. Postural orientation techniques (Karantonis et al., 2006) were used in the study to identify the sit-stand, stand-sit, lie-stand and stand-lie movements, and accuracy up to 84% was achieved using neural network and k-nearest neighbour algorithms. Similarly, a study was conducted to distinguish between fall and ADL using a triaxial accelerometer (Bourke et al., 2007). The use of sensors, i.e., wearable vs. deployed in the environment and less vs. more sensors, for better recognition accuracy remains controversial in the research community. Different approaches have their own advantages and limitations depending on the type of applications. Bao and Intille (2004) developed and evaluated algorithms to determine the daily human activities using five bi-axial accelerometers, which can be worn on different areas of the subject. They support the use of more sensors on different areas of the body for better results and achieve an average accuracy of 84% in determining

different activities. Another work on identifying user activities using cell phone accelerometers (Kwapisz et al., 2010) under the Wireless Sensor Data Mining (WISDM) project was conducted to identify daily activities including sitting, standing, walking, jogging, ascending stairs and descending stairs. Supervised learning approach was used by obtaining data from 29 subjects and accuracy above 90% was achieved. The authors in the study (Kwapisz et al., 2010) claimed that a better accuracy can be achieved even with less number of sensors by accumulating large data contrary to a previous study (Bao and Intille, 2004). The findings of this approach are considered as a parameter in achieving subject-independent activity recognition. Various surveys, including the survey of Lara and Labrador (2013) on human activity recognition using wearable sensors have highlighted the subject-independent activity recognition as a major issue in this field, however, to date there is no optimal solution for subject-independent recognition of human activities using wearable sensors. Several studies have attempted to determine the position- or orientation-independent activity recognition by creating classification models of the data while keeping the Smartphone in various areas of the body and with different orientations. Khan et al. (2010) presented a hierarchical classification approach by distinguishing the data into three different categories, including resting, upper body movement and lower body movement. They achieved position independence by placing the accelerometer in different areas of the body, i.e., chest pocket, front left trouser pocket (TP), front right TP, rear TP and inner jacket pocket, for data collection. Another study (Thiemjarus, 2010) was conducted to achieve device orientation independence by keeping the Smartphone at a fixed waist-mounted position and the data collected by changing all the possible orientations of the phone, e.g., horizontal and vertical orientations. Similarly, a study was conducted by comparing two classification algorithms, i.e., k-nearest neighbours and quadratic discriminant analysis in human activity recognition process (Siirtola and Röning, 2012). The authors conducted the entire activity recognition process, from pre-processing to classification, on two platforms, Symbian and Android, to show platform independence in activity recognition. They also created an offline and online testing on different subjects to prove subject-independent activity recognition and claimed that activity recognition is a lightweight process that can be easily performed on Smartphones. Since the authors used only two classification algorithms and only the signal magnitude as a feature vector, determining whether the same results can be achieved by using different classification algorithms and different feature vectors is difficult. However, from these studies and a work on human activity recognition (Awan et al., 2013), we attempted to achieve subject-independent activity recognition. The large amount of data for classification purpose has been proven to improve the accuracy of activity recognition (Kwapisz et al., 2010), and the cloud can be a good choice for data storage when compared with previous conventional approaches (Kmiecik,

Subject-independent human activity recognition using Smartphone accelerometer with cloud support 2013), thus we attempted to obtain the accumulative advantages of these approaches and devise a new methodology to achieve subject-independent human activity recognition.

3

Methodology

Recognising human activity only with a Smartphone accelerometer is a challenging task due to variability in signals caused by the different positioning of the Smartphone to the user and sensitivity to environmental factors such as temperature. By using signal processing and classification procedures, human physical activities can be recognised accurately up-to some extent and customised sets of models can be developed to recognise activities of a particular user by applying the supervised learning approach. However, identifying the activities of a user, which is new or strange to the system, is more challenging and difficult. Experimentation (Awan et al., 2013) and literature surveys, including (Kwapisz et al., 2010) have shown that a large amount of training data can lead to accuracy in the activity recognition process. We used cloud features for large training data storage and classification models to achieve more accurate activity recognition, which in turn proved to be a scalable solution for subject-independent human activity recognition. A brief description of the proposed methodology, including the components involved in achieving the subject-independent activity recognition is shown in Figure 1. The proposed solution was composed of four components; a data collector, a data storage cloud, a workstation/server module and an activity recogniser. The functioning of these components is explained in the following subsections. Figure 1

The proposed methodology with necessary components

3.1 Data collector Although we adopted a recursive approach to obtain training data for developing classification models, the test data can be reused as training data to develop an incremental classifier to improve accuracy and achieve subject independence. However, an initial set of training

175

data for first-time classification or using volunteers for gathering large amount of training data were necessary. This mobile-based component was used to obtain raw data from the Smartphone accelerometer, label data according to the activity performed by the user, and extract necessary predefined features to be used by the machine learning algorithms for creating classification models. The position of the Smartphone to the user while performing the activity plays an important role in the activity recognition process. The same activity performed by the same subject cannot be determined if the position of the Smartphone does not remain consistent. When considering the literature and real life scenarios, we selected four common positions of the Smartphone to overcome the restriction of a particular position while using our application. The user can select the most convenient position. Table 1 shows the Smartphone positions while gathering data for training or recognising activity. Table 1

Smartphone positions while performing an activity

Position

Description

Hand Palm (HP) Held in the hand facing upward

Remarks Interaction with phone while performing activity

Trousers’ Pocket Trousers’ front left Usual position while (TP) pocket facing outward doing other tasks Waist Mounted (WM)

Held on waist belt in vertical position

Easy for patients/elderly persons under care

Armband (AB)

Kept in the AB, left arm vertical position

Performing sports activities

The raw data were labelled according to the application selected by the user from the application interface. A set of 11 ADL selected for this study were performed by setting the Smartphone in the four different positions mentioned in Table 1; a total of 44 labelled raw datasets was obtained for the experiments. Table 2 shows the labelling of activities for the 44 raw datasets. For initial training datasets, two people performed all the activities for 4–5 min per activity and repeated by keeping the Smartphone in four different positions. Approximately 200 min of data were collected from each person for the initial classification modelling process. The next task of the data collector was to extract the set of features from the raw data for modelling the classifiers. Once the data collector extracted the defined features, they were sent to the data storage cloud. The process of feature extraction can be performed on the cloud side or on a workstation/server module. Because we were developing a recognition system that required the feature extraction process online, we reused the same procedure in the activity recogniser module. Six features that are commonly used and provide good results in recognising human activities by the Smartphone accelerometer were chosen based on a

176

M.A. Awan et al.

literature survey. The list of features along with a brief description is shown in Table 3. Table 2

List of activities for four different Smartphone positions

size is unknown because different activities require short and long spans depending on the situation. However, 2–6 s is recommended in most studies on activity recognition using a Smartphone accelerometer (Awan et al., 2013; Bao and Intille, 2004; Khan et al., 2010; Kmiecik, 2013; Kwapisz et al., 2010; Siirtola and Röning, 2012; Thiemjarus, 2010).

Activity – Activity – Hand Trousers’ Palm (HP) Pocket (TP)

Activity – Waist Activity – Mounted (WM) Armband (AB)

Laying-HP

Laying-TP

Laying-WM

Laying-AB

3.2 Data storage cloud

Sitting-HP

Sitting-TP

Sitting-WM

Sitting-AB

Standing-HP

Standing-TP

Standing-WM

Standing-AB

Using the cloud for data storage is a new scalable approach when compared with the conventional approaches of storing training data locally on the Smartphone or on the workstation for later use. With this approach, data can be accessed from anywhere with access to the internet. The classification process, which requires significant computing resources for creating classification models from a large amount of data can also be deployed on the cloud but includes a cost factor. We used the cloud only for data storage purposes which is freely available up to a certain storage capacity. The data collector can send the training data to the cloud anytime when sufficient data are available. The storage cloud is composed of 2 parts; 1 for storing training data and the other to store classification models. As shown in Figure 1, the storage cloud interacted with the other three parts of the system. The data collector module uploads training data to the cloud when there is sufficient data for a particular activity. The workstation module can download training data from the cloud and upload the classification models once the classification process is completed. The activity recogniser downloads the classification models from the cloud and uploads the test data as training data to the cloud.

Walking-HP

Walking-TP

Walking-WM

Walking-AB

Jogging-HP

Jogging-TP

Jogging-WM

Jogging-AB

AscStairs-HP

AscStairs-TP

AscStairs-WM AscStairs-AB

DescStairs-HP DescStairs-TP AscElevatorHP

DescStairs-WM DescStairs-AB

AscElevator-TP AscElevatorWM

AscElevatorAB

DescElevator- DescElevatorHP TP

DescElevatorWM

DescElevatorAB

AscEscalatorHP

AscEscalatorWM

AscEscalatorAB

AscEscalatorTP

DescEscalator- DescEscalator- DescEscalator- DescEscalatorHP TP WM AB Table 3

List of features selected for the classification process (Wikipedia.org)

Feature

Description

Mean

Average of a number of values

Standard Deviation A quantity calculated to indicate the extent (SD) of deviation for a group as a whole Correlation between axis

A mutual relationship or connection between 2 or more axes.

Variance

The fact or quality of being different, divergent, or inconsistent

Mode

The most common value in a dataset

Kurtosis

Kurtosis characterises the relative peakedness or flatness of a distribution compared with normal distribution

Another important aspect in raw data processing is to use the time domain or frequency domain features depending on the type of application. In this study, we considered only time domain features, leaving the extraction of frequency domain features for future work. For all the activities, the accelerometer data were collected every 20 ms and 50 samples per second, the normal frequency of the accelerometer embedded in a Smartphone. We collected data for 2.56 s, i.e., 128 values for each feature vector of each activity. The selection of 128 values was chosen to maintain the power of 2 to generate the frequency domain signals from time domain signals for future work. We used the sliding window method in extracting the feature vector without overlapping to reduce the workload on the mobile module. The optimal sliding window

3.3 Workstation module Although the workstation needs to be constantly on and online for generating and uploading classification models to the cloud, it is a good choice compared with the price of cloud services and considering the classification burden on a mobile module. The workstation module downloads the training data on a periodic basis from the cloud when sufficient training data are available for creating new models. Machine learning algorithms are applied to the training data to produce classification models for the human activity recognition process. These models uploaded to the cloud and are available to the activity recogniser for activity recognition. The choice of classification algorithm is debatable based on different factors, especially time and accuracy. Some applications require quick responses, e.g., fall detection in healthcare applications and others require accuracy. We selected the six most commonly used machine learning algorithms for detecting human activity using a Smartphone accelerometer. The performance and accuracy of these algorithms were compared based on different datasets and their features. Table 4 shows the list of these algorithms with a brief description.

Subject-independent human activity recognition using Smartphone accelerometer with cloud support Table 4

List of classification algorithms (Wikipedia.org)

Algorithm

Description

Decision Tree A decision support tool that uses a tree-like graph (J48) Naïve Bayes

A simple probabilistic classifier based on Bayes’ theorem

Bayesian Network

A probabilistic directed acyclic graphical model that represents a set of random variables and their conditional dependencies

K-Nearest Neighbour

A non-parametric method for classifying objects based on closest training examples in the feature space

Multilayer Perceptron

A feed forward artificial neural network model that maps sets of input data onto a set of appropriate outputs

Logistic Regression

A type of regression analysis for predicting the outcome of a categorical dependent variable based on one or more predictor variables

177

recognise the human activity. We designed the following four components in our proposed methodology and the implementation is described in detail in the following subsections: •

data collector – a mobile-based module to obtain training data

•

data storage cloud – a database to store training data and classification models

•

workstation module – a module used to generate classification models

•

activity recogniser – a mobile-based module to recognise the user’s current activity.

Figure 2

The working flow of the system

3.4 Activity recogniser The last component in our proposed methodology was a mobile-based module that recognises human activities based on pre-generated classification models. The activity recogniser downloads the up-to-date classifier from the cloud if available, before starting the recognition process. The recognition process is involved in obtaining the raw data from the accelerometer, extracting the necessary predefined features from the raw data and comparing the data pattern of the current activity with the classifier model. If the current activity is recognised, an output is generated showing the current activity of the user that can be used by other applications. The test data are also sent to the storage cloud as training data for creating robust classification models. The process of raw data collection and feature extraction was the same as explained previously in the data collector module. The recognition process was performed on different users, irrespective of training and test data from each user to prove the concept of subject independence. The overall working flow of the proposed methodology is shown in Figure 2.

4

Implementation

Human activity recognition using a Smartphone accelerometer follows a standard procedure starting from raw data collection to activity recognition irrespective of a specific methodology. We used the proposed methodology to improve recognition results and to obtain subjectindependent activity recognition. A common mechanism that we used for human activity recognition is presented in Figure 3. As shown in Figure 3, the raw data were collected from the accelerometer for a specific event and then the data were labelled according to the performed activity followed by the feature extraction and data training procedures. Once the data classifier was generated, new samples were passed to

4.1 Data collector module We developed an android application to collect raw data from the accelerometer, label that data based on performed activity, extract selected features from the data and send to the cloud for storage. The user can select an activity from the list of activities represented by radio buttons, select the position of the phone and then click the start button to begin the raw data collection from the accelerometer. The start button changes to a stop button once the data collection starts and clicking on the same button again stops the data collection procedure. The extract feature button triggers the event of defined features extraction from the raw data and the send button is used to send the training data to the cloud. We used the REST API of the StackMob (https://www. stackmob.com) in our android application to obtain the StackMob’s cloud service for data storage. For the initial classification purpose, as mentioned earlier, we collected data from two people for each activity and for each defined position and then extracted the features from those datasets for classification. Figure 4 shows the sample data patterns of all the activities as an example, while holding the Smartphone by hand. These data patterns show the mean value feature calculated for each x-, y- and z-axis of the data of 2.56 s containing 128 values for each axis.

178

M.A. Awan et al.

Figure 3

Human activity recognition mechanism using a Smartphone accelerometer (see online version for colours)

Figure 4

Data patterns of all activities for mean x-, y- and z-axis values (see online version for colours)

4.2 Data storage module Unlike previous approaches to storing data locally or transferring to a workstation/server through Wi-Fi or Bluetooth, we used the cloud database service, which is more scalable. We used the StackMob’s cloud service because of compatibility with android devices, constant uptime and acceptable limit of free storage. The StackMob Android SDK is model-based, meaning that if we declare a class that represents an object, we can directly save and fetch from that object and StackMob will perform appropriate save and fetch calls to the data store, populating the object asynchronously (https://developer.stackmob .com). We used the StackMob service for storing and

retrieving training data and classification models to and from the cloud, respectively.

4.3 Workstation module This module was used to release the burden of classification process from the mobile module and to provide an interface between the cloud data and generation of classification models. This module was used to download the training data from cloud storage, perform classification process to generate classification models and to store these models on the cloud for the activity recognition process. The WEKA (Waikato Environment for Knowledge Analysis) (http://www.cs.

Subject-independent human activity recognition using Smartphone accelerometer with cloud support

4.4 Activity recogniser module

waikato.ac.nz/ml/weka) tool was used on the workstation for classification processing. Six algorithms, mentioned in Table 4, were used for the classification process. As mentioned, we used the incremental approach for classification modelling based on the size of the training data, thus each specific model had different versions (e.g., version-01 with data size N, etc.). The experiments were performed by maintaining the Smartphone in four different positions (Table 1), using six features for each dataset (Table 3) and six different algorithms (Table 4) for classification with a total of 144 (4 × 6 × 6) basic classification models, irrespective of each model’s version. Table 5 summarises the list of the basic 144 models. The entries HP, TP, WM and AB in the table represent the Smartphone positions against each feature and algorithm. A large number of classification models were generated for evaluation of the system with the maximum possible features and algorithms to have a better choice for different types of applications depending on the time and accuracy. In the present study only accuracy was considered. Table 5

179

A modular approach was used to develop the proposed system. Activity recogniser requires several of the same functions used in the data collector module, i.e., raw data collection and feature extraction. We used the same required classes from the data collector module in our activity recogniser application. The activity recogniser had to obtain the current available versions of the classification models from the cloud storage, recognise the user activity based on the selected Smartphone position, notify the user’s current activity and upload the test data to the cloud storage. The start/stop button was used to start and stop the recognition process, respectively. The phone position was selected before the recognition process because every position had a different classification model. The download button was used to obtain the current up-to-date classification model from the cloud storage and the test data could be uploaded to the cloud by clicking the send button on the interface. The Eclipse (http://www.eclipse.org) IDE was used for the implementation of the proposed system.

Summary of basic classification models

Algorithm Feature

J48

Bayesian network

Naïve Bayes

K-Nearest neighbour

Multilayer perceptron

Logistic regression

H P

T P

W M

A H B P

T P

W M

A B

H P

T P

W M

A H B P

T P

W M

A B

H P

T P

W M

A B

H P

T P

W M

A B

Standard Deviation (SD)

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

Correlation

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

Variance

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

Mode

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

Kurtosis

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

ʺ

Mean

5

Experimentation and evaluation

The experiments were performed in various stages. First, we obtained training data from two people as described earlier and uploaded the information to the cloud. Next, the data were downloaded on the workstation module to build 144 classification models as mentioned in Table 5. The data filtration was performed by removing several seconds of accelerometer data at the start and the end of a performed activity depending on the Smartphone position (Table 1). For example, as the user started the data collector application, data collection from the accelerometer began, but the user required several seconds to place the Smartphone in a specific position before performing the activity and similarly removing the phone from that position after finishing the activity to close the application. The reason for building 144 classification models at the initial stage was to compare machine learning algorithms, various extracted features and Smartphone positions for classification purposes in terms of accuracy, and to find better options for our real-time offline and online testing

because finding volunteers for long-term training and testing data collection was difficult. Time is an important factor in recognising human activities for real-time applications, but this study focused only on accuracy and the time element will be evaluated in our future work. The details of various experimentation and evaluation stages are described in the following subsections.

5.1 Comparison of classification algorithms, feature vectors and Smartphone positions A comparative study among selected algorithms (Table 4), defined features (Table 3) and designated positions (Table 1) was conducted to select the best choice of algorithm, feature and position for offline and online testing on real-time data based on the following criteria: •

feature vs. feature

•

algorithm vs. algorithm

•

position vs. position.

180

M.A. Awan et al.

As an example, Figures 5 and 6 show the data patterns of mean and standard deviation (SD) of x-, y- and z-axes based on the window length of 2.56 s, respectively. The accuracy by confusion matrix using k-nearest neighbour algorithm is also shown. Although keeping all other factors, i.e., algorithm, position, and window length consistent, the selection of feature has a large impact on accuracy. The activities such as lying down, sitting, standing have steady data patterns which are difficult to recognise. In the given example, we compared the data pattern of two features, the mean value and SD. As shown, that the mean value maintained the uniqueness of each activity data pattern and thus improve the accuracy results. Comparative summary of all features mentioned in Table 3 against each algorithm stated in Table 4 and each specified position (Table 1), are presented in Tables 6–9. The data used in this comparison were collected from two people. All subjects performed each activity (total 11 activities) 4–5 min (50 raw data samples in each second; accelerometer’s frequency) and an extracted feature vector of 2.56 s (window length), thus, ~1000 samples of data from each person and each Smartphone position were collected. We used a 10-fold cross-validation testing procedure for the initial experiments. The data were not collected in a controlled lab environment; therefore, a small variation in the results may exist. Several classification algorithms required a longer time, but produced more accurate results, e.g., Bayesian Network algorithm compared with Naïve Bayes algorithm. Because the time factor was not an issue in this study, only the impact of the feature vector, classification algorithm and Smartphone position on accuracy were summarised. Table 6

The extracted features mean and mode provided adequate results when compared with other extracted features since they maintained the uniqueness of each axis even in cases of activities that had steady data patterns (Figures 5 and 6). We did not show the figures of all comparisons to save the space. The k-Nearest Neighbour algorithm performed the best among the selected algorithms. J48 and Bayesian Network also produced good results compared with other algorithms. This conclusion was based on experimental results and the statistical working mechanism of these algorithms was not considered. A fixed and tight orientation of the Smartphone at a specific body position was also observed to provide more accurate results. In our case, the WM and AB positions produced more accurate results because the Smartphone was more fixed and steadied at these positions compared with handheld and TP positions. An unpredictable result (13% accuracy) was observed when we applied the Multilayer Perceptron algorithm on the mode feature of the data (Table 6). In this case, the algorithm required hours to complete the classification model because the x-axis was considered a nominal value according to the Weka tool, although this was deemed irrelevant to the focal point of our study. After completing this stage of experimentation, we selected the k-Nearest Neighbour, J48, and Bayesian Network algorithms for classification and mean and mode values for feature vectors in further experiments. The WM and HP positions were selected for offline and online testing, respectively. The WM position produced better accuracy and was a good choice for offline testing while online testing was better performed with the HP position.

Average accuracy percentage of activities (Smartphone position – AB) Algorithm J48 (%)

Naïve Bayes (%)

Bayesian network (%)

K-Nearest neighbour (%)

Multilayer perceptron (%)

Logistic regression (%)

Mean

97.88

98.25

97.32

99.07

98.04

97.94


79.58

63.02

74.06

87.16

59.15

67.30

Correlation

66.79

58.12

59.67

72.56

60.08

56.63

Variance

79.94

57.81

74.06

85.87

51.78

66.17

Mode

86.38

90.71

94.17

92.05

13.00

85.55

Kurtosis

60.96

50.08

50.95

71.89

51.93

47.60

Feature

Table 7

Average accuracy percentage of activities (Smartphone position – HP) Algorithm J48 (%)

Naïve Bayes (%)





Mean

95.30

87.56

92.41

95.70

88.51

86.26


77.87

62.99

72.03

83.82

60.74

66.53

Correlation

69.78

60.59

62.09

78.37

63.94

61.19

Variance

78.22

61.29

72.03

82.12

48.50

65.63

Mode

94.35

84.55

94.60

96.30

84.25

72.45

Kurtosis

53.10

44.45

47.10

61.74

45.95

43.91

Feature

Subject-independent human activity recognition using Smartphone accelerometer with cloud support Figure 5

The data pattern and confusion matrix based on mean value (see online version for colours)

Figure 6

The data pattern and confusion matrix based on standard deviation (SD) (see online version for colours)

Table 8

181

Average accuracy percentage of activities (Smartphone position – TP) Algorithm J48 (%)

Naïve Bayes (%)





Mean

97.65

91.24

94.69

97.80

94.19

92.79


73.91

56.88

71.61

79.07

53.48

57.39

Correlation

58.79

51.88

50.22

62.84

54.23

45.72

Variance

73.46

55.73

71.61

78.17

47.37

56.23

Mode

90.53

75.06

88.13

91.89

79.02

57.74

Kurtosis

57.84

47.72

52.38

62.69

48.87

47.42

Feature

182

M.A. Awan et al.

Table 9

Average accuracy percentage of activities (Smartphone position – WM) Algorithm J48 (%)

Naïve Bayes (%)





98.34

96.78

97.56

99.63

96.97

95.82


88.91

64.38

78.08

88.10

64.80

72.38

Correlation

72.24

58.45

59.51

78.72

60.29

54.69

Variance

84.05

54.36

78.08

86.85

44.58

70.91

Mode

89.74

85.65

90.48

92.82

82.93

70.97

Kurtosis

63.19

42.60

46.09

71.00

45.73

39.48

Feature Mean

5.2 Offline testing Our experiments and evaluation focused on two hypotheses, which were formulated based on a literature survey and our previous work; first, by increasing the training data samples, recognition accuracy can be improved and secondly, subject-independent human activity recognition can be achieved with a large amount of training data from different users. Offline testing was defined as obtaining the real-time data of a user while performing an activity for a specific time, extract features and test the accuracy of generated samples using classification models at a later time. Ten volunteers participated in offline testing and the procedure was conducted as follows: 1

Subject 1 performed all activities individually (2–3 min for each activity).

2

Features were extracted and data samples stored on the cloud. A validation method was applied on the downloaded samples from the cloud at the workstation.

3 4

Classification models were generated and stored on the cloud.

5

Similarly, subject 2 performed activities and data samples generated as in step 1 and 2. Data samples of subject 2 were tested using the supplied test set method on the workstation module (offline testing) by applying the classification models generated in step 4.

6

7

New classification models were generated by accumulating the data samples of subject 1 and 2 and stored on the cloud.

8

Steps 5–7 were repeated for every new subject.

The graph in Table 10 and Figure 7 shows the summary of test results. The mean value feature vector and WM Smartphone position were used in these experiments. A high percentage of accuracy was achieved using all the selected algorithms as the training data continued to increase. The recognition accuracy increased approximately from 97% to 100% by using Bayesian Network and J48 algorithms, when the training data samples were increased from 668 to 4516. The k-Nearest Neighbour algorithm showed improvement in recognition accuracy

from 99% to 100% only with a small increase of training data samples from 668 to 2270. The results proved that we obtained subject independence and up to 100% accuracy was achieved by accumulating additional training data.

5.3 Online testing This stage of experiments was more difficult as compared with offline testing. In offline testing, every subject performed activities and data samples were generated and stored for further testing. In online testing, every data sample was tested exactly at the time of generation using the saved classification model and data samples were saved to upgrade the classification models. The HP position of the Smartphone was selected for convenience for online testing and mode value as a feature vector because it provided good results as explained in Section 5.1. A defined set of 11 activities and an unknown activity were represented symbolically in the application to count the correctly classified activities. The user kept a tally counter while performing the activity and marked incorrect each time when an unknown or wrong activity classification was found. Then, correctly classified activities were counted by subtracting the incorrect count from the total number of data samples for each activity. The same 10 volunteers participated in the online testing and the procedure was conducted as follows: 1

Subject 1 performed all activities individually (2–3 min for each activity).

2

Features were extracted and data samples stored on the cloud.

3

A validation method was applied on the downloaded samples from the cloud at the workstation.

4

Classification models were generated and stored on the cloud.

5

Classification models were downloaded on the Smartphone.

6

Subject 2 performed an activity and each generated data sample was tested using downloaded classifier (online testing) and incorrect classifications were marked with a tally counter.

Subject-independent human activity recognition using Smartphone accelerometer with cloud support 7

Generated data samples were uploaded to the cloud and the total count saved locally.

8

Correct classified instances were determined by subtracting the incorrect count from the total.

9

Steps 6–8 were repeated for each activity.

data at the start and end of an activity as described previously. All three algorithms used for classification in offline testing were also used for online testing. The k-Nearest Neighbour algorithm performed the best again and the accuracy improved up to 100% from 93% in achieving subject independence from a training data sample of 570–4609. The accuracy of J48 and Bayesian Network algorithms increased from 91% and 88% to 99%, respectively, for the same set of training data samples. In conclusion, the training data size had a large impact on the accuracy of recognition and human activities can be recognised independent of the subject as we proved in our experiments and evaluation procedure. However, the training data size is not the only factor in recognising human activity independent of subject and various other factors, including age, gender and social behaviour may also contribute. These issues should be considered to provide a large-scale solution of subject-independent human activity recognition.

10 New classification models were generated by accumulating the data samples of subject 1 and 2 and stored on the cloud. 11 Steps 5–10 were repeated for every new subject. A graph in Table 11 and Figure 8 shows the summary of test results. The mode value feature vector and HP Smartphone position were used in this online testing. The experiments were carefully performed and the transition errors were not considered, defined as errors that occurred when transitioning from one activity to another. Transition errors were dependent on the window length size along with other parameters. We fixed the window length at 2.56 s and data filtration was performed by removing a few seconds of raw Table 10

183

Summary of offline testing results Correctly classified instances

Training data samples

Test data samples

Bayesian network

J48

K-Nearest neighbour

668

541

528

530

1209

505

491

498

1714

556

544

2270

559

548

2829

570

3399 3970

Accuracy percentage Bayesian network

J48

K-Nearest neighbour

536

97.597%

97.9667%

99.0758%

504

97.2277%

98.6139%

99.802%

545

550

97.8417%

98.0216%

98.9209%

554

559

98.0322%

99.1055%

100%

565

570

570

99.1228%

100%

100%

571

559

569

571

97.8984%

99.6497%

100%

546

541

545

546

99.0842%

99.8168%

100%

4516

517

517

517

517

100%

100%

100%

5033

561

561

561

561

100%

100%

100%

Figure 7

Accuracy percentage against various data samples (offline testing) (see online version for colours)

184

M.A. Awan et al.

Table 11

Summary of online testing results Correctly classified instances

Training data samples

Test data samples

Bayesian network

570

520

1090

479

1569 2051

Accuracy percentage

J48

K-nearest neighbour

Bayesian network

J48

K-nearest neighbour

460

478

486

88.46%

91.92%

93.46%

441

437

448

92.07%

91.23%

93.53%

482

464

462

473

96.26%

95.85%

98.13%

507

495

497

501

97.63%

98.03%

98.82%

2558

567

559

559

566

98.59%

98.59%

99.82%

3125

455

447

452

455

98.24%

99.34%

100%

3580

506

498

503

504

98.42%

99.41%

99.60%

4086

523

519

521

522

99.23%

99.62%

99.81%

4609

548

544

547

548

99.27%

99.82%

100%

Figure 8

6

Accuracy percentage against various data samples (online testing) (see online version for colours)

Conclusion and future work

In this paper, we demonstrated a methodology to achieve human activity recognition independent of the subject using a Smartphone accelerometer. A system was proposed and developed to recognise human activities using a Smartphone accelerometer irrespective of the user for training and test datasets. A set of 11 basic ADL, lying down, sitting, standing, walking, jogging, ascending stairs, descending stairs, ascending in an elevator, descending in an elevator, ascending on an escalator and descending on an escalator, were recognised independent of the subject. We also performed a comparative study on a selected set of classification algorithms, extracted features, and Smartphone positions of the user to determine their impact on activity recognition accuracy. Finally, we conducted an offline and online testing of the system in recognising the activities of different subjects and achieved an average accuracy above 90% in recognising all the activities independent of the subject, which shows the scalability and effectiveness of our proposed methodology. We focused on the importance of training data size in achieving the subject-independent human activity recognition, which is not the only factor. In the future we

will also consider other components, including age, gender and subject’s social behaviour. During experimentation and comparison of different algorithms, features and Smartphone positions, only the accuracy component was considered; however, time is also an important component for real-time applications. We are considering a study on this aspect of the human activity recognition in the future.

Acknowledgements This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT and Future Planning (2012R1A1A2043400).

References Avci, A., Bosch, S., Marin-Perianu, M., Marin-Perianu, R. and Havinga, P. (2010) ‘Activity recognition using inertial sensing for healthcare, wellbeing and sports applications: a survey’, ARCS 2010: Proceedings of the Architecture of Computing Systems (ARCS) Conference, Hannover, Germany, pp.1–10.

Subject-independent human activity recognition using Smartphone accelerometer with cloud support Awan, M.A., Guangbin, Z. and Kim, S.D. (2013) ‘A dynamic approach to recognize activities in WSN’, International Journal of Distributed Sensor Networks, Vol. 2013 [Online], http://www.hindawi.com/journals/ijdsn/2013/385276/ (Accessed on 15 October 2013). Bao, L. and Intille, S.S. (2004) ‘Activity recognition from userannotated acceleration data’, Lecture Notes in Computer Science, Pervasive Computing, Vol. 3001, pp.1–17. Bourke, A.K., O’Brien, J.V. and Lyons, G.M. (2007) ‘Evaluation of a threshold-based tri-axial accelerometer fall detection algorithm’, Gait & Posture, Vol. 26, No. 2, pp.194–199. Cook, D.J. Augusto, J.C and Jakkula, V.R. (2009) ‘Ambient intelligence: technologies, applications, and opportunities’, Pervasive and Mobile Computing, Vol. 5, No. 4, pp.277–298. Guan, D., Ma, T., Yuan, W., Lee, Y.K. and Jehad Sarkar, A.M. (2011) ‘Review of sensor-based activity recognition systems’, IETE Technical Review, Vol. 28, No. 5, pp.418–433. Jafari, R., Li, W., Bajcsy, R., Glaser, S. and Sastry, S. (2007) ‘Physical activity monitoring for assisted living at home’ in BSN 2007, IFMBE Proceedings, Vol. 13, pp.213–219. Karantonis, D.M., Narayanan, M.R., Mathie, M., Lovell, N.H. and Celler, B.G. (2006) ‘Implementation of a real-time human movement classifier using a tri-axial accelerometer for ambulatory monitoring’, IEEE Transactions on Information Technology in Biomedicine, Vol. 10, No. 1, pp.156–167.

185

Khan, A.M., Lee, Y.K., Lee, S. and Kim, T.S. (2010) ‘Accelerometer’s position independent physical activity recognition system for long-term activity monitoring in the elderly’, Medical & Biological Engineering & Computing, Vol. 48, No. 12, pp.1271–1279. Kmiecik, L.S. (2013) Cloud Centered, Smartphone Based Longterm Human Activity Recognition Solution, [Online] A Project Report, Department of Computing, Imperial College London, London, http://www.doc.ic.ac.uk/teaching/distinguishedprojects/2013/l.kmiecik.pdf (Accessed on 10 October 2013). Kwapisz, J.R., Weiss, G.M. and Moore, S.A. (2010) ‘Activity recognition using cell phone accelerometers’, ACM SIGKDD Explorations Newsletter, Vol. 12, No. 2, pp.74–82. Lara, O.D. and Labrador, M.A. (2013) ‘A survey on human activity recognition using wearable sensors’, IEEE Communications Surveys & Tutorials, Vol. 15, No. 3, pp.1192–1209. Siirtola, P. and Röning, J. (2012) ‘Recognizing human activities user-independently on smartphones based on accelerometer data’, International Journal of Interactive Multimedia and Artificial Intelligence, Vol. 1, No. 5, pp.38–45. Thiemjarus, S. (2010) ‘A device-orientation independent method for activity recognition’, BSN 2010: Proceedings of the International Conference on Body Sensor Networks, Singapore, pp.19–23.