Automatic Key Pose Selection for 3D Human Action ... - Springer Link

9 downloads 0 Views 563KB Size Report
poses” model that represents human actions as histograms of key-pose occurrences over ..... Action Performances # Motion Cycles # Body Postures #. Walk. 67.
Automatic Key Pose Selection for 3D Human Action Recognition Wenjuan Gong, Andrew D. Bagdanov, F. Xavier Roca, and Jordi Gonz` alez Universitat Aut` onoma de Barcelona, Computer Vision Center Campus UAB Edifici O, 08193 Bellaterra, Spain

Abstract. This article describes a novel approach to the modeling of human actions in 3D. The method we propose is based on a “bag of poses” model that represents human actions as histograms of key-pose occurrences over the course of a video sequence. Actions are first represented as 3D poses using a sequence of 36 direction cosines corresponding to the angles 12 joints form with the world coordinate frame in an articulated human body model. These pose representations are then projected to three-dimensional, action-specific principal eigenspaces which we refer to as aSpaces. We introduce a method for key-pose selection based on a local-motion energy optimization criterion and we show that this method is more stable and more resistant to noisy data than other key-poses selection criteria for action recognition. Keywords: Human action recognition, direction cosine, key poses, bag of words.

1

Introduction

Human action recognition is an important problem in computer vision. Applications include video surveillance, automatic video indexing and human computer interaction. In surveillance systems installed in places requiring high security, such as banks, human action recognition can be applied to detect abnormal human actions and potentially dangerous situations before they become truly dangerous. Human action characterization is also making inroads in the area of security and safety monitoring. Behavior analysis systems are being built to monitor the safety of children and the elderly, and in such scenarios, abnormal action detection can be used to detect dangerous situations like falling down. Automatic video indexing for video and image libraries can be enhanced using human action recognition and by allowing semantics-based access to multimedia content. Human action recognition can also be applied in human computer interaction, where recognition results can be used to understand human behaviors so computers can react accordingly. Despite increased interest in recent years, human action recognition remains a challenging problem. The flexibility of the human body and the variability of human actions produce high-dimensional motion data. How to represent these F.J. Perales and R.B. Fisher (Eds.): AMDO 2010, LNCS 6169, pp. 290–299, 2010. c Springer-Verlag Berlin Heidelberg 2010 

Automatic Key Pose Selection for 3D Human Action Recognition

291

data in a compact and effective way is the first challenge. Also, as a classification problem, the extreme variability of pose and body articulation occurring in instances of the same action type make class characterization very difficult. Conversely, similarities of pose and cadence across different action types, for example running and jogging, make identifiable action classes difficult to discriminate. How to model human actions to be generic enough for one action and specific enough to distinguish different actions is another challenge. Some researchers utilize 3D motion data in the original dimensional space, for example [7], [8] and [12], while others explore polar angles [3] or direction cosines [14]. Our representation to human action recognition is as follows. We use a 3D stick figure of twelve limbs to model the human body. The orientation of each limb is represented by three direction cosines of the angles made by the limb and the world coordinate system. Direction cosines posses a number of useful invariants, and in particular by using them we eliminate the influence of different limb lengths. Another advantage of using direction cosines compared to Euler angles is that they do not lead to discontinuities in motion sequences. Also, compared to quaternion, they have a direct geometric interpretation [17]. Motion sequences are then represented as a sequence of static human postures, each represented as a vector of direction cosines. Since the natural constraints of the human body motion lead to highly correlated data in the original space [16], we derive a compact, non-redundant representation of human pose by applying Principle Component Analysis (PCA) to these pose sequences. This space (aSpace) then becomes the basis for vocabulary selection. Considering the impressive performance of recent applications of bag of words architecture in classification [6,5,2], we introduce this architecture to solve the action recognition problem. In the bag of words model, the target is represented as an unordered collection of representative features (vocabulary). In our approach, we modify the bag of words architecture by computing the vocabulary as the most representative poses, or key poses, resulting in a bag of poses model for human action recognition. The main contributions of this work are: – We calculate the most representative poses, or key poses, as the vocabulary. By doing this, we pick the most representative features and the vocabulary is very concise. – We extend the human walking model in [14] to include ‘jump’, ‘box’ and, ‘run’ actions. And we utilize all these action models as priors in action recognition. – We compare our method of extracting key poses with other methods: poses corresponding to randomly spaced frames [1], equally spaced frames [10], local maximum distances [3] and centers of clusters calculated using k-means clustering. Our method has a comparatively stable performance and is more resistant to noisy data. In the next section we introduce our representation of human posture and human motion. Section 3 explains our classification method, and we give experimental

292

W. Gong et al.

(a)

(b)

Fig. 1. (a) The 3D stick figure model used for representing human pose. Ten principal joints corresponding to the markers used in motion capture are used [15]. (b) Angles (θlx , θly , θlz ) between the limb l and the axes [14].

results in Section 4. We conclude in section 5 with a discussion of our results and future work.

2

Human Posture and Human Motion Models

We use the same representations for human posture and human motion as in [14]. In this section, we give a brief introduction to how we represent a static human posture using direction cosines derived from a human body model. Then, we explain our extensions to the human motion models of [14]. 2.1

Representing Human Posture and Motion

In our method, a human body is modeled as twelve rigid body parts: hip, torso, shoulder, neck, two thighs, two legs, two arms and two forearms. These parts are connected by a total of ten inner joints, as shown in figure 1(a). Body segments are structured in a hierarchical manner, constituting a kinematic tree rooted at the hip, which determines the global rotation of the whole body. A static human posture is represented as a vector of limb orientations. Limb orientation is modeled using three parameters, without modeling self rotation of limbs around its axes, as shown in figure 1(b). The posture of the subject in each frame is represented using a vector of direction cosines measured on twelve limbs. This results in a 36-dimensional representation: y x z ), cos(θ12 ), cos(θ12 )], ψ = [cos(θ1x ), cos(θ1y ), cos(θ1z ), . . . , cos(θ12

(1)

Automatic Key Pose Selection for 3D Human Action Recognition

293

where θlx , θly and θlz are the angles between the limb l and the axes as shown in figure 1(b). After representing static human postures using direction cosines, we represent a motion sequence of one performer as a sequence of static postures: Ψ = [ψ1 , ψ2 , . . . , ψn ],

(2)

where n is number of postures (frames) in this motion performance. 2.2

The Projected Parameter Space: aSpace

Using the representations mentioned above, a human motion sequence with n postures will have a 36 × n dimensional representation. Since the natural constraints of the human body motion lead to highly correlated data in the original space [16], we apply PCA to decrease dimensions of the representation (refer to [14] for more details). Figure 2 are visualizations of variations in aSpaces re-projected to the original parameter space. From figure 2, we can see what pose variations each eigenvector accounts for in the eigenspace decomposition. For example, we can see that the most significant eigenvectors in the walking and running spaces correspond to their characteristic, vigorous, and simultaneous motion of the arms and legs. The most significant eigenvector for ’box’, on the other hand, accounts for the boxer’s characteristic footwork and torso rotation. And the first three eigenvectors represent most of the energies from all the actions. 2.3

Human Motion Model

The authors of [14] propose a method of modeling human walking. First, they synchronize the data (the same method as in [13]) to find corresponding postures from different motion performances with different frame numbers and map all performances to the same length. Second, they calculate mean performance from synchronized data and model human walking using this mean performance, the standard deviation, the mean direction of motion and the covariance matrix of the error. This model achieved high performance in tracking walking humans. Considering the capability of extracting common characteristics of walking action, we introduce this model into action recognition problem. We consider four actions: ‘walk’, ‘run’, ‘box’ and ‘jump’. Figure 3 shows the mean performances for all actions in their respective aSpaces. We observe that mean performances from different actions are distinguishable enough and based on these models, we propose “bag of poses” architecture for action recognition.

3

Bag of Poses

The bag of words architecture (sometimes called bag of features) was first proposed for text document analysis and further adopted for computer vision applications. The main steps in the bag of words model are as follows: compute

294

W. Gong et al.

555 200

100

77777

10 10 10 10 10

100

11 8 11 8 11 11 11 8 333 1 222 8 8

12 12 12 12 12 15 15 13 13 15 13 15 13 13 15

−300

15 13 15 15 13 13 0

−200

−300 −200 −100

0 100

0

200

200

777

−100 14 14 14 14 14

−200

15 13 −300 −200 −100

6

y

y

14 14 12 1414 12 1412 12 12 15 13

−200

4

11 11 11 11 11 3 1 2 88888

0

−100

−100

9 10 10 10 10

100

11 3 11 11 11 11 1 2 888

0

5

200 77

y

0

555 99 444 666

200

999 44 66666 10 10 1010 10

0 100

14 14 14 14 14

−200

−200

−200 0

0

200

200

z

12 12 12 12 12 13 15 15 13 15 13 15 15 13 13

−300 −300 −200 −100

100

200

200

z

z

x

(a)

x

x

(b)

(c)

(a) The three variations for aWalk. 55555

200

100 50

200

999994444466666 10 10 77 10 8 10 7 11 10 8 7 7 11 11 8 1111 88 333 1 22

100

0

y

15 14 14

−150 −200 −250 −200 −100

15 13 15 13 0

−200 0

100

0

77

−100 −150

15 13 1513 1513

−300 −300 −200 −100

0

200

200

6 6

8 8 8 8 8 3 1 2

−50 15 13 1414141414 12 12 12 12 12 15 13

−200

15 13

11 11 11 11 11

50

−100

13

12 12 14 15 13 12 12 14 14 12

4

10 10 10 10

100

−50 −100

9

150

y

0

5 5

55555

200

9999 44 6666 106 7 8 10 11 7 1010 10 7 11 8 77 11 1111 8 8 8 33 1 2 2

150

100

14 14 14 14 14

−200 −200 −100

0 200

200

z

0

1512 12 1213 12 12 15 13

−200

15 13 15 13 13200 15 200

100

0

z

x

x

(d)

(e)

(f)

(b) The three variations for aRun. 5 5 5 5 11 5 9 68 8 99411 64 9 4694846 6 11 10 10 8 10 10 7 7 7 77 11 811 10

200

100

150 100

5555 11 5 11 999 6 84844686668 9 44 8 119 7 10 10 7777 10 1010

200

100

8

50

331 222 333 22

0

55555 8 9 4448 666 99 8 11 11 11 11 11 10 77877 10 7 101010

200

11 3 2 3111 2

0

33 1 22

0 −50

−100 14 14 12 14 14

−200

12

12 15

15 15 13 15 15 13

−300 −200 −100

0

12 12 13

−100 −100

200

12 12 12 12 12

14 14 14 14 14

−200

−200 15

−250 −200 0

100

14 12 12 14 14 14 14 12 12 1213

−150

13

13

−200

−100

0

200

13 15 15 15 13 15 13 13 100 200 200

(g)

−200 0

−300 −300 −200 −100

15 15 15 15 15

13 1313 1313

−200 0

0

100

(h)

200

200

(i)

(c) The three variations for aBox. 55 5 9 5 9 4 44 66 9 6 910 4 6 7 11 9 510 8 10 7 4 10 6 77 10 11 87

200 150 100 50

11 11 11

0

200

100

0

3 1 2 8 88

55 5 99 4 59 4 4 6 6 6 95 41010 6 7 10 77 910 7 10 4 6 7 11 1111 3 3 1 22 88 11 11 8 8

200

100 8

0

−50

−150

14 14 14 1414 12 12 12 1212

−200 −250 −200 −100

0

−200 0

100

200

(j)

14 14 14 14 14

−200

1515151515 1313131313 200

−300 −300 −200 −100

8

−100

−100

−100

5 55 59 59 4 4 6 6 9 9 1044 66 9 10 4 6 77 10 7 10 7 7 1110 11 11 3 1 2 8 11 8 8 11 8

12 12 12 12 12 15 15 15 15 15

0

−200 0

100

200

14 1414 1412 14 12121212 15 13 15 13 15 13 15 15 13 13

−200

13 13 13 13 13 200

(k)

−300 −300 −200 −100

0

−200 0

100

200

200

(l)

(d) The three variations for aJump. Fig. 2. Visualizations of variations in aSpaces re-projected to the original parameter space. Figures in the first, second and third columns correspond to eigenvectors with the largest, the second largest and the third largest eigenvalues separately. Figures in different rows are from different actions. Variations of postures in one figure are re-projections of −3 to 3 times its corresponding eigenvector in aSpaces.

Automatic Key Pose Selection for 3D Human Action Recognition

0.6

0.6

0.4

0.4

0.2

0.2

0

0

295

0.6 0.4 0.5

0.4 0.2

0.2 0 −0.5 −1

−0.5

−0.2 −0.5 −0.4 1.5

0.5 1

−0.4

(a)

−0.5

−0.2

0

0 −0.2

0

0

−0.2

0

0.5

−1

0

−0.4 −0.8

0.5 1

0.5

0

−0.5

−1

−1.5

1

−0.5 −0.6

−0.4

−0.2

(b)

0

0.2

(c)

0.4

0.6

0.8

−1

−0.4 −1.2

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.5

(d)

Fig. 3. Mean performances for (a) ‘walk’, (b) ‘run’, (c) ‘box’ and (d) ‘Jump’

descriptors for input data; compute representative words using k-means clustering over feature descriptors to form vocabulary; quantize descriptors into words and represent input data as histograms over the vocabulary, of a bag of words. We modify the architecture by calculating the vocabulary using energy optimization over human motion sequences instead of k-means clustering. Since we represent motion sequences as occurrences of poses, we name our model the “bag of poses”. 3.1

Vocabulary

In bag of words architecture, k-means is the most frequently used method for vocabulary computation. Instead of k-means, we calculate words as poses with local maximum and local minimum energy differences. Given the mean performance of an action, the motion energy at the i-th frame is defined as [9]: i

i−1

Ei = |ψ ak − ψ ak |2 ,

(3)

where |.| denotes Euclidean distance. We calculate frame numbers with local maximum and local minimum motion energies. After obtaining the frame numbers with local maximum and local minimum energies in aSpace, we reconstruct them to get human poses in the original space:  ψ = Eb ψ + ψ,

(4)

where Eb is the matrix of principal eigenvectors constituting basis for this particular aSpace. This step is important, as each aSpace represents an independent and action-specific sub-manifold of the original space, and as such two aSpace representations from different actions are not directly comparable. Re-projecting to the original space, however, makes this comparison possible. Using the above method, we calculate key poses for each action separately and then concatenate them to compose the vocabulary. Figure 4 shows key poses extracted for all actions. These key poses are now the vocabulary of poses used in our model. For each human motion sequence, each human pose is assigned to a single key pose by computing the Euclidean distance between it and each key pose in the vocabulary. The result is a the histogram of occurrence of these key poses for each motion sequence.

296

W. Gong et al.

(a)

(c)

(b)

(d)

Fig. 4. Key poses extracted using energy measurement for (a) ‘walk’, (b) ‘run’, (c) ‘box’, and (d) ‘jump’

3.2

Classification

A support vector machine is trained using training motion sequences. We use a linear kernel in all experiments. We tried voting for classification: given a test motion sequence, we calculate the nearest key poses for all postures and label them with the action type of the nearest key poses, finally the test sequence is labeled as the action type with the maximum occurrences. The performance of voting is not as good as histogram and support vector machine, due to the fact that for action recognition, the occurrences of key poses are also important characteristics. We will show classification results in the next section.

4

Experimental Results

We test our method on CMU Graphics Lab Motion Capture Database1 . This database includes 23 motion categories (run, walk, jump, varied and so on). The data are joint positions in world coordinates captured using a marker set. In our experiments, we consider four actions: ‘walk’, ‘run’, ‘box’ and ‘jump’. We split 1

http://mocap.cs.cmu.edu/

Automatic Key Pose Selection for 3D Human Action Recognition

297

Table 1. The composition of the dataset for all experiments. Each action type consists of a number of performances, which in turn contain a number of cycles of that action and finally the postures comprising these cycles. Action Performances # Motion Cycles # Body Postures # Walk 67 126 16,891 Run 23 23 2,032 Box 1 37 6,865 Jump 18 24 3,504

1 0.98 0.96 0.94 0.92 0.9 0.88 0.86

Random Equal Kmeans Energy

0.84 0.82

0

10

20

30

40

50

Fig. 5. Classification accuracy of different methods with different scale of noises

the motion cycles for each recorded motion sequence and then synchronize the motion cycles using methods in [13]. Thus, a motion cycle becomes the basic unit of classification in our experiments. The composition of the data is shown in Table 1. We add zero-mean Gaussian noise to the performance data with covariance equals κ · σ, where σ is the covariance between the mean performance Ψ and all the training performance in an action, and κ is a scale factor controlling the noisy degree. Noise from reconstructions of human poses from videos or images are not necessarily the same distribution as those from capturing motion data. We choose to add noise of the same distribution instead of adding some random unknown noise to check the trend of performances and the stability of the methods. After adding noise to the test data, we perform leave-one-out cross validation over the entire dataset. To compare with our method, we implemented four additional methods for computing the vocabulary of key poses: 1. Poses corresponding to randomly spaced frames: Ψ r = {ψ1 , ψ2 , ..., ψk }, where each ψj is randomly selected. 2. Poses corresponding to equally spaced frames [10]: Ψ e = {ψ1 , ψ2 , ..., ψk }, where each ψj and ψj+1 are equally spaced in time.

298

W. Gong et al.

3. Poses with local maximum Mahalanobis distances [3]: Key poses are defined from a probabilistic point of view where characteristic postures are the least likely body postures of each mean performance. 4. The center poses of the clusters: Ψ r = {ψ1 , ψ2 , ..., ψk }, where ψj is the center of the cluster Cj calculated using the k-means methods. We test the four methods with step size as 5 and noise κ ranging from 0 to 50. N equals 210, the number of all the performances. We set the number of key poses in the first and the second methods 26, the same as the vocabulary size in our method. Also, we set the number of the clusters in the fourth method as 26. The experiment results are shown in figure 5. The horizontal axis represents the added noises, the vertical axis represents the classification accuracy and we compare our method with other three methods. From the figure, we can see that methods based on ‘randomly spaced key poses’, ‘equally spaced key poses’ and ‘key poses from energy optimization’ outperform method based on ‘k-means clustering’. Method based on ‘distance’ is not shown in the figure, since the rest four methods outperform this one. The reason that method based on ‘distance’ does not give good performance might be different ways of comparison with the vocabulary: in our architecture, we compute the distance using Euclidean distance while in method based on ‘distance’, they use Mahalanobis distance. Methods based on ‘equally spaced key poses’ and ‘randomly spaced key pose’ outperform our ‘energy optimization’ method with small noises, but their performances sometimes increase while increasing noise scales. We conclude that our selected key poses are more representative, so our method is more stable than the others and more resistant to noisy data.

5

Conclusions and Future Work

We propose a “bag of poses” model for action recognition and report on experiments with the CMU Graphics Lab Motion Capture Database. We show that our method is effective and more stable than other implemented methods. By using a statistical method to model human actions, we extend the work of [14] to include more actions. Our resulting model is able to represent all actions using a very compact representation consisting of histograms of key pose occurrences. Experiments show that our approach is also robust over a wide range of noise intensities in the underlying pose representation. We should note that there is a body of work in computer graphics on action characterization in 3D motion capture databases [4,11]. Although the goals of these approaches are different from us, and they focus mostly on retrieval, an eventual comparison with these techniques in the future could be fruitful.

Acknowledgements This work has been supported by the Spanish Research Program ConsoliderIngenio 2010: MIPRCV (CSD200700018) and by the Spanish projects TIN200914501-C02-01 and TIN2009-14501-C02-02.

Automatic Key Pose Selection for 3D Human Action Recognition

299

References 1. Arie, B.J., Wang, Z., Pandit, P., Rajaram, S.: Human Activity Recognition Using Multidimensional Indexing. IEEE Trans. Pattern Anal. Mach. Intell. 24(8), 1091– 1104 (2002) 2. Bosch, A., Zisserman, A., Munoz, X.: Representing shape with a spatial pyramid kernel. In: CIVR ’07, pp. 401–408. ACM, Amsterdam (2007) 3. Gonz` alez, J., Varona, J., Roca, F.X., Villanueva, J.J.: Analysis of Human Walking Based on a Spaces. In: Perales, F.J., Draper, B.A. (eds.) AMDO 2004. LNCS, vol. 3179, pp. 177–188. Springer, Heidelberg (2004) 4. Kovar, L.: Automated Extraction and Parameterization of Motions in Large Data Sets. ACM Transactions on Graphics 23, 559–568 (2004) 5. Lazebnik, S., Schmid, C., Ponce, J.: Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories. In: CVPR ’06, Anchorage, Alaska, pp. 2169–2178 (2006) 6. Li, F., Perona, P.: A Bayesian Hierarchical Model for Learning Natural Scene Categories. In: CVPR ’05, Washington, DC, USA, pp. 524–531 (2005) 7. Lv, F., Nevatia, R., Lee, M.: 3D Human Action Recognition Using Spatio-temporal Motion Templates. In: CVHCI ’05, p. 120 8. Lv, F., Nevatia, R.: Recognition and Segmentation of 3D Human Action Using HMM and Multi-Class AdaBoost. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954, pp. 359–372. Springer, Heidelberg (2006) 9. Lv, F., Nevatia, R.: Single View Human Action Recognition using Key Pose Matching and Viterbi Path Searching. In: CVPR’07, Minneapolis, Minnesota, USA , pp. 1–8 (2007) 10. Masoud, O., Papanikolopoulos, N.: A Method for Human Action Recognition. Image and Vision Computing 21(8), 729–743 (2003) 11. M¨ uller, M., R¨ oder, T., Clausen, M.: Efficient content-based retrieval of motion capture data. ACM Transactions on Graphics 24(3), 677–685 (2005) 12. Raptis, M., Wnu, K., Soatt, S.: Flexible Dictionaries for Action Classification. In: Proceedings of the Workshop on Machine Learning for Visual Motion Analysis (2008) 13. Rius, I., Gonz` alez, J., Mozerov, M., Roca, F.X.: Automatic learning of 3D pose variability in walking performances for gait analysis. International journal for computational vision and biomechanics 1, 33–43 (2007) 14. Rius, I., Gonz` alez, J., Varona, J., Roca, F.X.: Action-specific motion prior for efficient Bayesian 3D human body tracking. Pattern Recognition 42(11), 2907– 2921 (2009) 15. Sigal, L., Black, J.M.: HumanEva: Synchronized video and Motion Capture Dataset for Evaluation of Articulated Human Motion. Brown University (2006) 16. Zatsiorsky, M.V.: Kinematics of Human Motion. Human Kinetics Publishers (1998) 17. Zatsiorsky, M.V.: Kinetics of Human Motion. Human Kinetics Publishers (2002)