Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

23 Jan 2018  ·  Sijie Yan, Yuanjun Xiong, Dahua Lin ·

Dynamics of human body skeletons convey significant information for human action recognition. Conventional approaches for modeling skeletons usually rely on hand-crafted parts or traversal rules, thus resulting in limited expressive power and difficulties of generalization. In this work, we propose a novel model of dynamic skeletons called Spatial-Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data. This formulation not only leads to greater expressive power but also stronger generalization capability. On two large datasets, Kinetics and NTU-RGBD, it achieves substantial improvements over mainstream methods.

PDF Abstract
Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Multimodal Activity Recognition EV-Action ST-GCN (Skeleton Kinect) Accuracy 79.6 # 2
Multimodal Activity Recognition EV-Action ST-GCN (Skeleton Vicon) Accuracy 50.7 # 7
3D Human Pose Estimation Human3.6M ST-GCN Average MPJPE (mm) 57.4 # 233
Action Recognition ICVL-4 ST-GCN Accuracy 80.23% # 2
Action Recognition IRD ST-GCN Accuracy 74.03% # 2
Skeleton Based Action Recognition NTU RGB+D ST-GCN Accuracy (CV) 88.3 # 90
Accuracy (CS) 81.5 # 92
Skeleton Based Action Recognition NTU RGB+D ST-GCN [Vanilla, 2D Skeleton] Accuracy (CV) 95.1 # 45
Accuracy (CS) 90.1 # 39
Skeleton Based Action Recognition NTU RGB+D ST-GCN [Vanilla, 3D Skeleton] Accuracy (CV) 93.2 # 67
Accuracy (CS) 86.6 # 65
Skeleton Based Action Recognition NTU RGB+D ST-GCN [PYSKL, 3D Skeleton] Accuracy (CV) 96.5 # 22
Accuracy (CS) 90.7 # 34
Skeleton Based Action Recognition NTU RGB+D 120 ST-GCN [PYSKL, 3D Skeleton] Accuracy (Cross-Subject) 86.2 # 30
Accuracy (Cross-Setup) 88.4 # 26
Skeleton Based Action Recognition NTU RGB+D 120 ST-GCN [PYSKL, 2D Skeleton] Accuracy (Cross-Subject) 84.7 # 37
Accuracy (Cross-Setup) 89.0 # 23
Skeleton Based Action Recognition UAV-Human ST-GCN CSv1(%) 30.25 # 6
CSv2(%) 56.14 # 5
Skeleton Based Action Recognition Varying-view RGB-D Action-Skeleton ST-GCN Accuracy (CS) 71% # 2
Accuracy (CV I) 25% # 3
Accuracy (CV II) 56% # 3
Accuracy (AV I) 53% # 2
Accuracy (AV II) 43% # 7

Methods


No methods listed for this paper. Add relevant methods here