Results of the summarizing process for the arXiv paper: 2103.03663v1

This paper presents a system for real-time RGBD-based estimation of 3D human pose, focusing on the body pose, hand pose and facial expression. The system utilizes a parametric 3D deformable human mesh model (SMPL-X) as a representation and leverages the Kinect Azure RGB-D camera for data input. The authors train estimators for body pose and facial expression parameters using previously published landmark extractors and custom annotated datasets. Hand pose is estimated directly using a previously published method. The predictions from these estimators are combined to generate a temporally-smooth human pose. To train the facial expression extractor, the authors annotate a large talking face dataset with facial expression parameters. For the body pose, they collect and annotate a dataset of 56 people captured from a rig of 5 Kinect Azure RGB-D cameras, in addition to utilizing a large motion capture AMASS dataset. The results show that the RGB-D body pose model outperforms state-of-the-art RGB-only methods while achieving comparable accuracy to slower RGB-D optimization-based solutions. The entire system runs at 30 frames per second on a server with a single GPU. In summary, this paper presents an advanced system for real-time extended body pose estimation that incorporates accurate estimations of body pose, hand pose and facial expressions using RGBD data inputs. The system demonstrates improved accuracy compared to RGB-only methods and achieves real-time performance on standard hardware configurations.

• System for real-time RGBD-based estimation of 3D human pose
• Focus on body pose, hand pose, and facial expression
• Utilizes parametric 3D deformable human mesh model (SMPL-X) and Kinect Azure RGB-D camera
• Estimators trained for body pose, facial expression parameters using landmark extractors and custom annotated datasets
• Hand pose estimated using a previously published method
• Predictions combined to generate temporally-smooth human pose
• Facial expression extractor trained with annotated talking face dataset
• Body pose dataset collected and annotated from 56 people captured by 5 Kinect Azure RGB-D cameras, in addition to utilizing a large motion capture AMASS dataset
• Results show outperformance of RGB-D body pose model compared to state-of-the-art RGB-only methods, comparable accuracy to slower RGB-D optimization-based solutions
• Entire system runs at 30 frames per second on a server with a single GPU
• Advanced system for real-time extended body pose estimation incorporating accurate estimations of body pose, hand pose, and facial expressions using RGBD data inputs
• Demonstrates improved accuracy compared to RGB-only methods and achieves real-time performance on standard hardware configurations.

Real-Time RGBD-Based Estimation of 3D Human Pose

In recent years, the field of computer vision has seen a surge in research on real-time estimation of human pose from RGB images. This paper presents an advanced system for real-time extended body pose estimation that incorporates accurate estimations of body pose, hand pose and facial expressions using RGBD data inputs. The authors train estimators for body pose and facial expression parameters using previously published landmark extractors and custom annotated datasets. Hand pose is estimated directly using a previously published method. The predictions from these estimators are combined to generate a temporally-smooth human pose.

System Overview

The system utilizes a parametric 3D deformable human mesh model (SMPL-X) as a representation and leverages the Kinect Azure RGB-D camera for data input. To train the facial expression extractor, the authors annotate a large talking face dataset with facial expression parameters. For the body pose, they collect and annotate a dataset of 56 people captured from a rig of 5 Kinect Azure RGB-D cameras, in addition to utilizing a large motion capture AMASS dataset. The entire system runs at 30 frames per second on a server with a single GPU.

Results

The results show that the RGB-D body pose model outperforms state-of-the-art RGB-only methods while achieving comparable accuracy to slower RGB-D optimization based solutions. In summary, this paper presents an advanced system for real time extended body pose estimation that incorporates accurate estimations of body poses, hand poses and facial expressions using RGB D data inputs which demonstrates improved accuracy compared to rgb only methods while achieving real time performance on standard hardware configurations