# Real-time RGBD-based Extended Body Pose Estimation

**Authors:** Renat Bashirov, Anastasia Ianina, Karim Iskakov, Yevgeniy Kononenko, Valeriya Strizhkova, Victor Lempitsky, Alexander Vakhitov  
**arXiv:** [2103.03663](https://arxiv.org/abs/2103.03663)  
**Date:** 2021-03-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2103.03663v1/

## Key points

- System for real-time RGBD-based estimation of 3D human pose
- Focus on body pose, hand pose, and facial expression
- Utilizes parametric 3D deformable human mesh model (SMPL-X) and Kinect Azure RGB-D camera
- Estimators trained for body pose, facial expression parameters using landmark extractors and custom annotated datasets
- Hand pose estimated using a previously published method
- Predictions combined to generate temporally-smooth human pose
- Facial expression extractor trained with annotated talking face dataset
- Body pose dataset collected and annotated from 56 people captured by 5 Kinect Azure RGB-D cameras, in addition to utilizing a large motion capture AMASS dataset
- Results show outperformance of RGB-D body pose model compared to state-of-the-art RGB-only methods, comparable accuracy to slower RGB-D optimization-based solutions
- Entire system runs at 30 frames per second on a server with a single GPU
- Advanced system for real-time extended body pose estimation incorporating accurate estimations of body pose, hand pose, and facial expressions using RGBD data inputs
- Demonstrates improved accuracy compared to RGB-only methods and achieves real-time performance on standard hardware configurations.

## Summary

This paper presents a system for real-time RGBD-based estimation of 3D human pose, focusing on the body pose, hand pose and facial expression. The system utilizes a parametric 3D deformable human mesh model (SMPL-X) as a representation and leverages the Kinect Azure RGB-D camera for data input. The authors train estimators for body pose and facial expression parameters using previously published landmark extractors and custom annotated datasets. Hand pose is estimated directly using a previously published method. The predictions from these estimators are combined to generate a temporally-smooth human pose. To train the facial expression extractor, the authors annotate a large talking face dataset with facial expression parameters. For the body pose, they collect and annotate a dataset of 56 people captured from a rig of 5 Kinect Azure RGB-D cameras, in addition to utilizing a large motion capture AMASS dataset. The results show that the RGB-D body pose model outperforms state-of-the-art RGB-only methods while achieving comparable accuracy to slower RGB-D optimization-based solutions. The entire system runs at 30 frames per second on a server with a single GPU. In summary, this paper presents an advanced system for real-time extended body pose estimation that incorporates accurate estimations of body pose, hand pose and facial expressions using RGBD data inputs. The system demonstrates improved accuracy compared to RGB-only methods and achieves real-time performance on standard hardware configurations.