# Learning Human Motion Representations: A Unified Perspective

**Authors:** Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, Yizhou Wang  
**arXiv:** [2210.06551](https://arxiv.org/abs/2210.06551)  
**Date:** 2022-10-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2210.06551v2/

## Key points

- There is a need for a unified perspective in human-centric video tasks
- The proposed framework includes a pretraining stage that trains a motion encoder to recover 3D motion from noisy partial 2D observations
- Motion representations incorporate geometric, kinematic, and physical knowledge about human motion, making them easily transferable to multiple downstream tasks
- Dual-stream Spatio-temporal Transformer (DSTformer) neural network was used to implement the motion encoder
- Heterogeneity of available data resources is a significant challenge in developing a unified representation
- The proposed framework achieves state-of-the-art performance on all three downstream tasks by fine-tuning the pretrained motion encoder with a simple regression head
- The framework has potential for future applications as it mines and utilizes commonalities across different tasks despite having models designed for different problems that have learned typical human motion patterns.

## Summary

In the field of human-centric video tasks, there is a need for a unified perspective that can tackle various challenges by learning human motion representations from large-scale and heterogeneous data resources. This is where the proposed framework comes in, which presents a pretraining stage that trains a motion encoder to recover 3D motion from noisy partial 2D observations. The resulting motion representations incorporate geometric, kinematic, and physical knowledge about human motion, making them easily transferable to multiple downstream tasks. To implement the motion encoder, the Dual-stream Spatio-temporal Transformer (DSTformer) neural network was used. It captures long-range spatio-temporal relationships among skeletal joints comprehensively and adaptively, leading to the lowest 3D pose estimation error when trained from scratch. One significant challenge in developing a unified representation is the heterogeneity of available data resources. Motion capture systems provide high-fidelity 3D motion data but are limited in terms of captured video appearances. Action recognition datasets offer annotations of action semantics but lack human pose labels or feature limited daily activities' motions. In contrast, in-the-wild human videos offer diverse range appearance and motion but require considerable effort to obtain precise 2D pose annotations. The proposed framework addresses this challenge by providing a new perspective on learning human motion representations that can be shared across all relevant tasks. It achieves state-of-the-art performance on all three downstream tasks by simply fine-tuning the pretrained motion encoder with a simple regression head (1–2 layers), demonstrating the versatility of learned motion representations. The proposed framework has potential for future applications as it mines and utilizes commonalities across different tasks despite having models designed for different problems that have learned typical human motion patterns. In conclusion, this work provides an innovative approach to tackling various human-centric video tasks using learned humanmotion representations from large scale and heterogeneous data resources through pretraining and fine tuning with a simple regression head. The proposed framework is versatile and achieves stateof -the art performance on all three downstream tasks demonstrating its potential for future applications.