# ViTPose++: Vision Transformer for Generic Body Pose Estimation

**Authors:** Yufei Xu, Jing Zhang, Qiming Zhang, Dacheng Tao  
**arXiv:** [2212.04246](https://arxiv.org/abs/2212.04246)  
**Date:** 2022-12-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.04246v3/

## Key points

- Authors Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao explore the capabilities of plain vision transformers in body pose estimation
- ViTPose model leverages simplicity, scalability, flexibility, and transferability of knowledge within vision transformers
- ViTPose utilizes non-hierarchical vision transformer as encoder and lightweight decoder for top-down or bottom-up decoding
- Model scalable from 20M to 1B parameters while maintaining high performance and efficiency
- ViTPose demonstrates flexibility in attention type, input resolution, pre-training strategies, and fine-tuning methods
- Introduction of ViTPose+ extension addresses heterogeneous body keypoint categories through knowledge factorization with task-specific feed-forward networks
- Empirical evidence supports ease of transferring knowledge from large ViTPose models to smaller ones using a simple knowledge token
- Experimental results show superior performance of ViTPose on challenging benchmarks like MS COCO Human Keypoint Detection at both top-down and bottom-up settings
- ViTPose+ achieves state-of-the-art results across various body pose estimation tasks without compromising on inference speed

## Summary

, , , , 

In their paper titled "ViTPose++: Vision Transformer for Generic Body Pose Estimation," authors Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao explore the capabilities of plain vision transformers in the context of body pose estimation. They introduce ViTPose as a baseline model that leverages the simplicity, scalability, flexibility, and transferability of knowledge within vision transformers. Specifically, ViTPose utilizes a non-hierarchical vision transformer as an encoder to extract features and a lightweight decoder to decode body keypoints in either a top-down or bottom-up fashion. The model can be scaled from 20M to 1B parameters, showcasing its ability to achieve high performance while maintaining efficiency. Moreover, ViTPose demonstrates flexibility in attention type, input resolution, pre-training strategies, and fine-tuning methods. The authors also introduce a novel extension called ViTPose+ which addresses heterogeneous body keypoint categories in different pose estimation tasks through knowledge factorization. This involves incorporating task-agnostic and task-specific feed-forward networks within the transformer architecture. The study showcases empirical evidence supporting the ease of transferring knowledge from large ViTPose models to smaller ones using a simple knowledge token. Experimental results highlight the superior performance of ViTPose compared to existing methods on challenging benchmarks such as MS COCO Human Keypoint Detection at both top-down and bottom-up settings. Furthermore, the ViTPose+ model achieves state-of-the-art results across various body pose estimation tasks including MS COCO, AI Challenger, OCHuman, MPII for human keypoint detection, COCO-Wholebody for whole-body keypoint detection, as well as AP-10K and APT-36K for animal keypoint detection without compromising on inference speed. This research significantly advances the field of body pose estimation by demonstrating the effectiveness of vision transformers in handling diverse keypoint categories with high performance and efficiency.