# STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting

**Authors:** Zenghao Chai, Chen Tang, Yongkang Wong, Mohan Kankanhalli  
**arXiv:** [2406.04629](https://arxiv.org/abs/2406.04629)  
**Date:** 2024-06-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.04629v1/

## Key points

- Significant progress in text-to-image (T2I) generative models
- Creation of 4D avatars with authentic human motions valuable in film and gaming industries
- Challenges in existing methods for text-based 4D avatar generation: shape diversity limitations and animation artifacts
- Introduction of a new approach called STAR to address challenges
- STAR incorporates geometry and skeleton differences, motion retargeting techniques, T2I and T2V priors, and SDS module for coherent optimization
- STAR can synthesize high-quality 4D avatars with vivid animations aligned with text descriptions
- Ablation studies highlight contributions of each component in STAR
- Novel approach to generating realistic 4D avatars from textual input
- For more information, including source code and demos, visit the project website at https://star-avatar.github.io.

## Summary

In recent years, there has been significant progress in text-to-image (T2I) generative models, leading to the creation of 3D content from arbitrary text descriptions. This intersection of computer vision and graphics communities has sparked interest in generating human-like characters from text. Building on this progress, the creation of 4D avatars – animatable characters with authentic human motions – has become valuable in industries such as film and gaming. Existing methods for text-based 4D avatar generation follow an optimization-by-animation paradigm, where image priors are used to generate 3D avatars from text descriptions, followed by the application of desired human motions. However, this approach faces challenges such as shape diversity limitations and animation artifacts. To address these challenges, a new approach called  is proposed. STAR takes into account the geometry and skeleton differences between template meshes and target avatars and corrects mismatched source motions using pretrained motion retargeting techniques. By incorporating skeleton-conditioned T2I and text-to-video (T2V) priors, STAR introduces a hybrid Score Distillation Sampling (SDS) module to provide multi-view and frame-consistent supervision signals for coherent optimization of geometry, texture, and motion in an end-to-end manner. Quantitative and qualitative experiments demonstrate that STAR can synthesize high-quality 4D avatars with vivid animations that align well with text descriptions. Ablation studies highlight the contributions of each component in STAR. The proposed method offers a novel approach to generating realistic 4D avatars from textual input. For more information, including source code and demos, visit the project website at https://star-avatar.github.io.