# UniMuMo: Unified Text, Music and Motion Generation

**Authors:** Han Yang, Kun Su, Yutong Zhang, Jiaben Chen, Kaizhi Qian, Gaowen Liu, Chuang Gan  
**arXiv:** [2410.04534](https://arxiv.org/abs/2410.04534)  
**Date:** 2024-10-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.04534v1/

## Key points

- Authors introduce UniMuMo - a multimodal model for text, music, and motion generation
- Model aligns unpaired music and motion datasets using rhythmic patterns
- Inputs are converted into token-based representations using unified encoder-decoder transformer architecture
- Architectural enhancements support multiple generation tasks within a single framework
- Motion is encoded using a music codebook to map it into the same feature space as music
- Proposal of a novel music-motion parallel generation scheme consolidating all tasks into a single transformer decoder architecture

## Summary

, , , , 

In their paper titled "UniMuMo: Unified Text, Music and Motion Generation," authors Han Yang, Kun Su, Yutong Zhang, Jiaben Chen, Kaizhi Qian, Gaowen Liu, and Chuang Gan introduce UniMuMo - a groundbreaking multimodal model that seamlessly processes diverse input data types such as text, music, and motion to generate outputs across all three modalities. To overcome the challenge of lacking time-synchronized data, the researchers align unpaired music and motion datasets by leveraging rhythmic patterns from existing large-scale music-only and motion-only datasets. The model operates by converting text, music, and motion inputs into token-based representations using a unified encoder-decoder transformer architecture. Several architectural enhancements are introduced to support multiple generation tasks within a single framework. One key innovation is encoding motion using a music codebook to map it into the same feature space as music. Furthermore, the researchers propose a novel music-motion parallel generation scheme that consolidates all music and motion generation tasks into a single transformer decoder architecture with a unified training task of joint music-motion generation.