# SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models

**Authors:** Xin Zhang, Dong Zhang, Shimin Li, Yaqian Zhou, Xipeng Qiu  
**arXiv:** [2308.16692](https://arxiv.org/abs/2308.16692)  
**Date:** 2023-08-31  
**Summary page:** https://www.summarizepaper.com/en/paper/2308.16692v2/

## Key points

- Current large language models in speech language modeling rely on discrete speech representations
- Speech tokens are categorized into semantic and acoustic tokens but not optimized for speech language modeling
- SLMTokBench benchmark evaluates effectiveness of different types of speech tokens
- SpeechTokenizer proposed to unify semantic and acoustic tokens using an Encoder-Decoder architecture with RVQ
- Unified Speech Language Model (USLM) developed based on SpeechTokenizer outperforms VALL-E in zero-shot Text-to-Speech tasks
- Innovative techniques such as RVQ-based disentanglement and unified tokenization offer promising solutions for advancing large-scale speech language models
- Availability of code and models on GitHub facilitates accessibility and collaboration within the research community

## Summary

In the realm of speech language modeling, current large language models rely on discrete speech representations. These representations are typically categorized into semantic tokens and acoustic tokens. However, these existing speech tokens are not specifically optimized for speech language modeling. To address this limitation, a benchmark known as SLMTokBench was established to evaluate the effectiveness of different types of speech tokens in building speech language models. The results from this benchmark indicate that neither semantic nor acoustic tokens are ideal for this purpose. In response to these findings, a novel approach called SpeechTokenizer was proposed. This unified speech tokenizer leverages an Encoder-Decoder architecture with residual vector quantization (RVQ) to disentangle various aspects of speech information hierarchically across different RVQ layers. By unifying semantic and acoustic tokens, SpeechTokenizer aims to overcome the limitations of using multiple models to extract these discrete tokens separately. Furthermore, a Unified Speech Language Model (USLM) was developed based on SpeechTokenizer. Experimental results demonstrate that SpeechTokenizer performs comparably to existing models in terms of speech reconstruction and exhibits strong performance on the SLMTokBench benchmark. Additionally, USLM outperforms VALL-E in zero-shot Text-to-Speech tasks. The study also delves into related work in the field of speech language modeling and highlights the importance of efficiency and quality in further developing these models. Through innovative techniques such as RVQ-based disentanglement and unified tokenization, SpeechTokenizer offers a promising solution for advancing the capabilities of large-scale speech language models. The availability of code and models on GitHub further facilitates accessibility and collaboration within the research community towards achieving more efficient and high-quality speech generation outcomes. Overall, this study underscores the significance of specialized  designed specifically for . By proposing a unified approach with SpeechTokenizer and USLM, the research aims to enhance both content accuracy and quality in generated speech while streamlining the modeling process.