# A Multi-Stage Multi-Codebook VQ-VAE Approach to High-Performance Neural TTS

**Authors:** Haohan Guo, Fenglong Xie, Frank K. Soong, Xixin Wu, Helen Meng  
**arXiv:** [2209.10887](https://arxiv.org/abs/2209.10887)  
**Date:** 2022-09-22  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.10887v1/

## Key points

- Multi-Stage, Multi-Codebook (MSMC) approach to high-performance neural Text-to-Speech (TTS) synthesis
- Vector-quantized, variational autoencoder (VQ-VAE) based feature analyzer for encoding Mel spectrograms
- Progressive down-sampling of spectrograms in multiple stages to create MSMC Representations (MSMCRs)
- Quantization of MSMCRs using multiple VQ codebooks
- Training with multi-stage predictors and a combined loss function including reconstruction MSE and triplet loss
- Use of a neural vocoder for synthesizing speech waveforms from predicted MSMCRs
- Evaluation on an English TTS database, achieving higher MOS scores than the baseline system
- Compact versions of the proposed TTS system also maintain high MOS scores
- Ablation studies show that both multiple stages and multiple codebooks contribute to performance improvement.

## Summary

The paper presents a Multi-Stage, Multi-Codebook (MSMC) approach to high-performance neural Text-to-Speech (TTS) synthesis. The authors propose using a vector-quantized, variational autoencoder (VQ-VAE) based feature analyzer to encode Mel spectrograms of speech training data. This encoding process involves progressively down-sampling the spectrograms in multiple stages, resulting in MSMC Representations (MSMCRs) with different time resolutions. These MSMCRs are then quantized using multiple VQ codebooks. To train the system, multi-stage predictors are employed to map the input text sequence to MSMCRs progressively. This is achieved by minimizing a combined loss function that includes both reconstruction Mean Square Error (MSE) and "triplet loss". The triplet loss helps improve the discriminative power of the encoded representations. During synthesis, a neural vocoder is used to convert the predicted MSMCRs into final speech waveforms. The proposed approach is evaluated using an English TTS database consisting of 16 hours of speech data recorded by a female speaker. The results show that the proposed TTS system achieves an MOS score of 4.41, outperforming the baseline system with an MOS score of 3.62. Furthermore, the authors demonstrate that even compact versions of their proposed TTS system, which have significantly fewer parameters, can still preserve high MOS scores. Ablation studies conducted on the system reveal that both multiple stages and multiple codebooks contribute to achieving high TTS performance. In summary, this paper introduces a novel approach for neural TTS synthesis using a Multi-Stage, Multi-Codebook framework. The experimental results demonstrate its effectiveness in producing high-quality synthetic speech with improved MOS scores compared to baseline systems.