# HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesis

**Authors:** Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, Seong-Whan Lee  
**arXiv:** [2311.12454](https://arxiv.org/abs/2311.12454)  
**Date:** 2023-11-21  
**Summary page:** https://www.summarizepaper.com/en/paper/2311.12454v1/

## Key points

- Revolutionary approach to zero-shot speech synthesis
- Utilizes hierarchical speech synthesis frameworks for enhanced expressiveness and stability
- Achieves human-level quality in zero-shot speech synthesis with 10k hours of data
- Various objective metrics used for evaluation, including Mel error distance, PESQ, pitch, periodicity, voice/unvoice F1 score, and log-scale F0 consistency
- Subjective metrics like nMOS and voice similarity MOS used for VC tasks
- Ablation studies conducted to assess effectiveness of each component
- Significant improvement in zero-shot speech synthesis performance compared to previous models
- HierVST enhances voice style transfer capabilities
- Prosody MOS ratings validate naturalness of synthetic speech samples
- Availability of audio samples and source code on GitHub for further research and development

## Summary

is a revolutionary approach to zero-shot speech synthesis, providing a fast and robust solution for text-to-speech (TTS) and voice conversion (VC) tasks. By utilizing hierarchical speech synthesis frameworks, the system enhances the expressiveness and stability of synthetic speech, surpassing Large Language Models (LLM) and diffusion-based models. Despite being limited to 10k hours of data due to resource constraints,  achieves human-level quality in zero-shot speech synthesis. Various objective metrics were used to evaluate the system's performance in reconstruction and resynthesis tasks, including Mel error distance, perceptual evaluation of speech quality (PESQ), pitch, periodicity, voice/unvoice F1 score, and log-scale F0 consistency. For VC tasks, subjective metrics such as naturalness mean opinion score (nMOS) and voice similarity MOS were used alongside objective metrics like UTMOS, character error rate (CER), word error rate (WER), automatic speaker verification equal error rate (EER), and speaker encoder cosine similarity. Ablation studies were conducted to assess the effectiveness of each component in . The system's performance in zero-shot speech synthesis was significantly improved compared to previous end-to-end models, with HierVST enhancing voice style transfer capabilities. Additionally, prosody MOS ratings further validated the naturalness of synthetic speech samples. Overall, presents a significant advancement in zero-shot speech synthesis technology, showcasing superior performance in TTS and VC tasks while addressing key limitations of existing autoregressive models. The availability of audio samples and source code on GitHub facilitates further research and development in this field.