# Any-to-Many Voice Conversion with Location-Relative Sequence-to-Sequence Modeling

**Authors:** Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, Helen Meng  
**arXiv:** [2009.02725](https://arxiv.org/abs/2009.02725)  
**Date:** 2020-09-06  
**Summary page:** https://www.summarizepaper.com/en/paper/2009.02725v3/

## Key points

- Introduces a new approach to any-to-many voice conversion using location-relative, sequence-to-sequence (seq2seq) modeling
- Utilizes text supervision during training and combines a bottle-neck feature extractor (BNE) with a seq2seq synthesis module
- Trains an encoder-decoder based hybrid connectionist temporal classification attention (CTC-attention) phoneme recognizer with a bottle neck layer in its encoder during the training stage
- Uses the BNE to extract speaker independent, dense and rich spoken content representations from spectral features
- Trains a multi-speaker location relative attention based seq2seq synthesis model to reconstruct spectral features from the bottle neck features, which conditions on speaker representations for speaker identity control in the generated speech
- Down samples input spectral feature along the temporal dimension to overcome difficulties of aligning long sequences using seq2seq models
- Equips synthesis model with a discretized mixture of logistic (MoL) attention mechanism
- Can conduct any-to-many voice conversion since the phoneme recognizer is trained with large speech recognition data corpus
- Outperforms existing methods in terms of both naturalness and speaker similarity according to objective and subjective evaluations
- Feature selection and model design strategies are effective in this approach according to ablation studies
- Proposed VC approach can be extended to support any-to-any VC achieving high performance according to objective and subjective evaluations
- Presents an innovative solution for any-to-many voice conversion that can potentially improve various applications such as virtual assistants or personalized voice services.

## Summary

This paper introduces a novel approach to any-to-many voice conversion using location-relative, sequence-to-sequence (seq2seq) modeling. The proposed method utilizes text supervision during training and combines a bottle-neck feature extractor (BNE) with a seq2seq synthesis module. During the training stage, an encoder-decoder based hybrid connectionist temporal classification attention (CTC-attention) phoneme recognizer is trained which has a bottle neck layer in its encoder. A BNE is obtained from the phoneme recognizer and used to extract speaker independent, dense and rich spoken content representations from spectral features. To reconstruct spectral features from the bottle neck features, a multi-speaker location relative attention based seq2seq synthesis model is trained. This model conditions on speaker representations for speaker identity control in the generated speech. To overcome the difficulties of aligning long sequences using seq2seq models, the input spectral feature is down sampled along the temporal dimension. Additionally, the synthesis model is equipped with a discretized mixture of logistic (MoL) attention mechanism. The proposed approach can conduct any-to-many voice conversion since the phoneme recognizer is trained with large speech recognition data corpus. Objective and subjective evaluations show that this approach outperforms existing methods in terms of both naturalness and speaker similarity. Ablation studies confirm that feature selection and model design strategies are effective in this approach. Furthermore, this proposed VC approach can be extended to support any-to-any VC (also known as one/few shot VC), achieving high performance according to objective and subjective evaluations. Overall, this study presents an innovative solution for any-to-many voice conversion that can potentially improve various applications such as virtual assistants or personalized voice services.