# Vec-Tok-VC+: Residual-enhanced Robust Zero-shot Voice Conversion with Progressive Constraints in a Dual-mode Training Strategy

**Authors:** Linhan Ma, Xinfa Zhu, Yuanjun Lv, Zhichao Wang, Ziqian Wang, Wendi He, Hongbin Zhou, Lei Xie  
**arXiv:** [2406.09844](https://arxiv.org/abs/2406.09844)  
**Date:** 2024-06-14  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.09844v1/

## Key points

- Recent advancements in zero-shot voice conversion aim to transform source speech into an unseen target voice while maintaining linguistic content
- Challenges such as semantic losses and training-inference mismatch impact performance in this field
- Vec-Tok-VC+ is a novel prompt-based model that excels in achieving voice conversion with just a 3-second target speaker prompt
- The model features a residual-enhanced K-Means decoupler and teacher-guided refinement to address training-inference mismatch
- A multi-codebook progressive loss function is designed to improve speaker similarity and content accuracy
- Objective and subjective evaluations show that Vec-Tok-VC+ outperforms strong baselines in naturalness, intelligibility, and speaker similarity
- The research paper on Vec-Tok-VC+ has been accepted by INTERSPEECH2024, highlighting its significant advancement in robust zero-shot voice conversion with progressive constraints implemented within a dual-mode training strategy

## Summary

In the field of zero-shot voice conversion (VC), recent advancements have been made to transform source speech into an arbitrary unseen target voice while maintaining linguistic content. However, challenges such as semantic losses and training-inference mismatch continue to impact performance. To address these issues, researchers have introduced Vec-Tok-VC+, a novel prompt-based model that excels in achieving voice conversion with just a 3-second target speaker prompt. The key innovation lies in its residual-enhanced K-Means decoupler and teacher-guided refinement for eliminating training-inference mismatch. Furthermore, a multi-codebook progressive loss function is designed to improve speaker similarity and content accuracy. Objective and subjective evaluations have shown that Vec-Tok-VC+ surpasses strong baselines in naturalness, intelligibility, and speaker similarity. This research paper has been accepted by INTERSPEECH2024 and showcases how Vec-Tok-VC+ represents a significant advancement in robust zero-shot voice conversion with progressive constraints implemented within a dual-mode training strategy.