# StreamVC: Real-Time Low-Latency Voice Conversion

**Authors:** Yang Yang, Yury Kartynnik, Yunpeng Li, Jiuqiang Tang, Xing Li, George Sung, Matthias Grundmann  
**arXiv:** [2401.03078](https://arxiv.org/abs/2401.03078)  
**Date:** 2024-01-05  
**Summary page:** https://www.summarizepaper.com/en/paper/2401.03078v1/

## Key points

- StreamVC is a state-of-the-art streaming voice conversion solution
- Matches voice timbre of target speech while preserving content and prosody of source speech
- Generates resulting waveform with low latency directly from input signal, ideal for real-time communication scenarios
- Developed based on SoundStream neural audio codec architecture and training strategy
- Boasts lightweight yet high-quality speech synthesis capabilities
- Learns soft speech units causally to preserve pitch stability without compromising source timbre information
- Utilizes LibriTTS train-clean-100 subset to derive cluster centroids for HuBERT pseudo-labels in training dataset
- Extensive training over 1.3 million steps with batch size of 128
- Evaluation dataset created to assess generalization capabilities with unseen source and target speakers during training
- Comprehensive evaluations conducted on pairs of source and target speech utterances from various datasets like LibriTTS and VCTK

## Summary

StreamVC is a state-of-the-art streaming voice conversion solution that seamlessly matches the voice timbre of a target speech while preserving the content and prosody of the source speech. Unlike previous approaches, this innovative technology generates the resulting waveform with low latency directly from the input signal, making it ideal for real-time communication scenarios such as calls and video conferencing where voice anonymization may be necessary. Developed based on the architecture and training strategy of the SoundStream neural audio codec, StreamVC boasts lightweight yet high-quality speech synthesis capabilities. A key aspect highlighted in this research is its ability to learn soft speech units causally, effectively preserving pitch stability without compromising source timbre information. To ensure optimal performance, StreamVC utilizes the LibriTTS train-clean-100 subset to derive cluster centroids for HuBERT pseudo-labels, creating a robust training dataset consisting of 555.15 hours of speech from 2311 speakers. The model undergoes extensive training over 1.3 million steps with a batch size of 128. To evaluate its generalization capabilities, an evaluation dataset is created where both source and target speakers are unseen during training. Comprehensive evaluations are conducted on a significant number of pairs of source and target speech utterances selected from various datasets like LibriTTS and VCTK. Baseline models and evaluation metrics are employed to assess the effectiveness and efficiency of StreamVC in comparison to existing methods. The results showcase promising advancements in real-time low-latency voice conversion technology, paving the way for enhanced communication experiences across diverse applications.