# VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

**Authors:** Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, David Harwath  
**arXiv:** [2403.16973](https://arxiv.org/abs/2403.16973)  
**Date:** 2024-03-25  
**Summary page:** https://www.summarizepaper.com/en/paper/2403.16973v1/

## Key points

- VoiceCraft is a groundbreaking neural codec language model achieving state-of-the-art performance in audio applications.
- Developed by Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath using Transformer decoder architecture and unique token rearrangement procedure.
- Demonstrates ability to produce edited speech nearly indistinguishable from unedited recordings in terms of naturalness.
- Outperforms previous state-of-the-art models like VALLE and XTTS-v2 in diverse scenarios with different accents, speaking styles, recording conditions, background noise, and music.
- Introduction of RealEdit dataset for rigorous testing under various conditions.
- Detailed results show impressive scores in both objective metrics like speaker similarity and subjective human evaluation metrics.
- Slight gap in naturalness compared to ground truth recordings on platforms like YouTube utterances but overall highly competitive performance.

## Summary

VoiceCraft is a groundbreaking  neural codec language model that has achieved state-of-the-art performance in both  and  applications on various types of audio content such as audiobooks, internet videos, and podcasts. Developed by Puyuan Peng, Po-Yao Huang, Daniel Li, Abdelrahman Mohamed, and David Harwath, VoiceCraft utilizes a Transformer decoder architecture along with a unique token rearrangement procedure that combines causal masking and delayed stacking to enable seamless generation within an existing sequence. In terms of  tasks, VoiceCraft has demonstrated the ability to produce edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness. Human evaluations have confirmed the high quality of the edited output. Additionally, when it comes to , VoiceCraft outperforms previous state-of-the-art models including VALLE and XTTS-v2. The model excels in diverse scenarios with different accents, speaking styles, recording conditions, background noise, and music. One notable aspect of VoiceCraft's evaluation is the introduction of a challenging and realistic dataset called RealEdit for . This dataset ensures that the model's performance is rigorously tested under various conditions. The authors encourage readers to explore demos showcasing VoiceCraft's capabilities at https://jasonppy.github.io/VoiceCraft_web. Furthermore, detailed results from  reveal that VoiceCraft achieves impressive scores in both objective metrics like speaker similarity and subjective human evaluation metrics. The model demonstrates exceptional performance in terms of intelligibility and speaker similarity compared to ground truth data. While there may be a slight gap in naturalness between VoiceCraft-generated speech and ground truth recordings on platforms like YouTube utterances, overall performance remains highly competitive. Overall, VoiceCraft represents a significant advancement in the field of speech processing technology with its ability to deliver top-tier results across various applications in real-world settings. Its innovative approach and superior performance make it a valuable tool for professionals working with audio content across different mediums.