# F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching

**Authors:** Yushen Chen, Zhikang Niu, Ziyang Ma, Keqi Deng, Chunhui Wang, Jian Zhao, Kai Yu, Xie Chen  
**arXiv:** [2410.06885](https://arxiv.org/abs/2410.06885)  
**Date:** 2024-10-09  
**Summary page:** https://www.summarizepaper.com/en/paper/2410.06885v1/

## Key points

- F5-TTS is a fully non-autoregressive text-to-speech system based on flow matching with Diffusion Transformer (DiT).
- It simplifies the process by padding the text input with filler tokens to match the length of the input speech, eliminating the need for complex designs like duration models and phoneme alignment.
- The model uses denoising techniques initially proven feasible by E2 TTS but addresses challenges of slow convergence and low robustness through refined input text representation using ConvNeXt and an inference-time Sway Sampling strategy.
- F5-TTS achieves faster training and improved inference RTF of 0.15 compared to state-of-the-art diffusion-based TTS models, showcasing natural, expressive zero-shot ability, seamless code-switching capability, and efficient speed control.
- The model produces fluent and faithful speech outputs, demonstrating versatility and effectiveness in synthesizing high-quality speech.
- Authors provide demo samples at https://SWivid.github.io/F5-TTS and release all code and checkpoints for community development.
- Ethics considerations include potential risks like voice identification spoofing, recommending watermarks implementation and audio output detection mechanisms.
- Detailed ablation studies were conducted to evaluate F5-TTS's efficiency compared to E2 TTS, highlighting the importance of refining input representations for better alignment with speech modalities.

## Summary

In this research project, the authors introduce F5-TTS, a fully non-autoregressive text-to-speech system that is based on flow matching with Diffusion Transformer (DiT). Unlike traditional models that require complex designs such as duration models, text encoders, and phoneme alignment, F5-TTS simplifies the process by padding the text input with filler tokens to match the length of the input speech. This denoising technique for speech generation was initially proven feasible by E2 TTS; however, E2 TTS faced challenges with slow convergence and low robustness. To address these issues, the authors first refine the input text representation using ConvNeXt to improve alignment with speech. They also propose an inference-time Sway Sampling strategy to enhance model performance and efficiency. This sampling strategy can be easily applied to existing flow matching-based models without requiring retraining. As a result of these enhancements, F5-TTS achieves faster training and an improved inference RTF of 0.15 compared to state-of-the-art diffusion-based TTS models. Trained on a public 100K hours multilingual dataset, F5-TTS demonstrates highly natural and expressive zero-shot ability, seamless code-switching capability, and efficient speed control. The model produces fluent and faithful speech with Flow matching (F5-TTS), showcasing its versatility and effectiveness in synthesizing high-quality speech outputs. The authors provide demo samples at https://SWivid.github.io/F5-TTS and release all code and checkpoints to encourage further community development. Additionally, the authors acknowledge Tianrui Wang, Xiaofei Wang, Yakun Song, Yifan Yang, Yiwei Guo, and Yunchong Xiao for their valuable discussions during the project. In terms of ethics considerations, given the potential risks associated with misuse of the model such as voice identification spoofing, it is recommended to implement watermarks and develop detection mechanisms for audio outputs. Furthermore, detailed ablation studies were conducted to evaluate F5-TTS's efficiency compared to E2 TTS. Small models were trained on a Mandarin dataset using different configurations to assess alignment learning capabilities. The experiments highlighted the importance of refining input representations for better alignment with speech modalities. Overall, this paper presents a comprehensive analysis of F5-TTS's architecture and performance improvements over existing models in the field of text-to-speech synthesis.