# WESPER: Zero-shot and Realtime Whisper to Normal Voice Conversion for Whisper-based Speech Interactions

**Authors:** Jun Rekimoto  
**arXiv:** [2303.01639](https://arxiv.org/abs/2303.01639)  
**Date:** 2023-03-03  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.01639v1/

## Key points

- Recognizing whispered speech and converting it to normal speech opens up possibilities for semi-silent communication in public settings without disturbing others
- Traditional methods of speech conversion often fall short in quality or rely on speaker-dependent datasets
- WESPER is a zero-shot, real-time whisper-to-normal speech conversion mechanism using self-supervised learning techniques
- WESPER comprises a Speech-to-Unit (STU) encoder and a Unit-to-Speech (UTS) decoder for reconstructing original speech from hidden units
- WESPER eliminates the need for paired datasets specific to individual speakers, making the process user-independent
- WESPER can reconstruct converted speech in any target speaker's voice using only unlabeled data from that speaker
- WESPER enhances the quality of converted whispers while preserving their prosodic characteristics, benefiting individuals with speech or hearing impairments

## Summary

, , , , 

In the field of speech interaction, recognizing whispered speech and converting it to normal speech opens up a plethora of possibilities. Whispered speech, characterized by significantly lower sound pressure compared to normal speech, can be utilized as a semi-silent form of communication in public settings without disturbing others. This not only enhances overall speech quality but also proves beneficial for individuals with speech or hearing impairments. Traditional methods of speech conversion often fall short in terms of conversion quality or rely on speaker-dependent datasets containing pairs of whispered and normal speech utterances. To overcome these limitations, a groundbreaking solution called WESPER has been introduced. WESPER stands out as a zero-shot, real-time whisper-to-normal speech conversion mechanism that leverages self-supervised learning techniques. At the core of WESPER lies a sophisticated architecture comprising a Speech-to-Unit (STU) encoder and a Unit-to-Speech (UTS) decoder. The STU encoder generates hidden speech units common to both whispered and normal speech, while the UTS decoder reconstructs the original speech from these encoded units. Notably, WESPER's innovative approach eliminates the need for paired datasets specific to individual speakers, making the conversion process user-independent. One of the key advantages of WESPER is its ability to reconstruct converted speech in any target speaker's voice using only unlabeled data from that speaker. This remarkable feature underscores the versatility and adaptability of WESPER in catering to diverse user needs without compromising on conversion quality or natural prosody. Extensive testing has confirmed that WESPER successfully enhances the quality of converted whispers while preserving their inherent prosodic characteristics. Furthermore, the proposed approach has demonstrated promising results in facilitating effective speech reconstruction for individuals with speech or hearing disabilities. Overall, WESPER represents a significant advancement in whisper-based speech interactions, offering an efficient and user-friendly solution that holds immense potential for revolutionizing communication technologies and improving accessibility for individuals with diverse communication needs.