# Cross-Attention is all you need: Real-Time Streaming Transformers for Personalised Speech Enhancement

**Authors:** Shucong Zhang, Malcolm Chadwick, Alberto Gil C. P. Ramos, Sourav Bhattacharya  
**arXiv:** [2211.04346](https://arxiv.org/abs/2211.04346)  
**Date:** 2022-11-08  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.04346v1/

## Key points

- The paper presents a novel approach to personalised speech enhancement (PSE) using a streaming Transformer-based model with adaptive target speaker representations
- PSE is the process of extracting only the speech of a target user and removing everything else from an audio clip, which can improve users' experiences of audio AI modules in real-world applications such as real-time automatic speech recognition (ASR) and audio-call enhancement
- The proposed model uses cross-attention to dynamically adapt the target speaker representation for each input frame in real time, instead of using fixed embeddings
- Extensive experiments were conducted on two different sizes of Transformer models and compared them with bidirectional VoiceFilter models and bidirectional cross-attention models
- The results show that their proposed cross-attention approach consistently outperforms competitive baselines even when their model is only approximately half the size
- Their approach achieves better performance than fixed embeddings in scenarios where there are changes in background noise or speakers' voices
- This work presents an innovative solution for improving PSE performance in real world applications by dynamically adapting target speaker representations using cross attention in a streaming Transformer based model.

## Summary

In this paper, the authors present a novel approach to personalised speech enhancement (PSE) using a streaming Transformer-based model with adaptive target speaker representations. PSE is the process of extracting only the speech of a target user and removing everything else from an audio clip, which can improve users' experiences of audio AI modules in real-world applications such as real-time automatic speech recognition (ASR) and audio-call enhancement. To achieve personalisation, a target speaker's voice profile is typically extracted from an enrolment audio as a static embedding vector and used to condition the output of a PSE model. However, fixed embeddings may not be optimal under all conditions. The proposed model uses cross-attention to dynamically adapt the target speaker representation for each input frame in real time. The authors conducted extensive experiments on two different sizes of Transformer models and compared them with bidirectional VoiceFilter models and bidirectional cross-attention models. They trained their models for 100/200 epochs with batch sizes of 320/640 on 350 hours of training data, following an optimizer with 16k warm-up steps. The training objective was to reduce the power-law compressed spectral distance between cleaned spectrograms produced by their models and ground truth data. The results show that their proposed cross-attention approach consistently outperforms competitive baselines even when their model is only approximately half the size. The authors also demonstrate that their approach achieves better performance than fixed embeddings in scenarios where there are changes in background noise or speakers' voices. Overall, this work presents an innovative solution for improving PSE performance in real world applications by dynamically adapting target speaker representations using cross attention in a streaming Transformer based model. This approach has significant potential for enhancing users' experiences with audio AI modules deployed in various settings, including ASR systems and teleconferencing platforms.