# On Metric Learning for Audio-Text Cross-Modal Retrieval

**Authors:** Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D. Plumbley, Wenwu Wang  
**arXiv:** [2203.15537](https://arxiv.org/abs/2203.15537)  
**Date:** 2022-03-29  
**Summary page:** https://www.summarizepaper.com/en/paper/2203.15537v1/

## Key points

- Audio-text cross-modal retrieval involves retrieving specific audio clips or captions based on queries in different modalities
- Development of robust feature representations for audio and text modalities is crucial, along with precise alignment between them
- NT-Xent loss from self-supervised learning shows consistent performance in audio-text retrieval tasks compared to triplet-based losses
- Free-form language-based audio-text retrieval is more complex than tag-based approaches due to sequence data nature of both audio and text
- Researchers aim to enhance performance by adopting strategies from video retrieval models and leveraging pre-trained models to address data scarcity issues

## Summary

, , , , 


In the realm of audio-text cross-modal retrieval, the task involves retrieving a specific audio clip or caption from a pool of candidates based on a query in another modality. This challenge necessitates the development of robust feature representations for both audio and text modalities, as well as the precise alignment between them. While existing cross-modal retrieval models primarily utilize metric learning objectives to map data into an embedding space where similar data points are clustered together and dissimilar ones are separated, audio-text retrieval remains relatively unexplored compared to image-text and video-text retrievals. In their study titled "On Metric Learning for Audio-Text Cross-Modal Retrieval," Xinhao Mei, Xubo Liu, Jianyuan Sun, Mark D. Plumbley, and Wenwu Wang delve into exploring the impact of various metric learning objectives on audio-text retrieval tasks. Through an extensive evaluation conducted on the AudioCaps and Clotho datasets, they highlight that the NT-Xent loss derived from self-supervised learning exhibits consistent performance across diverse datasets and training configurations, surpassing popular triplet-based losses. The authors note the recent surge in audio captioning advancements leading to the release of publicly available datasets conducive to free-form language-based audio-text retrieval tasks. They emphasize the heightened complexity of free-form language-based audio-text retrieval compared to tag-based approaches due to both audio and text (captions) being sequence data. The paper underscores the interchangeability of terms such as "audio-text" and "audio-caption" within its context. Building upon established cross-modal retrieval models, including separate sub-networks for audio encoding and text encoding, this research focuses on addressing challenges specific to free-form language-based audio-text retrieval. By adopting strategies from video retrieval models and leveraging pre-trained models to mitigate data scarcity issues, researchers aim to enhance performance in this intricate domain. Furthermore, the study delves into detailed discussions on triplet loss optimization techniques involving positive pairs (anchor-positive) apij and negative pairs sij within a constant training setting. By scrutinizing different metric learning objectives' efficacy in enhancing cross-modal retrieval accuracy, this work contributes valuable insights towards advancing state-of-the-art techniques in audio-text cross-modal information retrieval systems.