# Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation

**Authors:** Yusong Wu, Ke Chen, Tianyu Zhang, Yuchen Hui, Taylor Berg-Kirkpatrick, Shlomo Dubnov  
**arXiv:** [2211.06687](https://arxiv.org/abs/2211.06687)  
**Date:** 2022-11-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2211.06687v1/

## Key points

- Authors propose a pipeline of contrastive language-audio pretraining to develop an audio representation
- LAION-Audio-630K is a large collection of 633,526 audio-text pairs from different data sources released by the authors
- A contrastive language-audio pretraining model is constructed that considers different audio encoders and text encoders
- Feature fusion mechanism and keyword-to-caption augmentation are incorporated into the model design to process audio inputs of variable lengths and enhance performance
- Comprehensive experiments are performed to evaluate the model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification
- The proposed model achieves superior performance in the text-to-audio retrieval task and state-of-the-art performance in zero-shot setting for audio classification tasks
- LAION Audio 630K dataset and contrastive language-audio pretraining model are both publicly available
- Future work includes collecting even larger datasets for training and applying representations into more downstream tasks such as audio synthesis and separation.

## Summary

In this paper, the authors propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To achieve this goal, they first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, they construct a contrastive language-audio pretraining model that considers different audio encoders and text encoders. The authors incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to enable it to process audio inputs of variable lengths and enhance performance. The authors perform comprehensive experiments to evaluate their model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that their model achieves superior performance in the text-to-audio retrieval task. In the zero-shot setting for audio classification tasks, the model achieves state-of-the art performance and is able to obtain comparable results in nonzero shot settings. The proposed LAION Audio 630K dataset and contrastive languageaudio pretraining model are both publicly available. Future work includes collecting even larger datasets for training and applying representations into more downstream tasks such as audio synthesis and separation. Overall, this paper contributes to the field of multimodal representation learning by proposing a novel pipeline for developing an effective audio representation through contrastive languageaudio pretraining using a large scale dataset with feature fusion mechanisms and keywordtocaption augmentation.