# Versatile Audio-Visual Learning for Handling Single and Multi Modalities in Emotion Regression and Classification Tasks

**Authors:** Lucas Goncalves, Seong-Gyun Leem, Wei-Cheng Lin, Berrak Sisman, Carlos Busso  
**arXiv:** [2305.07216](https://arxiv.org/abs/2305.07216)  
**Date:** 2023-05-12  
**Summary page:** https://www.summarizepaper.com/en/paper/2305.07216v1/

## Key points

- Existing audio-visual emotion recognition models lack flexibility for practical applications
- Authors propose a versatile audio-visual learning (VAVL) framework
- VAVL can handle both unimodal and multimodal systems for emotion regression and classification tasks
- Goal is to develop a system that works with one modality and predicts emotional attributes or recognizes categorical emotions interchangeably
- Challenges include accurately interpreting and integrating diverse data sources, handling missing or partial information, and allowing direct switch between regression and classification tasks
- Authors implement an audio-visual framework that can be trained even without paired data for part of the training set
- Effective representation learning achieved through audio-visual shared layers, residual connections, and unimodal reconstruction task
- Experimental results show VAVL outperforms strong baselines on CREMA-D and MSP-IMPROV corpora
- VAVL achieves state-of-the-art performance in emotional attribute prediction on MSP-IMPROV corpus
- Study enhances flexibility of audio-visual emotion recognition models by incorporating versatile learning techniques

## Summary

The existing audio-visual emotion recognition models often lack the necessary flexibility for practical applications. To address this limitation, the authors propose a versatile audio-visual learning (VAVL) framework that can handle both unimodal and multimodal systems for emotion regression and classification tasks. The goal is to develop a system that can work even when only one modality is available and can be used interchangeably for predicting emotional attributes or recognizing categorical emotions. Achieving such flexibility in a multimodal emotion recognition system is challenging due to the difficulties in accurately interpreting and integrating diverse data sources. Additionally, it is also challenging to robustly handle missing or partial information while allowing a direct switch between regression and classification tasks. To overcome these challenges, the authors implement an audio-visual framework that can be trained even when audio and visual paired data is not available for part of the training set. They achieve effective representation learning through audio-visual shared layers, residual connections over shared layers, and a unimodal reconstruction task. Experimental results demonstrate that the proposed VAVL architecture outperforms strong baselines on both the CREMA-D and MSP-IMPROV corpora. Notably, VAVL achieves state-of-the-art performance in the emotional attribute prediction task on the MSP-IMPROV corpus. Overall, this study presents a novel approach to enhance the flexibility of audio-visual emotion recognition models by incorporating versatile learning techniques. The proposed VAVL framework demonstrates improved performance compared to existing methods on various datasets while providing greater flexibility with respect to missing or partial information as well as switching between regression and classification tasks.