# Decoupled Multimodal Distilling for Emotion Recognition

**Authors:** Yong Li, Yuanzhi Wang, Zhen Cui  
**arXiv:** [2303.13802](https://arxiv.org/abs/2303.13802)  
**Date:** 2023-03-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2303.13802v1/

## Key points

- The challenge in human multimodal emotion recognition (MER) is perceiving emotions through language, visual, and acoustic modalities due to heterogeneities and varying contributions.
- Decoupled Multimodal Distillation (DMD) is a novel approach proposed to enhance the discriminative features of each modality by decoupling their representations into modality-irrelevant and modality-exclusive spaces through a self-regression process.
- DMD utilizes Graph Distillation Units (GD-Units) for each decoupled part, allowing specialized knowledge distillation with dynamic graph structures for flexible knowledge transfer.
- Experimental results show that DMD consistently outperforms state-of-the-art MER methods, demonstrating its effectiveness in enhancing emotion recognition accuracy.
- Implementation details involve extracting unimodal language features using GloVe and BERT-base-uncased pre-trained models, encoding video frames via Facet for facial action unit representation, and processing acoustic modality data.
- On the CMU-MOSI dataset, DMD achieves superior performance compared to existing methods such as EF-LSTM, LF-LSTM, TFN, LMF, MFM, RAVEN, MCTN, MulT, PMR, MISA*, FDMER*, and MICA*.
- DMD facilitates adaptive crossmodal knowledge distillation for improved emotion recognition across diverse modalities.

## Summary

, , , , 


In the field of human multimodal emotion recognition (MER), the challenge lies in effectively perceiving emotions through language, visual, and acoustic modalities due to inherent heterogeneities and varying contributions of different modalities. To address this issue, a novel approach called decoupled multimodal distillation (DMD) is proposed in this work. DMD aims to enhance the discriminative features of each modality by decoupling their representations into modality-irrelevant and modality-exclusive spaces through a self-regression process. This approach utilizes graph distillation units (GD-Units) for each decoupled part, allowing for specialized and effective knowledge distillation. Each GD-Unit consists of a dynamic graph where vertices represent modalities and edges indicate dynamic knowledge distillation, enabling flexible knowledge transfer with automatically learned distillation weights. Experimental results demonstrate that DMD consistently outperforms state-of-the-art MER methods, showcasing its effectiveness in enhancing emotion recognition accuracy. Visualization results reveal meaningful distributional patterns in the graph edges of DMD with respect to the modality-irrelevant and modality-exclusive feature spaces. The implementation details involve extracting unimodal language features using GloVe and BERT-base-uncased pre-trained models, encoding video frames via Facet for facial action unit representation, and processing acoustic modality data. Furthermore, on the CMU-MOSI dataset, DMD achieves superior performance compared to existing methods such as EF-LSTM, LF-LSTM, TFN, LMF, MFM, RAVEN, MCTN, MulT, PMR, MISA*, FDMER*, and MICA*. The refined detailed summary highlights the innovative nature of DMD in facilitating adaptive crossmodal knowledge distillation for improved emotion recognition across diverse modalities.