# What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models

**Authors:** Enis Berk Çoban, Michael I. Mandel, Johanna Devaney  
**arXiv:** [2406.04615](https://arxiv.org/abs/2406.04615)  
**Date:** 2024-06-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2406.04615v1/

## Key points

- Authors explore capabilities of Multimodal Large Language Models (MLLMs) for classification tasks
- Large Language Models (LLMs) demonstrate impressive reasoning abilities in connecting ideas and adhering to logical rules
- MLLMs can handle various data modalities, including sound and images, to describe audio recordings or visual content
- Study investigates limitations in effectively utilizing LLM's reasoning capabilities in generating audio captions with an audio MLLM
- Experiment reveals challenges in integrating auditory and textual information, hindering the model's full reasoning potential
- Findings highlight complexities in harnessing multimodal language models for classification tasks and suggest pathways for future research to enhance MLLM performance

## Summary

, , , , 

In their paper titled "What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models," authors Enis Berk Çoban, Michael I. Mandel, and Johanna Devaney explore the capabilities of Multimodal Large Language Models (MLLMs) in leveraging reasoning abilities for classification tasks. They highlight the impressive reasoning capabilities of Large Language Models (LLMs), particularly in connecting ideas and adhering to logical rules. MLLMs have evolved to handle various data modalities, including sound and images, enabling them to describe audio recordings or visual content. The authors build on previous research that demonstrated how freezing the LLM component in MLLMs allows the audio or visual encoder to caption sound or image inputs, facilitating text-based reasoning with the LLM component. However, their study focuses on investigating whether an audio MLLM can effectively utilize its LLM's text-based reasoning when generating audio captions. They find that there are limitations in fully leveraging the LLM's reasoning capabilities in this context. One key insight from their experiment is that MLLMs may struggle to integrate auditory and textual information effectively, potentially disrupting the pathway for reasoning from the LLM to the audio encoder. This separation of representation between auditory and textual data could hinder the model's ability to capitalize on its full reasoning potential for generating accurate audio captions. Overall, this study sheds light on the complexities involved in harnessing multimodal language models for classification tasks, highlighting challenges related to integrating different data modalities and optimizing reasoning pathways within these models. The findings contribute valuable insights for future research aiming to enhance the performance of MLLMs in handling diverse types of information for improved classification accuracy.