# Teaching a Multilingual Large Language Model to Understand Multilingual Speech via Multi-Instructional Training

**Authors:** Pavel Denisov, Ngoc Thang Vu  
**arXiv:** [2404.10922](https://arxiv.org/abs/2404.10922)  
**Date:** 2024-04-16  
**Summary page:** https://www.summarizepaper.com/en/paper/2404.10922v1/

## Key points

- Recent advancements in language modeling have led to the development of Large Language Models (LLMs) that excel in natural language processing tasks.
- BLOOMZMMS is a novel model that integrates a multilingual LLM with a multilingual speech encoder to leverage LLM capabilities for speech-related tasks.
- Researchers demonstrate the transferability of linguistic knowledge from text to speech modalities through multi-instructional training on 1900 hours of transcribed data spanning 139 languages.
- Synthetic targets were generated to address limitations in task generalization, leading to robust zero-shot evaluation results across tasks like speech translation and multilingual spoken language understanding.
- Insights from related studies highlight the utilization of models like LLaMA-7B for English speech recognition and multiple languages by training custom speech encoders or integrating pretrained ones.

## Summary

, , , , 

Recent advancements in language modeling have paved the way for Large Language Models (LLMs) that excel at various natural language processing tasks. These models have shown great success in text-based applications, but their adaptation to the speech domain has been limited and challenging. In response to this gap, a novel model called BLOOMZMMS has been introduced, which integrates a multilingual LLM with a multilingual speech encoder. The goal of this integration is to leverage the capabilities of LLMs for tasks such as speech recognition and beyond. Through a multi-instructional training approach, the researchers behind BLOOMZMMS demonstrate the transferability of linguistic knowledge from text to speech modalities. By conducting experiments on 1900 hours of transcribed data spanning 139 languages, they establish that it is possible to effectively learn and align a multilingual speech representation with a multilingual LLM. Although initial results showed limitations in task generalization, these were addressed by generating synthetic targets using a multi-instructional style. The zero-shot evaluation results confirm the robustness of this approach across multiple tasks, including speech translation and multilingual spoken language understanding. This breakthrough opens up new possibilities for applying LLMs in the realm of speech processing. Furthermore, insights from related studies show how models like LLaMA-7B have been utilized for English speech recognition and across multiple languages through training custom speech encoders from scratch or integrating pretrained ones into the system. Overall, this research represents an important step towards bridging the gap between language models and speech processing technologies, offering promising avenues for further exploration and development in this field.