# LLark: A Multimodal Foundation Model for Music

**Authors:** Josh Gardner, Simon Durand, Daniel Stoller, Rachel M. Bittner  
**arXiv:** [2310.07160](https://arxiv.org/abs/2310.07160)  
**Date:** 2023-10-11  
**Summary page:** https://www.summarizepaper.com/en/paper/2310.07160v1/

## Key points

- Introduction of LLark, an instruction-tuned multimodal model for music understanding
- Challenges posed by the unique and complex structure of music for both humans and AI systems
- Creation of a dataset through augmentation of annotations from open-source music datasets into a unified instruction-tuning format
- Utilization of a multimodal architecture in the LLark model, integrating pretrained generative and language models
- Performance evaluations on tasks such as music understanding, captioning, and reasoning showing LLark matching or outperforming existing baselines in zero-shot generalization
- High agreement between human evaluations and LLark's responses in captioning and reasoning tasks
- Results demonstrating LLark's superiority over existing multimodal models in reasoning tasks related to audio and queries
- Impact analysis of language model and audio encoder, highlighting significant contributions to performance gains on benchmark tasks
- Training entirely from open-source music data with code availability upon paper release
- Additional metrics and evaluation results showcasing LLark's capabilities in music understanding for advancing AI-driven music analysis.

## Summary

In this paper, the authors introduce LLark, an instruction-tuned multimodal model designed for music understanding. The unique and complex structure of music poses challenges for both expert humans and existing AI systems. To address this, the authors detail their process for creating a dataset by augmenting annotations from various open-source music datasets and converting them into a unified instruction-tuning format. The proposed LLark model utilizes a multimodal architecture that integrates a pretrained generative model for music with a pretrained language model. Through evaluations on three types of tasks - music understanding, captioning, and reasoning - the authors demonstrate that LLark matches or outperforms existing baselines in zero-shot generalization for music understanding. Additionally, human evaluations show a high degree of agreement with the model's responses in captioning and reasoning tasks. The paper also includes results from reasoning tasks where LLark's outputs surpass existing multimodal models in terms of correspondence to audio and queries. An ablation study is conducted to investigate the impact of the language model and audio encoder as well as scaling behavior with respect to training dataset size. The findings suggest that both the Jukebox audio encoder and Llama 2 language model contribute significantly to performance gains on benchmark tasks. Overall, LLark is trained entirely from open-source music data and models, with the training code made available alongside the release of the paper. The authors provide additional metrics and evaluation results in supplementary sections showcasing LLark's capabilities in music understanding and its potential for advancing research in AI-driven music analysis.