# Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model

**Authors:** Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, Soujanya Poria  
**arXiv:** [2304.13731](https://arxiv.org/abs/2304.13731)  
**Date:** 2023-04-24  
**Summary page:** https://www.summarizepaper.com/en/paper/2304.13731v1/

## Key points

- The paper proposes a novel approach called TANGO for text-to-audio (TTA) generation
- It adopts an instruction-tuned LLM Flan-T5 as the text encoder for TTA generation
- The authors use an LDM-based approach with an LLM text encoder to generate audio from textual descriptions
- To augment training samples, pressure level-based mixing method is employed instead of randomly generated combinations of audio pairs
- The proposed model outperforms state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen
- The improvement is attributed to the adoption of audio pressure level based sound mixing for training set augmentation
- The paper's contribution is threefold: proposing TANGO, showing that using an instruction tuned LLM improves performance, and demonstrating that pressure level based sound mixing can be more effective than random mixtures for training set augmentation.
- Overall, this paper presents promising results in generating audio from textual descriptions using an LDM based approach with an instruction tuned LLM as the text encoder.

## Summary

This preprint paper explores the task of text-to-audio (TTA) generation and proposes a novel approach called TANGO. The authors draw inspiration from the success of large language models (LLMs) in natural language processing (NLP) tasks and adopt an instruction-tuned LLM Flan-T5 as the text encoder for TTA generation. Unlike prior works that either pre-trained a joint text-audio encoder or used a non-instruction-tuned model such as T5, the authors propose using an LDM-based approach with an LLM text encoder to generate audio from textual descriptions. To augment training samples, the authors employ a pressure level-based mixing method instead of randomly generated combinations of audio pairs. They show that their proposed model outperforms state-of-the-art AudioLDM on most metrics and stays comparable on the rest on AudioCaps test set despite training the LDM on a 63 times smaller dataset and keeping the text encoder frozen. The authors attribute this improvement to the adoption of audio pressure level based sound mixing for training set augmentation. The paper's contribution is threefold: firstly, it proposes a novel approach called TANGO for TTA generation; secondly, it shows that using an instruction tuned LLM greatly improves text to audio generation performance; thirdly, it demonstrates that pressure level based sound mixing can be more effective than random mixtures for training set augmentation. Overall, this paper presents promising results in generating audio from textual descriptions using an LDM based approach with an instruction tuned LLM as the text encoder. If trained on larger datasets such as AudioSet, TANGO could potentially improve its ability to recognize a wider range of sounds.