# AudioLM: a Language Modeling Approach to Audio Generation

**Authors:** Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, Neil Zeghidour  
**arXiv:** [2209.03143](https://arxiv.org/abs/2209.03143)  
**Date:** 2022-09-07  
**Summary page:** https://www.summarizepaper.com/en/paper/2209.03143v2/

## Key points

- Authors introduce AudioLM as a framework for high-quality audio generation with long-term consistency
- Framework maps input audio to discrete tokens and frames audio generation within this tokenized representation space
- Authors propose a hybrid approach combining masked language model activations and neural audio codec codes for effective synthesis
- AudioLM trained on raw audio waveforms learns to generate natural and coherent continuations based on short prompts
- Demonstrates ability to generate syntactically and semantically plausible speech continuations while preserving speaker identity and prosody even for unseen speakers
- Versatile application beyond speech generation to producing coherent piano music continuations
- Successfully generates musical sequences exhibiting coherence and musicality without symbolic representation of music data

## Summary

In their paper titled "AudioLM: a Language Modeling Approach to Audio Generation," authors Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour introduce AudioLM as a framework for high-quality audio generation with long-term consistency. The  framework operates by mapping input audio to a sequence of discrete tokens and framing audio generation as a  task within this tokenized representation space. The authors highlight the varying trade-offs between reconstruction quality and long-term structure offered by different existing . To address this challenge, they propose a hybrid  that aims to achieve both objectives effectively. This approach combines the discretized activations of a masked language model pre-trained on audio to capture long-term structure with the discrete codes generated by a neural audio codec for high-quality synthesis. Through training on extensive corpora of raw audio waveforms, AudioLM is able to learn how to generate natural and coherent continuations based on short prompts. Remarkably, when trained on speech data without any accompanying transcript or annotation information,<br>
AudioLM demonstrates the ability to generate syntactically and semantically plausible speech continuations while preserving speaker identity and prosody even for unseen speakers. Moreover,<br>
the authors showcase the versatility of their approach by extending its application beyond speech generation to producing coherent piano music continuations.<br>
Despite being trained without any symbolic representation of music data,<br>
AudioLM successfully generates musical sequences that exhibit coherence and musicality.<br>
Overall,<br>
the proposed AudioLM framework represents an innovative and effective method for generating high-quality audio content with long-term consistency across various domains such as speech and music.