# Melody transcription via generative pre-training

**Authors:** Chris Donahue, John Thickstun, Percy Liang  
**arXiv:** [2212.01884](https://arxiv.org/abs/2212.01884)  
**Date:** 2022-12-04  
**Summary page:** https://www.summarizepaper.com/en/paper/2212.01884v1/

## Key points

- Melody transcription in music information retrieval is challenging due to diverse audio, instrument ensembles, and musical styles.
- Leveraging representations from Jukebox, a generative model of broad music audio, leads to a 20% improvement in melody transcription performance.
- A new dataset with 50 hours of melody transcriptions obtained through crowdsourced annotations addresses the lack of sufficient training data.
- Generative pre-training combined with the new dataset results in a significant 77% enhancement in melody transcription accuracy.
- The Sheet Sage system integrates beat detection, key estimation, and chord recognition to transcribe human-readable lead sheets directly from music audio.
- The research by Chris Donahue, John Thickstun, and Percy Liang on melody transcription via generative pre-training was published at ISMIR 2022.

## Summary

In the field of music information retrieval, detecting the notes of a melody in a music recording remains a challenging task. Existing strategies for melody transcription struggle to handle diverse audio containing various instrument ensembles and musical styles. To address this challenge, a new approach leverages representations from Jukebox, a generative model of broad music audio. This results in a 20% improvement in melody transcription performance compared to traditional spectrogram features. One major obstacle in melody transcription is the lack of sufficient training data. To overcome this limitation, a new dataset comprising 50 hours of melody transcriptions obtained through crowdsourced annotations of diverse music was created. The combination of generative pre-training and the new dataset led to a significant 77% enhancement in melody transcription accuracy compared to existing methods. Furthermore, by integrating solutions for beat detection, key estimation, and chord recognition, the Sheet Sage system was developed. This innovative system is capable of transcribing human-readable lead sheets directly from music audio. By pairing advanced melody transcription techniques with other music analysis tools, Sheet Sage offers a comprehensive solution for converting complex music recordings into easily interpretable musical notations. The research conducted by Chris Donahue, John Thickstun, and Percy Liang on melody transcription via generative pre-training has been published as a conference paper at ISMIR 2022. The study compares the proposed approach with baseline methods incorporating note segmentation heuristics and zero-shot models trained on related tasks like vocal transcription. Additionally, efforts are made to address the data deficit in melody transcription tasks through innovative methodologies and datasets creation.