The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation

Ilaria Manco*^1,2, Benno Weck*³, Seungheon Doh⁴, Minz Won⁵, Yixiao Zhang¹, Dmitry Bogdanov³, Yusong Wu⁶, Ke Chen⁷, Philip Tovstogan³, Emmanouil Benetos¹, Elio Quinton², George Fazekas¹, Juhan Nam⁴

¹ QMUL, ² UMG, ³ UPF, ⁴ KAIST, ⁵ ByteDance, ⁶ MILA, ⁷ UCSD

* equal contribution

This repository contains starter code for the Song Describer Dataset (SDD).

• Paper (accepted to the ML for Audio workshop @ NeurIPS 2023)
• Zenodo
• Datasheet
• Hugging Face (HF demo built by @Renumics)

Dataset overview

sdd.mp4

"A retro-futurist drum machine groove drenched in bubbly synthetic sound effects and a hint of an acid bassline."

"Elegant and sophisticated Latin jazz piece with a Cuban base and a whispered melodic female voice."

"Calm sitar and Indian tabla with dramatic synthetic strings background."

SDD contains ~1.1k captions for 706 permissively licensed music recordings. It is designed for use in evaluation of models that address music-and-language (M&L) tasks such as music captioning, text-to-music generation and music-language retrieval. More information about the data, collection method and validation is provided in the data card, together with more in-depth documentation in the datasheet.

Subset	Tracks	Captions	Annotators	Cap len (avg)	Vocab size	Audio len
full	706	1106	142	21.7 ± 12.4	2859	~ 2 min
valid	546	746	114	18.2 ± 7.6	1942	~ 2 min

Downloading the dataset

The dataset is available to download from Zenodo:

wget -P data https://zenodo.org/record/10072001/files/song_describer.csv https://zenodo.org/record/10072001/files/audio.zip
unzip data/audio.zip -d data/audio

A download script is also available here.

Code setup

To use this code, we recommend creating a new python3 virtual environment:

python -m venv venv 
source venv/bin/activate

Then, clone the repository and install the dependencies:

git clone https://github.com/mulab-mir/song-describer-dataset.git
cd song-describer-dataset
pip install -r requirements.txt

Reproducing the analysis in the paper

The overview statistics presented in the paper can be reproduced via the code in the dataset_stats.ipynb notebook. Further exploratory analysis of the data can be found in the data_exploration.ipynb notebook

Using the dataset

PyTorch

[Coming soon]

Hugging Face

[coming soon]

Benchmarking M&L models with SDD

[coming soon]

Cite

If you use the dataset or the code in this repo, please consider citing our work:

@inproceedings{manco2023thesong,
  title={The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation}, 
  author={Manco, Ilaria and Weck, Benno and Doh, Seungheon and Won, Minz and Zhang, Yixiao and Bogdanov, Dmitry and Wu, Yusong and Chen, Ke and Tovstogan, Philip and Benetos, Emmanouil and Quinton, Elio and Fazekas, György and Nam, Juhan},
  booktitle={Machine Learning for Audio Workshop at NeurIPS 2023}, 
  year={2023},
}

License

This repository is released under the MIT License. Please see the LICENSE file for more details. The dataset is released under the CC BY-SA 4.0 license.

Contact

If you have any questions, please get in touch: [email protected].