Modulation Extraction for LFO-driven Audio Effects

Christopher Mitcheltree, Christian J. Steinmetz, Marco Comunità, and Joshua D. Reiss

---

Instructions for Reproducibility

1. Clone this repository and open its directory.
2. Install the requirements using
   conda env create --file=conda_env_cpu.yml or
   conda env create --file=conda_env.yml
   for GPU acceleration.
   requirements_pipchill.txt and requirements_all.txt are also provided as references, but are not needed when using the conda_env.yml files.
3. The source code can be explored in the mod_extraction/ directory.
4. All models from the paper can be found in the models/ directory.
5. Create an out directory (mkdir out).
6. Create a data directory (mkdir data).
7. Download the IDMT-SMT-Guitar and EGFx datasets and place their root directories in the data directory.
8. Rename the root directories of the datasets to idmt_4, egfx_clean_44100, egfx_phaser_44100, egfx_flanger_44100, and egfx_chorus_44100.
9. Use the methods in scripts/split_datasets.py to split each dataset into train, validation, and test sets.
10. (Optional) Download and install the free Melda Phaser and Melda Flanger plugins.
11. (Optional) Open and modify the ableton/melda_data_generation.als project file in Ableton to export the train, val, and test datasets for the unseen digital effects evaluation task. Input the data/idmt_4/ directory from step 8 when Ableton asks for the location of the missing audio files.
12. All models can be evaluated by modifying scripts/validate.py and the corresponding configs/eval_ ... .yml config file and then running python validate.py from the scripts/ directory.
    Make sure your PYTHONPATH has been set correctly by running a command like export PYTHONPATH=$PYTHONPATH:BASE_DIR/mod_extraction/.
13. (Optional) All models can be trained by modifying scripts/train.py and the corresponding configs/train_ ... .yml config file and then running python train.py from the scripts/ directory.
14. (Optional) Neutone files for running the effect models as a VST can be exported by modifying and running the scripts/export_neutone_models.py file.
15. The source code is currently not documented, but don't hesitate to open an issue if you have any questions or comments.

---

Citation

Accepted to the 26th International Conference on Digital Audio Effects (DAFx23), Copenhagen, Denmark, 4 - 7 September 2023.

@inproceedings{mitcheltree2023modulation,
    title={Modulation Extraction for LFO-driven Audio Effects},
    author={Christopher Mitcheltree and Christian J. Steinmetz and Marco Comunità and Joshua D. Reiss},
    booktitle={International Conference on Digital Audio Effects (DAFx)},
    year={2023}
}

---

Abstract

Low frequency oscillator (LFO) driven audio effects such as phaser, flanger, and chorus, modify an input signal using time-varying filters and delays, resulting in characteristic sweeping or widening effects. It has been shown that these effects can be modeled using neural networks when conditioned with the ground truth LFO signal. However, in most cases, the LFO signal is not accessible and measurement from the audio signal is nontrivial, hindering the modeling process. To address this, we propose a framework capable of extracting arbitrary LFO signals from processed audio across multiple digital audio effects, parameter settings, and instrument configurations. Since our system imposes no restrictions on the LFO signal shape, we demonstrate its ability to extract quasiperiodic, combined, and distorted modulation signals that are relevant to effect modeling. Furthermore, we show how coupling the extraction model with a simple processing network enables training of end-to-end black-box models of unseen analog or digital LFO-driven audio effects using only dry and wet audio pairs, overcoming the need to access the audio effect or internal LFO signal. We make our code available and provide the trained audio effect models in a real-time VST plugin.

---

Send feedback and questions to Christopher Mitcheltree.
Scroll to top.