Downstream Task Instructions

Task 1 - Training and Testing of the Phone Classification Task

Run the following command to train a phone classifier:

# using spectrogram as baseline
python3 runner_mockingjay.py --train_phone
# using mockingjay representations
python3 runner_mockingjay.py --train_phone --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_phone --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_phone --run_apc

Run the following command to test representations using the phone classifier:

# testing spectrogram as baseline
python3 runner_mockingjay.py --test_phone
# testing mockingjay representations
python3 runner_mockingjay.py --test_phone --run_mockingjay
# testing the fine-tuned mockingjay model on downstream tasks
python3 runner_mockingjay.py --test_phone --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --dckpt=mockingjay_phone_libri_sd1337_MelBaseFT_topline/mockingjay-500000.ckpt
# testing apc representations
python3 runner_mockingjay.py --test_phone --run_apc

To link and compare with the CPC work, we can train with the identical aligned phone labels and train/test split as in the CPC paper. Make sure that you have followed the instructions in the downstream task preprocessing wiki page, and the Kaldi feature extraction wiki page. Then edit the following attributes in your .yaml config files, and use the following commands:

# mockingjay_libri_fmllrBase.yaml
data_path: 'data/libri_fmllr_cmvn' # Features extracted by Kaldi (http://www.kaldi-asr.org/downloads/build/6/trunk/egs/librispeech/)
phone_path: 'data/cpc_phone'
train_set: ['train-clean-100']  

# using spectrogram as baseline
python3 runner_mockingjay.py --train_cpc_phone
# using mockingjay representations
python3 runner_mockingjay.py --train_cpc_phone --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_cpc_phone --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_cpc_phone --run_apc

Task 2 - Training and Testing of the Sentimental Classification Task

Run the following command to train a sentiment classifier:

# using spectrogram as baseline
python3 runner_mockingjay.py --train_sentiment
# using mockingjay representations
python3 runner_mockingjay.py --train_sentiment --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_sentiment --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_sentiment --run_apc

Run the following command to test representations using the phone classifier:

# testing spectrogram as baseline
python3 runner_mockingjay.py --test_sentiment
# testing mockingjay representations
python3 runner_mockingjay.py --test_sentiment --run_mockingjay
# testing the fine-tuned mockingjay model on downstream tasks
python3 runner_mockingjay.py --test_sentiment --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --dckpt=mockingjay_sentiment_libri_sd1337_MelBaseFT_topline/mockingjay-500000.ckpt
# testing apc representations
python3 runner_mockingjay.py --test_sentiment --run_apc

Task 3 - Training and Testing of the Speaker Verification Task

Run the following command to train a sentiment classifier:

# using spectrogram as baseline
python3 runner_mockingjay.py --train_speaker
# using mockingjay representations
python3 runner_mockingjay.py --train_speaker --run_mockingjay
# fine-tune the mockingjay model on downstream tasks
python3 runner_mockingjay.py --train_speaker --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --ckpt=mockingjay_libri_sd1337_MelBase/mockingjay-500000.ckpt
# using apc representations
python3 runner_mockingjay.py --train_speaker --run_apc

Run the following command to test representations using the phone classifier:

# testing spectrogram as baseline
python3 runner_mockingjay.py --test_speaker
# testing mockingjay representations
python3 runner_mockingjay.py --test_speaker --run_mockingjay
# testing the fine-tuned mockingjay model on downstream tasks
python3 runner_mockingjay.py --test_speaker --run_mockingjay --fine_tune --config=config/mockingjay_libri_MelBase.yaml --dckpt=mockingjay_speaker_libri_sd1337_MelBaseFT_topline/mockingjay-500000.ckpt
# testing apc representations
python3 runner_mockingjay.py --test_speaker --run_apc

Note that the arguments --ckpdir --ckptY --dckpt --apc_path needs to be set correctly for the above command to run properly.