Pre-training Text-to-Text Transformers for Concept-centric Common Sense

This code is for ICLR2021 paper: Pre-training Text-to-Text Transformers for Concept-centric Common Sense. Checkout our Project website for details!

Installation

conda create -n calm python==3.7
conda activate calm
python setup.py install
cd CALM

Preprocess for CALM

wiki pre-process

cat wiki.doc | tail -n +500000 | head -n 500000 > wiki/wiki.train.raw
cat wiki.doc | tail -n +1000000 | head -n 100000 > wiki/wiki.valid.raw

Generative Objective

python dataset_utils/concept_deshuffling_data_generation.py
python dataset_utils/keyword_lm_data_generation.py

Dataset creation for concept-order-recovering (COR) and concept-to-sentence (C2S).

Contrastive Objective

python dataset_utils/generate_discriminative_dataset.py

Dataset creation for generative question answering (QA) dataset.
There are three types of contrastive objectives (See Table 4 (b) in the paper).

Option 1: Multi-choice QA
Option 2: Generative QA
Option 3: True/False

For CALM, we use option 2, which is Generative QA.

Mix three dataset

python dataset_utils/mix_dataset.py

Pre-training

Pre-train CALM_mix

First, train the mix dataset.

python finetune.py \
    --data_dir datasets/mix \
    --output_dir outputs/calm_mix_base \
    --model_name_or_path t5-base \
    --tokenizer_name_or_path t5-base \
    --max_seq_length 256 \
    --learning_rate 5e-4 \
    --num_train_epochs 2 \
    --train_batch_size 8 \
    --graident_accumulation_steps 4 \
    --weight_decay 0.01 \
    --warmup_steps 10000 \
    --adam_epsilon 1e-6 \
    --n_gpu 4 \
    --gpu_nums 4,5,6,7 \
    --model_parallel

python finetune.py \
    --data_dir datasets/mix \
    --output_dir outputs/calm_mix_large_dp \
    --model_name_or_path t5-large \
    --tokenizer_name_or_path t5-large \
    --max_seq_length 256 \
    --learning_rate 5e-4 \
    --num_train_epochs 2 \
    --train_batch_size 8 \
    --graident_accumulation_steps 4 \
    --weight_decay 0.01 \
    --warmup_steps 10000 \
    --adam_epsilon 1e-6

Pre-train CALM

Then, train CALM using the checkpoint of mix dataset.

python finetune_generator_discriminator.py \
    --data_dir datasets/option2 \
    --checkpoint_dir outputs/calm_mix \
    --output_dir outputs/calm \
    --max_seq_length 256 \
    --learning_rate 5e-7 \
    --num_train_epochs 3 \
    --train_batch_size 8 \
    --graident_accumulation_steps 32 \
    --fp_16 False \
    --weight_decay 0.01 \
    --warmup_steps 10000 \
    --adam_epsilon 1e-6 \
    --n_gpu 8 \
    --gpu_nums 0,1,2,3,4,5,6,7

python finetune_generator_discriminator.py \
    --data_dir datasets/option2 \
    --checkpoint_dir outputs/calm_mix_base_dp \
    --output_dir outputs/calm_base_dp \
    --max_seq_length 256 \
    --learning_rate 5e-7 \
    --num_train_epochs 3 \
    --train_batch_size 8 \
    --graident_accumulation_steps 32 \
    --fp_16 False \
    --weight_decay 0.01 \
    --warmup_steps 10000 \
    --adam_epsilon 1e-6

Fine-tuning

Use checkpoint to fine-tune on the downstream tasks.

Model List

Our released models are listed as following. You can import these models by using HuggingFace's Transformers.

Model	CSQA	OBQA	PIQA	aNLI	Description
danny911kr/calm-mix-base	63.02	60.40	70.07	62.79	Mix-Only
danny911kr/calm-base	63.32	60.90	71.01	63.20
danny911kr/calm-mix-large	70.26	62.50	73.70	75.99	Mix-Only
danny911kr/calm-large	71.31	66.00	75.11	77.12