Dyck and Parity were proved theoretically that a transformer architecture could not model them. These limitations can be a drawback when applying transformers-based models to specific natural language tasks involving reasoning capabilities. We designed two different natural datasets to simulate the Dyck-2 and the Parity tasks.
Clone this repository
git clone https://github.com/ChadiHelwe/FailBERT.git
pip install -r requirements.txtpython run_natural_dyck_2_experiment.pypython run_natural_parity_experiment.pypython run_create_natural_dyck_2_dataset.py create-natural-dyck-2-dataset --path_dyck_2_dataset data/dyck_2/dyck_2_test_1.txt --path_natural_dyck_2_dataset data/natural_dyck_2/natural_dyck_2_test_1.csvpython run_natural_dyck_2.py train-modelpython run_natural_dyck_2.py download-pretrained-modelpython run_natural_dyck_2.py test-modelpython run_create_natural_parity_dataset.py create-natural-parity-dataset --path_natural_parity_dataset data/natural_parity/test1.csv --min_range_length 21 --max_range_length 40 --min_nbr_switch_operation 16 --max_nbr_switch_operation 20python run_utils.py create-equally-distributed-natural-dataset --path_dataset data/natural_parity/test1.csv --path_equally_distributed_dataset data/natural_parity/test1.csv --limit True --nbr_instances 2500python run_natural_parity.py train-modelpython run_natural_parity.py download-pretrained-modelpython run_natural_parity.py test-model