Polish RoBERTa

This repository contains pre-trained RoBERTa models for Polish as well as evaluation code for several Polish linguistic tasks. The released models were trained using Fairseq toolkit in the National Information Processing Institute, Warsaw, Poland. We provide two models based on BERT base and BERT large architectures. Two versions of each model are available: one for Fairseq and one for Huggingface Transformers.

Updates

08.03.2022 - Base and Large Polish Longformer models have been added to the Huggingface Hub. The models were initialized with Polish RoBERTa (v2) weights and then fine-tuned on a corpus of long documents, ranging from 1024 to 4096 tokens.

19.02.2022 - The models are now available on the Huggingface Hub.

24.01.2022 - Polish DistilRoBERTa model added. The model was trained using knowledge distillation with RoBERTa-v2 base as a teacher model. Distilled version has only has only half the encoder blocks of the original model, so it is suitable for deployment on devices with limited resources such as smartphones.

18.01.2022 - We release the second version of the large model. This version has been trained using the same procedure as RoBERTa‑base-v2: unigram tokenizer, whole word masking, more update steps with lower batch size. We also utilised larger vocabulary of 128k entries.

21.03.2021 - We release a new version of the base model. The updated model has been trained on the same corpus as the original model but we used different hyperparameters. We made the following changes: 1) Sentencepiece Unigram model was used instead of BPE, 2) The model was trained with whole-word masking objective instead of classic token masking, 3) We utilized the full context of 512 tokens so training examples could include more than one sentence (the original model was trained on single sencentes only), 4) Longer pretraining (400k steps).

Models

Model	L / H / A*	Batch size	Update steps	Corpus size	KLEJ Score**	Fairseq	Transformers
RoBERTa (base)	12 / 768 / 12	8k	125k	~20GB	85.39	v0.9.0	v3.4
RoBERTa‑v2 (base)	12 / 768 / 12	8k	400k	~20GB	86.72	v0.10.1	v4.4
RoBERTa (large)	24 / 1024 / 16	30k	50k	~135GB	87.69	v0.9.0	v3.4
RoBERTa‑v2 (large)	24 / 1024 / 16	2k	400k	~200GB	88.87	v0.10.2	v4.14
DistilRoBERTa	6 / 768 / 12	1k	10ep.	~20GB	84.55	n/a	v4.13

* L - the number of encoder blocks, H - hidden size, A - the number of attention heads
** Average KLEJ score over 5 runs, see evaluation section for detailed results

More details are available in the paper Pre-training Polish Transformer-based Language Models at Scale.

@InProceedings{dadas2020pretraining,
  title="Pre-training Polish Transformer-Based Language Models at Scale",
  author="Dadas, S{\l}awomir and Pere{\l}kiewicz, Micha{\l} and Po{\'{s}}wiata, Rafa{\l}",
  booktitle="Artificial Intelligence and Soft Computing",
  year="2020",
  publisher="Springer International Publishing",
  pages="301--314",
  isbn="978-3-030-61534-5"
}

Getting started

How to use with Fairseq

import os
from fairseq.models.roberta import RobertaModel, RobertaHubInterface
from fairseq import hub_utils

model_path = "roberta_large_fairseq"
loaded = hub_utils.from_pretrained(
    model_name_or_path=model_path,
    data_name_or_path=model_path,
    bpe="sentencepiece",
    sentencepiece_vocab=os.path.join(model_path, "sentencepiece.bpe.model"),
    load_checkpoint_heads=True,
    archive_map=RobertaModel.hub_models(),
    cpu=True
)
roberta = RobertaHubInterface(loaded['args'], loaded['task'], loaded['models'][0])
roberta.eval()
input = roberta.encode("Zażółcić gęślą jaźń.")
output = roberta.extract_features(input)
print(output[0][1])

How to use with HuggingFace Transformers

import torch, os
from transformers import RobertaModel, AutoModel, PreTrainedTokenizerFast

model_dir = "roberta_base_transformers"
tokenizer = PreTrainedTokenizerFast(tokenizer_file=os.path.join(model_dir, "tokenizer.json"))
model: RobertaModel = AutoModel.from_pretrained(model_dir)
input = tokenizer.encode("Zażółcić gęślą jaźń.")
output = model(torch.tensor([input]))[0]
print(output[0][1])

Evaluation

To replicate our experiments, first download the required datasets using download_data.py script:

python download_data.py

Next, run run_tasks.py script to prepare the data, fine-tune and evaluate the model. We used the following parameters for each task:

python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-NKJP --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-CDS-E --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-CDS-R --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 1 --tasks KLEJ-CBD --fp16 True --max-sentences 8 --update-freq 4 --resample 0:0.75,1:3
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-POLEMO-IN --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-POLEMO-OUT --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-DYK --fp16 True --max-sentences 8 --update-freq 4 --resample 0:1,1:3
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-PSC --fp16 True --max-sentences 8 --update-freq 4 --resample 0:1,1:3
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks KLEJ-ECR --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks 8TAGS --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks SICK-E --fp16 True --max-sentences 8 --update-freq 2
python run_tasks.py --arch roberta_base --model_dir roberta_base_fairseq --train-epochs 10 --tasks SICK-R --fp16 True --max-sentences 8 --update-freq 2

Evaluation results on KLEJ Benchmark

Below we show the evaluation results of our models on the tasks included in KLEJ Benchmark. We fine-tuned both models 5 times for each task. Detailed scores for each run and averaged scores are presented in Table 1 and Table 2.

Table 1. KLEJ results for RoBERTa base model

Run	NKJP	CDSC‑E	CDSC‑R	CBD	PolEmo‑IN	PolEmo‑OUT	DYK	PSC	AR	Avg
1	93.15	93.30	94.26	66.67	91.97	78.74	66.86	98.63	87.75	85.70
2	93.93	94.20	93.94	68.16	91.83	75.91	65.93	98.77	87.93	85.62
3	94.22	94.20	94.04	69.23	90.17	76.92	65.69	99.24	87.76	85.72
4	93.97	94.70	93.98	63.81	90.44	76.32	65.18	99.39	87.58	85.04
5	93.63	94.00	93.96	65.95	90.58	74.09	65.92	98.48	87.08	84.85
Avg	93.78	94.08	94.04	66.77	91.00	76.40	65.92	98.90	87.62	85.39

Table 2. KLEJ results for RoBERTa-v2 base model

Run	NKJP	CDSC‑E	CDSC‑R	CBD	PolEmo‑IN	PolEmo‑OUT	DYK	PSC	AR	Avg
1	94.80	94.20	94.30	69.62	90.58	78.74	71.23	98.62	87.99	86.68
2	94.27	94.50	94.44	70.67	90.17	78.95	69.64	99.08	87.98	86.63
3	93.73	94.30	94.64	70.67	91.41	78.14	74.44	98.92	87.64	87.10
4	94.07	93.90	94.58	70.00	91.00	78.14	69.94	98.93	87.22	86.42
5	94.31	94.20	94.71	70.46	91.00	77.94	71.67	98.48	88.15	86.77
Avg	94.24	94.22	94.54	70.28	90.83	78.38	71.38	98.81	87.80	86.72

Table 3. KLEJ results for RoBERTa large model

Run	NKJP	CDSC‑E	CDSC‑R	CBD	PolEmo‑IN	PolEmo‑OUT	DYK	PSC	AR	Avg
1	94.31	93.50	94.63	72.39	92.80	80.54	71.87	98.63	88.82	87.50
2	95.14	93.90	94.93	69.82	92.80	82.59	73.39	98.94	88.96	87.83
3	95.24	93.30	94.61	71.59	91.41	82.19	75.35	98.64	89.31	87.96
4	94.46	93.20	94.96	71.08	92.80	82.39	70.59	99.09	88.60	87.46
5	94.46	93.00	94.82	69.83	92.11	83.00	74.85	98.79	88.65	87.72
Avg	94.72	93.38	94.79	70.94	92.38	82.14	73.21	98.82	88.87	87.69

Table 4. KLEJ results for RoBERTa-v2 large model

Run	NKJP	CDSC‑E	CDSC‑R	CBD	PolEmo‑IN	PolEmo‑OUT	DYK	PSC	AR	Avg
1	95.82	94.10	95.02	74.54	93.07	85.43	76.70	98.47	89.24	89.15
2	95.72	93.90	95.10	74.55	93.49	84.01	74.71	98.93	89.02	88.83
3	95.43	94.30	95.36	70.97	93.21	82.59	76.61	98.15	89.31	88.44
4	95.97	94.40	95.12	75.10	92.80	85.83	74.05	98.93	89.14	89.04
5	95.92	94.70	95.09	75.66	93.07	82.79	75.35	98.62	88.78	88.89
Avg	95.77	94.28	95.14	74.16	93.13	84.13	75.48	98.62	89.10	88.87

Table 5. KLEJ results for DistilRoBERTa model

Run	NKJP	CDSC‑E	CDSC‑R	CBD	PolEmo‑IN	PolEmo‑OUT	DYK	PSC	AR	Avg
1	93.54	93.00	93.57	67.60	88.78	77.53	61.79	93.59	87.54	84.10
2	93.78	92.30	93.54	69.20	90.03	78.95	61.98	95.92	87.23	84.77
3	93.15	93.20	93.55	68.47	89.61	79.15	60.05	95.15	87.19	84.39
4	93.29	93.30	93.30	68.99	90.17	78.34	62.18	95.80	86.83	84.69
5	93.68	93.40	93.62	69.57	90.44	77.73	61.11	96.26	87.44	84.81
Avg	93.49	93.04	93.52	68.77	89.81	78.34	61.42	95.35	87.24	84.55

Evaluation results on other tasks

Task	Task type	Metric	Base model (v1)	Large model (v1)
SICK-E	Textual entailment	Accuracy	86.13	87.67
SICK-R	Semantic relatedness	Spearman correlation	82.26	85.63
Poleval 2018 - NER	Named entity recognition	F1 score (exact match)	87.94	89.98
8TAGS	Multi class classification	Accuracy	77.22	80.84