fix: add italian HateSpeech dataset

[rbroc]

Checklist for adding MMTEB dataset

Reason for dataset addition: This is a highly curated hate speech detection dataset, with binary (hateful/non-hateful) labels, as well as fine-grained labels for 25+ hate speech categories (https://huggingface.co/datasets/Paul/hatecheck-italian).
I am currently only using the binary labels, but it would be cool to also test models on the more fine-grained taxonomy (could just subclass the current task, and set the relevant column as label.

There are similar datasets for 9 more languages, released as separate datasets on HF. If this looks like a suitable dataset, I can go on to submit the remaining (through separate PRs, I assume).

Checklist

• I have tested that the dataset runs with the mteb package.
• I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• I have considered the size of the dataset and reduced it if it is too big (2048 examples is typically large enough for most tasks)
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.
• I have added points for my submission to the POINTS.md file.

Questions

• The dataset does not have a native training split (it only comes with a test split). I am currently splitting into equally sized (1845 sentences) train/test splits, stratifying by hate speech type, but we could consider alternative options.
• Don't know what happened with results/intfloat__multilingual-e5-small/model_meta.json but can revert if needed

[KennethEnevoldsen]

Lovely to see you here @rbroc! Everything is looking good. There are a few comments, but generally it looks good!

[rbroc]

thanks @KennethEnevoldsen! I've left a few comments open above -- especially a question about the multilingual nature of the dataset. Happy to double review points since I am explicitly re-requesting :)

[KennethEnevoldsen]

Happy to double review points since I am explicitly re-requesting :)

No problem, let us just keep it at one.

[rbroc]

last quick q @KennethEnevoldsen: are my points for this 2 + 4 (2 for new dataset + 4 point bonus) or 4 in total?
if it looks good, i'll go ahead, update points if needed, and merge and work on a separate PR for the multilingual expansion.

[KennethEnevoldsen]

Looks good!

re. points it is 6 (2+4) in total.

[rbroc]

workflows require approval, should be ready to merge after that.

[KennethEnevoldsen]

The dataset does not have a native training split (it only comes with a test split). I am currently splitting into equally sized (1845 sentences) train/test splits, stratifying by hate speech type, but we could consider alternative option

Hei Roberta, just saw this comment. I actually would prefer that you do not do the split, but rather just shuffle and downsample from the training. Then also switch the eval_split to "train". This is to avoid someone training on the "train" split assuming that we test on another split (which we don't).

[rbroc]

The dataset does not have a native training split (it only comes with a test split). I am currently splitting into equally sized (1845 sentences) train/test splits, stratifying by hate speech type, but we could consider alternative option

Hei Roberta, just saw this comment. I actually would prefer that you do not do the split, but rather just shuffle and downsample from the training. Then also switch the eval_split to "train". This is to avoid someone training on the "train" split assuming that we test on another split (which we don't).

not sure I fully understand. the dataset has no training split, it only comes with a test split. could also have a chat about this IRL if you think that's more productive.

[KennethEnevoldsen]

Ahh then this is perfectly fine. Merged into a temporary branch to run linting and then merging.