[EVAL] Add RewardBench

[lewtun]

Evaluation short description

• Why is this evaluation interesting?

RewardBench is perhaps the only evaluation suite to provide broad coverage of the strengths and weaknesses of reward models across domains like reasoning, chat, and safety. It would be great to have this in lighteval so one can have a unified evaluation suite and not have to hop across different repos and dependencies.

• How used is it in the community?

RewardBench is the de facto leaderboard for comparing reward models and is widely used by the post-training subset of the community for advanced methods like RL, rejection sampling, and others.

Evaluation metadata

Provide all available

• Paper url: https://arxiv.org/abs/2403.13787
• Github url: https://github.com/allenai/reward-bench
• Dataset url: https://huggingface.co/datasets/allenai/reward-bench

Note: this eval is not a typical LLM eval since it relies on sequence classification. I'm not sure if that is out of scope for lighteval

[clefourrier]

It's in scope for the generative models, not as much for the classifier ones (as we would need to add a whole new pipeline for loading and running them) - do you think having it only for gen models would already be interesting enough?

[lewtun]

It's in scope for the generative models, not as much for the classifier ones (as we would need to add a whole new pipeline for loading and running them) - do you think having it only for gen models would already be interesting enough?

OK I think that would be too restrictive since most reward models today are currently trained as sequence classifiers. Feel free to close the issue if you want :)