Why no results for closed-sourced models?

[mrconter1]

Hi!

Why don't you benchmark closed models? Is there any project that does that?

Best regards

[LSinev]

No results? This framework is not providing any result by itself, AFAIK. Though there are multiple projects that use it as a workhorse to benchmark different models.

For now, there is 2 API provided for closed models in https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/models and there is a template to add more.
There are following PRs waiting to be added (or reviewed or some other decision):
#395
#834
#936
#1996