Skeleton benchmark 1.0

[bkorycki]

The primary difference between 0.5 and 1.0 seems to be the inclusion of additional languages. WG1 says scores from different languages should not be aggregated, so I envision each language to be it's own benchmark. This will require some re-factoring of modelgauge hazards as well.

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[wpietri]

As we discussed in the standup, let's go for more with this. In particular, I'm hoping to see most or all of the bullet points in the issue: #398

[wpietri]

Oh, sorry, I didn't realize you had already bumped the modelgauge version in here when I started in on a PR for that. Let's get this merged and then maybe drop my PR if it's duplicative.

[wpietri]

Before I dive in to review, could you say how much of #398's bullet points are in this PR?

[bkorycki]

@wpietri

Before I dive in to review, could you say how much of #398's bullet points are in this PR?

• at least 3 prompts ✅
  • synthetic prompts from workstream 3? ❌ (they are fully fake)
  • not ground truth prompts ✅
• at least one hazard from workstream 1's definitions ✅ (dfm-- defamation)
• 1 test per hazard ✅
• llama guard 2 to start ✅
• hazard score is fraction unsafe ✅
• personas are all combined ✅
• benchmark scoring: use same reference models and approach as in 0.5, but in separate code ✅
• benchmarks are separated by language✅ and persona❌? Start with english and normal-ish persona
  • Modelgauge separates tests by language. So far there is just one for English.
  • The personas are grouped together right now because there's some conflicting info regarding this point

[wpietri]

Looks great! Thanks for going the distance.