Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do more to prevent ties #36

Open
KennethEnevoldsen opened this issue Aug 12, 2024 · 2 comments
Open

Do more to prevent ties #36

KennethEnevoldsen opened this issue Aug 12, 2024 · 2 comments

Comments

@KennethEnevoldsen
Copy link

When rating in the benchmark, we do get a lot of ties among the top models. A solution is probably to remove autogenerated examples for which the models agree. This would give us more a richer annotations scheme.

@isaac-chung
Copy link
Contributor

By "autogenerated examples" do you mean e.g. the "examples" on the "Arena (battle)" screen?

@KennethEnevoldsen
Copy link
Author

We can keep the pre-defined examples, but I was mainly thinking about the ones gained from the "random sample"-button.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants