Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix whitespace in the bigbench #47

Open
wants to merge 3 commits into
base: develop-hf
Choose a base branch
from

Conversation

Red-Caesar
Copy link
Collaborator

The condition to break the loop has not worked before if there is leading whitespace in the token.

@Red-Caesar Red-Caesar changed the title Remove leading whitespace in checks Fix whitespace in the bigbench May 14, 2024
@Red-Caesar
Copy link
Collaborator Author

Usually, for loglikelihood tasks, two arguments are used: context and continuation. Together, they are submitted as a single query in the LLM for calculation: context + continuation. In all benchmarks, the context doesn't have a space at the end, while the continuation starts with one. For example:

context="A:"
continuation=" zero"

However, it was found that in BigBench, the spaces are the opposite:

context="A: "
continuation="zero"

This disrupts the logic of calculating loglikelihood in the harness.

New mesaurements of the bigbench will be here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant