Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add another SUT or two #25

Closed
wpietri opened this issue Dec 18, 2023 · 2 comments
Closed

Add another SUT or two #25

wpietri opened this issue Dec 18, 2023 · 2 comments

Comments

@wpietri
Copy link
Contributor

wpietri commented Dec 18, 2023

Yifan's suggestions:

I'd suggest openai/gpt-3.5-turbo-0613 next (which is relatively inexpensive)
Oh, also EleutherAI/pythia-70m is also small enough to work on CPU
but it is not a built in model in HELM yet, so that forces us to think about how the SUT plugins will work, which is maybe good?
on the HELM CLI, you would run pythia-70m [like so](https://crfm-helm.readthedocs.io/en/latest/huggingface_models/)
GPU models: pythia, mpt, llama 2
llama 2 and the openai models also need credentials, so that also forces us to think about how credentials should be managed
@yifanmai
Copy link
Contributor

yifanmai commented Dec 18, 2023

Open source Hugging Face SUTs:

Closed source:

@yifanmai
Copy link
Contributor

Open source remote inference projects that use the OpenAI API format:

@wpietri wpietri closed this as completed Feb 3, 2024
bkorycki pushed a commit that referenced this issue Sep 12, 2024
It doesn't really do much now, but my next steps are to make BBQ real, so doing this first to make that cleaner.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants