Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Run OpenAI compatible API server for eval harness #30

Open
PicoCreator opened this issue Sep 14, 2023 · 1 comment
Open

[Feature] Run OpenAI compatible API server for eval harness #30

PicoCreator opened this issue Sep 14, 2023 · 1 comment

Comments

@PicoCreator
Copy link
Collaborator

PicoCreator commented Sep 14, 2023

RWKV-infctx-trainer uses its own inference code within its own codebase here:

This intentionally avoid the rwkv python inference project, as it is meant to be usable for testing/debugging architecture changes (which we may not port over to the inference project). While being terrible unoptimized (its ok!)

For example in rwkv-x-playground branch : https://github.com/RWKV/RWKV-infctx-trainer/tree/rwkv-x-playground
Every single v5XYZ is an experiment, some which will be abandoned.

What we need is to integrate this for evals, as such the following is recommended

Ideally we should have options to run only subset of the evals, so we can "fan out the eval runs" across multiple nodes if needed via github CI down the line.

@PicoCreator
Copy link
Collaborator Author

The long term goal, is to allow all the top harness to be automatically ranned against a trained model in a CI pipeline

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant