Benchmark outcomes record #392
Conversation
…benchmark_outcomes_record # Conflicts: # tests/test_record.py
Make modelgauge's notion of a SUT know how to instantiate itself and cache the instance used, so that the initalization info is available later.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
…re accurate. Add `run_uid`. Remove some duplication in the JSON. Add JSON output to normal benchmark run.
|
Ok, @bkorycki and @dhosterman, this is actually ready for final review now. |
Make modelgauge's notion of a SUT know how to instantiate itself and cache the instance used, so that the initalization info is available later.
…re accurate. Add `run_uid`. Remove some duplication in the JSON. Add JSON output to normal benchmark run.
…benchmark_outcomes_record
| @@ -55,6 +55,18 @@ class ModelGaugeSut(SutDescription, Enum): | |||
| WIZARDLM_13B = "wizardlm-13b", "WizardLM v1.2 (13B)", TogetherChatSUT, "WizardLM/WizardLM-13B-V1.2" | |||
| # YI_34B_CHAT = "yi-34b", "01-ai Yi Chat (34B)", TogetherChatSUT, "zero-one-ai/Yi-34B-Chat" | |||
|
|
|||
| def instance(self, secrets): | |||
There was a problem hiding this comment.
Why do we need these methods?
There was a problem hiding this comment.
Moving instance creation here will let me unify duplicate code, and it gives me a place to cache the instance actually used for the run, which is needed to dump out the outcome JSON.
src/modelbench/uid.py
Outdated
| import casefy | ||
|
|
||
|
|
||
| class HasUid: |
There was a problem hiding this comment.
I left a few comments about this in the other PR!
Make modelgauge's notion of a SUT know how to instantiate itself and cache the instance used, so that the initalization info is available later.
…re accurate. Add `run_uid`. Remove some duplication in the JSON. Add JSON output to normal benchmark run.
…benchmark_outcomes_record
|
Ok @dhosterman and @bkorycki, I think I have resolved all the outstanding issues and requests on this one. |
|
This works great so far, but it fails when attempting to use --anonymize. |
|
I also notice that in the |
Produces a JSON version of the benchmark alongside the HTML files. Not sure this is totally right; Looking forward to feedback on the format.