Add a self-calibration run #110
Conversation
…his makes it less likely to crash.
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
src/coffee/run.py
Outdated
| type=click.Path(file_okay=True, dir_okay=False, path_type=pathlib.Path), | ||
| ) | ||
| def calibrate(update: bool, file) -> None: | ||
| print("current standards") |
There was a problem hiding this comment.
Do we want to standardize on using click.echo instead of print? It looks like the benefits are fairly negligible, but are present, and I believe newhelm is using click.echo.
| help="Run benchmarks for the reference sut and update the reference scores.", | ||
| ) | ||
| @click.option( | ||
| "--file", |
There was a problem hiding this comment.
For --update and --file, maybe these could be more explicit about which file and what update they're doing? --update-reference-scores is unwieldy, but explicit, and could have a short version. --file could use some help.
There was a problem hiding this comment.
Good idea. Honestly, I'm not sure it should have a file option at all, so perhaps we should get rid of it. I've added some help text for both command and option, which should make it clearer what they're for.
This pulls the calibration constants out of the code and into a JSON config file and then adds a command to generate it. The current numbers are a bit suspect, as they use a very small number of examples. Once we solve some NewHELM issues, we should be able to increase that.