fully parallel benchmark runner (#441)

* Initial push at a parallel benchmark runner pipeline. There are some awkward parts where we work around the modelgauge set of objects. My theory is that we'll push changes into modelgauge once we have a clear plan. * Adding in handling of exceptions for suts and annotators. Exception now gets passed along and stored in the TestRun. * Clearing up relationships between test and benchmark versions of Run/Runner.
mlcommons · Sep 10, 2024 · c76dad4 · c76dad4
1 parent d63c473
commit c76dad4
Show file tree

Hide file tree

Showing 5 changed files with 747 additions and 7 deletions.
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -47,7 +47,7 @@ packages = [
 
 [tool.poetry.dependencies]
 python = ">=3.10,<3.13"
-modelgauge = ">=0.6.1"
+modelgauge = { extras = ["demo"], version = "^0.6.1" }
 jq = "^1.6.0"
 click = "^8.1.7"
 casefy = "^0.1.7"