From 76d67fae000c09c4d3874fea29510897dfe0ba76 Mon Sep 17 00:00:00 2001 From: Shriya Palsamudram Date: Fri, 12 Jul 2024 10:42:00 -0700 Subject: [PATCH] Formalize rule regarding benchmark lifecycle Address comment https://github.com/mlcommons/policies/pull/171#issuecomment-2102731288 --- submission_rules.adoc | 3 +++ 1 file changed, 3 insertions(+) diff --git a/submission_rules.adoc b/submission_rules.adoc index a8e5cd0..8446264 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -779,8 +779,11 @@ Otherwise, the committee will designate an investigator with no conflict of inte 2. The result-in-question is moved to open for noncompliance with the rules. 3. The result-in-question is removed due to intentional cheating. +## Benchmark Lifecycle +Developing high-quality benchmarks requires significant effort, computational resources, and commitment. Therefore, each benchmark is expected to remain part of the benchmark suite for a minimum of two years or four submission rounds, whichever comes first. +A benchmark may be considered for early retirement due to reasons such as, but not limited to, low industry adoption. Early retirement requests will be reviewed by the relevant working group, followed by a formal vote to determine the benchmark's status. ## Appendices