Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Results Guidelines for unverified claims and misrepresentation of verified results #137

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open
4 changes: 3 additions & 1 deletion MLPerf_Results_Messaging_Guidelines.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@ If you used an MLPerf™ benchmark to obtain a result for your product or servic

Since your results have not gone through MLCommons review and, therefore, have not been verified by MLCommons, you must indicate your results are not verified by using the term “unverified” next to each result score and by using the following language when publishing or otherwise discussing your results: “_Result not verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below.

Any unverified or Non-MLCommons Reviewed results derived for a competitor product or service for use in public claims must use the competitor's submission code repository if one exists instead of reference code repository. Any modifications made to such a third party MLPerf repo for the purpose of obtaining public unverified comparisons must be made in good faith and be made publicly available.

== Use of MLPerf™ Benchmark for MLCommons Reviewed and Verified Results

If you used an MLPerf benchmark to obtain a result for your product or service, you submitted your result for MLCommons review, and your result was verified through such review, you may indicate your results are verified when publishing or otherwise discussing your results, by indicating your results are “verified” or “official” or by otherwise following the examples below for verified results. You may also choose to use this language: “_Result verified by MLCommons Association._” You can include this statement in a footnote, as described in Section 3 below.
Expand Down Expand Up @@ -202,7 +204,7 @@ The MLPerf mark is owned by MLCommons, and only MLCommons and its authorized lic

Any MLCommons Member, Test Partner, or third party may report a violation of these Guidelines via email to the MLCommons Executive Director (“ED”) & Working Group (“WG”) chairs of the appropriate benchmark. Upon confirming the violation in their discretion, ED & WG chairs would inform the potential violator and request remedial action. If the ED, WG chairs, and potential violator are unable to reach a mutually satisfactory conclusion, the issue can be raised in WG to seek resolution via WG vote.

A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark.
Violating content must be taken down within first 24 hours of violation being reported. A non-exhaustive list of possible remedial actions or penalties based on the degree of violation is noted below. Taking or not taking any or all actions on this list or otherwise does not constitute a waiver of any enforcement rights or other rights in the MLPerf benchmarks, software, and/or trademark.

1. Requesting corrections to published materials in the form of marketing blog posts, journals, papers, and other media.
2. If the violation was at a public event such as a conference, the WG may direct the violator to issue a public statement to correct claims in ways that conform to these Guidelines.
Expand Down