README.md

Documentation

The MLCommons Task Force on Automation and Reproducibility is developing 3 projects:

• Collective Mind automation language (CM)
• Collective Knowledge Playground (CK)
• Modular Inference Library (MIL)

The goal is to help the community make it easier to reproduce research projects, automate benchmarking and optimization of AI/ML systems, and simplify transfer of research ideas to production across rapidly evolving software, hardware and data.

See the ACM REP'23 keynote and MLPerf submitters orientation for more details.

The development is led by Grigori Fursin and Arjun Suresh.

Copyright: 2021-2023 MLCommons

License: Apache 2.0

Collective Mind automation language (CM)

• Introduction
• Installation and customization
• Unified CLI and Python API
  • CM "script" automation
  • CM "cache" automation
  • CM "experiment" automation
  • List of all unified CM automations from MLCommons
  • List of all portable and reusable CM scripts from MLCommons
• Debugging
• Real-world use cases
• Tutorials
• Specifications
• Source code
• FAQ

Collective Knowledge playground (CK)

Note that this collaborative platform is under heavy development.

• Introduction
• Open access via x.cKnowledge.org
  • Participate in reproducibility and optimization challenges
    • GUI to run MLPerf inference benchmarks
    • GUI to prepare MLPerf inference submissions
    • Prototype of the LLM-based assistant to run MLPerf benchmarks out-of-the-box
  • See current leaderboard
  • Visualize and compare all past MLPerf results with derived metrics
• Development page
• Source code (CM script)

Modular Inference Library (MIL)

Note that this library is under heavy development.

• Development link

Collaborative development

This open-source technology is being developed by the MLCommons Task Force on Automation and Reproducibility, cTuning.org and cKnowledge.org.

• Check our ACM REP'23 keynote and Forbes article to learn more about our vision and development plans.
• Join our public Discord server and public conf-calls.
• Follow our news.
• Read about the previous version of the MLCommons CM automation language: cTuning CK framework (2014-2022).