[ Back to index ]
Collective Knowledge concept (CK) is motivated by our tedious experience reproducing 150+ Systems and Machine Learning papers and automating MLPerf benchmarks.
We have spent many months communicating with researchers and developers to understand their technical reports, README files, ad-hoc scripts, tools, command lines, APIs, specifications, dependencies, data formats, models and data to be able to reproduce their experiments and reuse their artifacts across continuously changing software, hardware and data.
This experience motivated us to develop a simple and cross-platform meta-framework (Collective Knowledge) that can help researchers and engineers solve above problems by transforming their projects into a database of portable, reusable and customizable components.
All such components provide a unified CLI, Python API and extensible JSON/YAML meta descriptions to existing artifacts, native scripts and workflows to make them more interoperable, deterministic, reusable and reproducible across continuously changing software and hardware.
We have donated the CK framework to the MLCommons foundation
to benefit everyone after it was successfully validated by Qualcomm, Arm, General Motors,
OctoML, Krai, HPE, Dell, Lenovo and other partners
to enable collaborative and reproducible development, optimization and deployment
of Pareto-efficient ML Systems in terms of accuracy, latency, throughput, energy, size and costs.
After this approach was successfully validated by Qualcomm, Arm, General Motors, OctoML, Krai, HPE, Dell, Lenovo and other organizations to modularize and automate MLPerf benchmarks, we have donated our prototype to the MLCommons foundation to continue developing it as a community effort.
Collective Mind workflow automation meta-framework (CM aka CK2) is the 2nd implementation of the CK concept being developed by the open MLCommons taskforce on automation and reproducibility.
This taskforce is using and enhancing CM to modularize ML and AI systems and automate their benchmarking, optimization and deployment across continuously changing software, hardware and data.
- Blog article about "MLOps Is a Mess But That's to be Expected" (March 2022)
- Journal article describing the CK concept
- "Reproducing 150 Research Papers and Testing Them in the Real World" (ACM TechTalk; Feb 2021)
- "Automating MLPerf design space exploration and production deployment" (HPCA'22)
- "Collaboratively Benchmarking and Optimizing Deep Learning Implementations" (General Motors; Jun 2017)
- MLOps projects, articles and tools
We would like to thank MLCommons, OctoML, all contributors and collaborators for their support, fruitful discussions, and useful feedback! See more acknowledgments in the CK journal article and our ACM TechTalk.