Model Compression Toolkit (MCT) is an open-source project for neural network model optimization under efficient, constrained hardware.
This project provides researchers, developers, and engineers tools for optimizing and deploying state-of-the-art neural networks on efficient hardware.
Specifically, this project aims to apply quantization to compress neural networks.
MCT is developed by researchers and engineers working at Sony Semiconductor Israel.
For more information and detailed API, please visit our project website.
MCT supports different quantization methods:
- Post training quantization (PTQ): Keras API, PyTorch API
- Gradient-based post training quantization (GPTQ): Keras API, PyTorch API
- Quantization aware training (QAT)*
| Quantization Method | Complexity | Computational Cost |
|---|---|---|
| PTQ | Low | Low (order of minutes) |
| GPTQ | Mild (parameters fine-tuning using gradients) | Mild (order of 2-3 hours) |
| QAT | High | High (order of 12-36 hours) |
In addition, MCT supports different quantization schemes for quantizing weights and activations:
- Power-Of-Two (hardware-friendly quantization [1])
- Symmetric
- Uniform
Core features:
- Graph optimizations: Transforming the model to an equivalent (yet, more efficient) model (for example, batch-normalization layer folding to its preceding linear layer).
- Quantization parameter search: Different methods can be used to minimize the expected added quantization-noise during thresholds search (by default, we use Mean-Square-Errorm but other metrics can be used such as No-Clipping, Mean-Average-Error, and more).
- Advanced quantization algorithms: To prevent a performance degradation some algorithms are applied such as:
- Shift negative correction: Symmetric activation quantization can hurt the model's performance when some layers output both negative and positive activations, but their range is asymmetric. For more details please visit [1].
- Outliers filtering: Computing z-score for activation statistics to detect and remove outliers.
- Clustering: Using non-uniform quantization grid to quantize the weights and activations to match their distributions.*
- Mixed-precision search: Assigning quantization bit-width per layer (for weights/activations), based on the layer's sensitivity to different bit-widths.
- Visualization: You can use TensorBoard to observe useful information for troubleshooting the quantized model's performance (for example, the model in different phases of the quantization, collected statistics, similarity between layers of the float and quantized model and bit-width configuration for mixed-precision quantization). For more details, please read the visualization documentation.
Some features are experimental and subject to future changes.
For more details, we highly recommend visiting our project website where experimental features are mentioned as experimental.
This section provides a quick starting guide. We begin with installation via source code or pip server. Then, we provide a short usage example.
See the MCT install guide for the pip package, and build from the source.
git clone https://github.com/sony/model_optimization.git
python setup.py install
pip install model-compression-toolkit
A nightly package is also available (unstable):
pip install mct-nightly
To run MCT, one of the supported frameworks, Tenosflow/Pytorch, needs to be installed.
For using with Tensorflow please install the packages: tensorflow, tensorflow-model-optimization
For using with PyTorch please install the packages: torch
Also, a requirements file can be used to set up your environment.
Currently, MCT supports compressing models of TensorFlow and PyTorch, and is tested on various versions:
| TensorFlow Version | PyTorch Version |
|---|---|
 |
 |
 |
 |
 |
 |
For an example of how to use the post-training quantization, using Keras, please use this link.
For an example using PyTorch, please use this link.
For more examples please see the tutorials' directory.
As part of the MCT library, we have a set of example networks on image classification. These networks can be used as examples when using the package.
- Image Classification Example with MobileNet V1 on ImageNet dataset
| Network Name | Float Accuracy | 8Bit Accuracy | Comments |
|---|---|---|---|
| MobileNetV1 [2] | 70.558 | 70.418 |
For more results please see [1]
We quantized classification networks from the torchvision library. In the following table we present the ImageNet validation results for these models:
| Network Name | Float Accuracy | 8Bit Accuracy |
|---|---|---|
| MobileNet V2 [3] | 71.886 | 71.444 |
| ResNet-18 [3] | 69.86 | 69.63 |
| SqueezeNet 1.1 [3] | 58.128 | 57.678 |
MCT aims at keeping a more up-to-date fork and welcomes contributions from anyone.
*You will find more information about contributions in the Contribution guide.
[1] Habi, H.V., Peretz, R., Cohen, E., Dikstein, L., Dror, O., Diamant, I., Jennings, R.H. and Netzer, A., 2021. HPTQ: Hardware-Friendly Post Training Quantization. arXiv preprint.
[2] MobilNet from Keras applications.