How Calibration Affects Accuracy After Quantization

Authors: Guoxuan Xia (correspondence to [email protected]), Sangwon Ha, Tiago Azevedo and Partha Maji.

Paper

Work undertaken as an intern at Arm Research, and accepted at the ICBINB workshop at NeurIPS 2021.

Description

Calibration and quantization are two seemingly separate concepts in deep learning. We find that in fact the robustness of accuracy to post-training quantization quantization is closely linked to confidence and calibration.

• Higher confidence predictions are less likely to change post-quantization.
• Higher confidence predictions will be more accurate.

We can understand post-training quantization as injecting noise into the weights and activations of a neural network. We can then see that high confidence, high accuracy predictions are less likely to change post-quantization, and low confidence, lower accuracy predictions are more likely to change. Thus accuracy remains robust to quantization.

Note that accuracy stays constant for both cases above.

Implementation Details

The repository uses vanilla PyTorch, with the exception of quantization. Although we only consider post-training quantization, we simulate this using the quantization-aware training functionality that already exists in PyTorch's built-in quantization functionality. Code can be found in utils/quant_utils.py and test.py.

Experiments

We provide scripts to replicate the experiments found in the paper, for:

• CIFAR-100: ResNet20, ResNet56
• ImageNet: ResNet50, MobileNetV2

We provide weights for trained CIFAR-100 models, as well as scripts for training from scratch. Weights for ImageNet models should be downloaded from PyTorch (using the above links) and placed in directories named:

• models/saved_models/resnet50_imagenet
• models/saved_models/mobilenetv2_imagenet

Before running the scripts, you need to edit the .json files in experiment_configs.

• "datapath" within "id_dataset" needs to be changed to point to the corresponding dataset in your system (in the same way you would pass the directory to the torchvision dataset).
• "results_savedir" within "test_params" should point to a (existing) directory where you want results to be saved. Note that logits for the ImageNet validation set will be saved, which are approximately 1.2GB per evaluation.

To run the scripts, navigate to experiment_scripts, make them executable using chmod +x * and run with ./<script>.sh

• <model>_<dataset>_train.sh will train and save a set of weights in models/saved_models.
• <model>_<dataset>_test.sh will evaluated a trained model at different levels of quantization, and then save the logits for all predictions. The logits are then loaded and used to generate plots.

Training and test parameters can be found in the corresponding .json files in experiment_configs. GPU IDs supplied as command line arguments will override those supplied in .json files.

Requirements

In order to run this project first clone the repository:

git clone https://github.com/Guoxoug/PTQ-acc-cal.git

The main requirements are:

python>=3.6
pytorch>=1.7
numpy
matplotlib
seaborn
scikit-learn
tqdm

You can create a working conda environment by using the command:

conda env create --file env.yml

Citation

If you find this code useful in your project please cite our paper:

@miscconda{xia2021underexplored,
      title={An Underexplored Dilemma between Confidence and Calibration in Quantized Neural Networks}, 
      author={Guoxuan Xia and Sangwon Ha and Tiago Azevedo and Partha Maji},
      year={2021},
      eprint={2111.08163},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}