OOD-calibration

This is the implement of paper: Adaptive Calibrator Ensemble for Predictive Calibration Under Distribution Shift

Dataset

We list the links of the used datasets and check carefully their licenses for our usage: ImageNet-Validation, ImageNet-V2, ImageNet-C(orruption), ImageNet-S(ketch), ImageNet-Adv(ersarial), ImageNet-R(endition), CIFAR-10, CIFAR-10-C.

Preperation

Logits are generated with "test_(model)_(dataset.py)" and "gen_ood_logits.py". Logits are stored in logits dir.

resnet152_files = [
                    # 'probs_resnet152_imgnet_logits.p',
                   'cal_resnet152_imgnet-v2-a_logits.p',
                   'cal_resnet152_imgnet-v2-b_logits.p',
                   'cal_resnet152_imgnet-v2-c_logits.p',
                   'cal_resnet152_imgnet-s_logits.p',
                   'cal_resnet152_imgnet-a_logits.p',
                   'cal_resnet152_imgnet-r_logits.p',
                   ]

(y_probs_val, y_val), (y_probs_test, y_test) = unpickle_probs(path)

Usage

Run our ensemble method with these three baselines (i.e., Vector Scaling, Temperature Scaling, Spline):

python run_ensemble.py --files <'resnet50_files', 'resnet152_files', 'vit_small_patch32_224_files', 'deit_small_patch16_224_files'>

Run our alternative method with these three baselines (i.e., Vector Scaling, Temperature Scaling, Spline):

python run_estimate_ratio.py --files <'resnet50_files', 'resnet152_files', 'vit_small_patch32_224_files', 'deit_small_patch16_224_files'>

Bibtex

@InProceedings{Zou_2023_ICCV,
    author    = {Zou, Yuli and Deng, Weijian and Zheng, Liang},
    title     = {Adaptive Calibrator Ensemble: Navigating Test Set Difficulty in Out-of-Distribution Scenarios},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {19333-19342}
}