Skip to content

Latest commit

 

History

History
102 lines (89 loc) · 6.46 KB

README.md

File metadata and controls

102 lines (89 loc) · 6.46 KB
Raw

A Modular System for Enhanced Robustness of Multimedia Understanding Networks via Deep Parametric Estimation, ACM Multimedia Systems 2024.

This repository contains the code to reproduce the results presented in our paper and in the presentation blog at the Samsung Research Website.

Comparison with standard approach SyMPIE architecture

If you find any of them useful for your research please consider citing us:

@article{
  barbato2024modular,
  title={A Modular System for Enhanced Robustness of Multimedia Understanding Networks via Deep Parametric Estimation},
  author={Barbato, Francesco and Michieli, Umberto and Yucel, Mehmet Karim and Zanuttigh, Pietro and Ozay, Mete},
  journal={Proceedings of the ACM conference on Multimedia Systems (MMSys)},
  doi={10.1145/3625468.3647623},
  year={2024}
}

Note that due to CUDA-related instabilities the per-corruption numbers you obtain after training may not match those shown in Table 2, but the overall accuracy and improvement will.

Prerequisites

  1. Clone this repository.
  2. Create a new python 3.10 environment using conda (anaconda, miniconda): conda create -n sympie python==3.10.14
  3. Install the dependencies using pip: pip install -r requirements.txt
  4. Download the pretrained checkpoints and place them into training/checkpoints:
    1. Segmentation pretraining
    2. SyMPIE pretrained
    3. AugMIX pretrained (optional)
    4. PRIME pretrained (optional)
    5. PIXMIX pretrained (optional)
  5. To train the architecture download the following datasets (optional):
    1. ImageNet, validation set needs to be formatted in a PyTorch-friendly way:
      1. Download and extract the dataset.
      2. Navigate to Data/CLS-LOC.
      3. Rename the val folder into val_raw
      4. Create a new empty val folder.
      5. Copy into the current folder the make_clas_val.py script provided in the root of this repo.
      6. Run the script.
    2. Cityscapes
    3. ACDC
  6. To validate the architecture download the following datasets:
    1. ImageNetC
    2. ImageNetC-Bar (optional)
    3. ImageNetC-Mixed (optional)
    4. CityScapes (optional)
    5. ACDC (optional)
    6. DarkZurich (optional)
  7. Move the text splits provided in data in the appropriate folder for each dayaset you downloaded.
  8. Update the paths listed in the config_paths.yaml with the ones for your datasets.

Training

To train our architecture with the default configuration run the following command from classification/training.

torchrun --nproc-per-node=8 train.py

During training the tensorboard logs are saved in classification/training/logs.

For the evaluation we will assume that, after training, the checkpoint file final.pth has been copied from your log folder to classification/training/checkpoints with the name sympie_rerun.pth.

Evaluation

To run the evaluation navigate to classification/eval. From there you can start validation with:

python validate.py

the default configuration uses a ResNet50 classifier with no SyMPIE module enabled. This is needed to compute the baseline against which the deltas generate by compare.py will be computed.

After completing the baseline evaluation you can get the accuracy for the default SyMPIE checkpoint by:

python validate.py --colm sympie

this will use the classification/training/checkpoints/sympie.pth checkpoint. To use your re-trained checkpoint simply change --colm sympie to --colm sympie_rerun.

To change the classifier you can set the --clas argument choosing from the following:

argument value classifier used
rn50_1 ResNet50 V1
rn50_2 ResNet50 V2
yucel ResNet50 Hybridaugment++ (ICCV23)
rn18 ResNet18 V1
rn34 ResNet34 V1
vgg13bn VGG13 with BN
vgg13 VGG13
vgg16bn VGG16 with BN
vgg16 VGG16
mnet_1 MobileNetV3-Large V1
mnet_2 MobileNetV2-Large V2
vitb16 ViT-B16
swint Swin-T
prime ResNet50 PRIME (ECCV22)
pixmix ResNet50 PIXMIX (CVPR21)
clip CLIP

To validate on ImageNetC-Bar run the script with the --bar flag.

SyMPIE (A Modular System for Enhanced Robustness of Multimedia Understanding Networks via Deep Parametric Estimation) by Francesco Barbato, Umberto Michieli, Mehmet Yucel, Pietro Zanuttigh, Mete Ozay is licensed under CC BY-NC-SA 4.0