Paper Title: Explaining Image Classifiers with Multiscale Directional Image Representation

Authors: Stefan Kolek, Robert Windesheim, Hector Andrade Loarca, Gitta Kutyniok, Ron Levie
Conference: CVPR 2023

Paper Contributions

Popular explanation methods such as GradCAM or Extremal Perturbations (smooth pixel space masks) produce overly smooth explanations that can only produce very rough localizations of relevant image regions. We introduce ShearletX and WaveletX, two new mask explanation methods for image classifiers that are able to overcome this limitation and seperate classifier relevant fine details in images without creating explanation artifacts. We also provide the first theoretical analysis and metrics for explanation artifacts of mask explanations. Moreover, we introduce Conciseness-Preciseness (CP) scores as a new metric for mask explanation goodness to measure the fidelity of a mask explanation adjusted for its conciseness.

ShearletX

ShearletX optimizes an explanation mask on the shearlet representation of the input image to extract the classifier relevant parts of the image. The optimization objective aims for a sparse shearlet mask, penallizes energy in the spatial domain of the explanation, and requires that the masked image approximately produces the same output as the unmasked image.

WaveletX

WaveletX differes from ShearletX only in that wavelets instead of sheralets are used. WaveletX is very similar to CartoonX but uses a spatial penalty term that resolves spatial ambiguities in CartoonX.

Explanation Artifacts

We characterize explanation artifacts as artificial edges, i.e., edges in the masked image that do not occur in the original image. Such artificial edges can form hallucinated patterns that activate a class label but do not explain the actual classification decision. We quantify the artificial edges with a new metric that we call Hallucination Score.

CP-Scores

We introduce the conciseness-preciseness (CP) scores as a new information theoretic metric to evaluate the goodness of mask explanations. CP scores measure the fidelity of the explanation adjusted for their conciseness.

Setup

Python 3.7.x and newer are supported:

# Clone project   
git clone https://github.com/skmda37/ShearletX.git 

# Enter directocry
cd ShearletX

# Create and activate virtual environment (or conda environment)
python -m venv env
source env/bin/activate   

# install pytorch wavelets package (see https://pytorch-wavelets.readthedocs.io/en/latest/readme.html for the docs)
git clone https://github.com/fbcotter/pytorch_wavelets
cd pytorch_wavelets
pip install .
pip install -r tests/requirements.txt
pytest tests/
cd ..

# install other project dependencies from requirements file   
pip install -r requirements.txt

Contents

1. code/: Contains the code to use ShearletX and WaveletX, and reproducing the paper experiments.
2. imgs/: Contains all images for this README.md.

How to Run?

First, do cd code/. Then you can either:

1. Explain models with ShearletX and WaveletX in visualize_example_explanations.ipynb





2. Visualize explanation artifacts in visualize_explanation_artifacts.ipynb





3. Reproduce the scatterplot experiments from Figure 4 by running python scatterplot.py. This will produce the scatterplots with the following different settings that were used in the paper:
   • Model: Resnet18, VGG19, or MobilenetV3
   • Area size for Smoothmask: 0.05, 0.1, 0.2
   The scatterplots will be saved in the folder ./code/scatterplot_figures.

Cite

@inproceedings{kolek2023explaining,
  title={Explaining Image Classifiers with Multiscale Directional Image Representation},
  author={Kolek, Stefan and Windesheim, Robert and Andrade Loarca, Hector and Kutyniok, Gitta and Levie, Ron},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2023},
  organization={IEEE}
}

License

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.