adds gradcam tutorial

[wyli]

fixes #76

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[wyli]

@rijobro I tried the occlusion sensitivity it's currently very slow for 3D, I guess we need something to occlude with a user-specified step, so that the map will be generated at a low resolution, then we can upsample the final map to the original image size

[rijobro]

Thanks, I'll have a look now!

Yes, generating a low-resolution grid followed by upsampling would be great. I think so far I sidestepped the slowness by allowing the user to optionally occlude within a bounding box, meaning you could just occlude a 2D slice if you wanted.

[rijobro]

Just made a PR with some suggested changes. Few things I noticed:

1. Notebook is in folder CAM, might be better to call it interpretability or something similar so we can bundle similar ideas together.
2. Couldn't check with the full data as the server was being slow (>8h for download)
3. LMDBDatabase is quite verbose for me, and is prone to error if the folder cache exists from a previous run. I had more joy by switching to PersistentDataset (not in my PR)
4. epoch_loss_values = metric_values = [] will create 1 list with shared memory
5. GradCAM is in visualisation, but occlusion sensitivity is in metrics. These should both probably be in the same place.

Just had a look at occlusion sensitivity and you're right, it's currently too slow. I'll have a look at adding a stride to it tomorrow.

[wyli]

Hi @Nic-Ma this is ready, could you review? (the pip command will need an update later)

[rijobro]

Just added occlusion sensitivity (which will also require a PR to MONAI for the striding). Results look good:

[wyli]

thanks, I'll double check the colour encoding, maybe there's an issue with GradCAM.
for the non-lesions occlusion, is it computed with respect to the background score or lesion score?

[rijobro]

I was just thinking that the non-lesions are tricky since, when a lesion is present, the visualisations should hopefully point towards those lesions. In the absence of lesions, I guess it will check the whole image, which is why the images in the middle column are largely uniform.

For the colours, I think the reason is that for the occlusion sensitivity I did baseline - occlusion_sensitivity, such that higher values signify that the particular pixel was more important in the decision process. Intuitively this seemed to make more sense to me.

[rijobro]

for the non-lesions occlusion, is it computed with respect to the background score or lesion score?

The scores are calculated relative to the true label of that image.

As a sanity check, I checked that the network predicted the correct class, which I printed in the title of the first row. In these cases, the network was correct for all of them. It would be interesting to find an example in which the network was wrong.

[Nic-Ma]

Thanks for the nice notebook.
@ericspod and I can help review it when it's ready.

Thanks.

[rijobro]

This is where I'm up to for the confusion matrix. Any idea why the GradCam shows little variation in some examples?

[wyli]

in my notebook I used cam(x=image[None], class_idx=1) which is always with respect to the lesion class, perhaps need to change it to cam(x=image[None], class_idx=true_label) to be consistent with occ. sens.?

[rijobro]

Much better, thanks. We use the_slice=72, but there's no guarantee that the lesion would lie in that slice. Do we have a way of knowing where the lesion was?

[wyli]

the patches' original masks are also included in the patch dataset as labels_*, but they are not currently used in the demo. the TP column doesn't look consistent... but the rest looks great

[Nic-Ma]

Hi @wyli ,

May I know the status of this PR? Is it OK for review and release in v0.4?

Thanks.

[wyli]

Hi @wyli ,

May I know the status of this PR? Is it OK for review and release in v0.4?

Thanks.

I think @rijobro is still finalising this. @rijobro shall we merge this first and have another PR to update it?

[rijobro]

@wyli All my suggested changes are in a PR to your repo (here). I think I'm dong making changes, so up to you if you want to merge the two PRs together now or at some point in the future.

[wyli]

thanks @rijobro ! (for some reason I don't get any notification of that PR)

[rijobro]

@wyli i notice you've switched to class_idx=None. What does none do in this case? Should it not be predicted_label to match the occlusion sensitivity?

[wyli]

@wyli i notice you've switched to class_idx=None. What does none do in this case? Should it not be predicted_label to match the occlusion sensitivity?

None means using the argmax over the predicted logits, I think it's the same as your version?
https://github.com/Project-MONAI/MONAI/blob/7e23dbd087853cb553feafe3107a91ec2e5d74ad/monai/visualize/class_activation_maps.py#L100

btw I still feel the CAM result is not as good as the occ., want to tune it but I don't have time today...

[rijobro]

None means using the argmax over the predicted logits, I think it's the same as your version?

Sounds good.

btw I still feel the CAM result is not as good as the occ., want to tune it but I don't have time today...

I agree. The two sets of images should convey similar information, and I'm not convinced that they do at the moment. Also we should have a chat about intensities. Since 0.4.0 has already been released, we're in less of a rush for this, so let's have a chat via Slack whenever you're free?

[wyli]

previously I could get this

but now I could only get this:

I feel the previous results are nicer. but I couldn't replicate them (note that the input images+random aug. in both cases I've been able to reproduce the original images but note the gradcam results) do you have any clue @rijobro ?

[wyli]

the difference comes from model.eval(), gradcam looks more accurate without model.eval()

[rijobro]

@wyli wow, that's slightly worrying isn't it? Partially related to this PR, which would make eval mode more common.

[rijobro]

To me it looks like there's a thresholding taking place in the lower set of images, and then if the colourmap is from min to max, that would explain the smoother gradients in the top set of images. Perhaps something like result[result<threshold]=0 would get the same result (where threshold= e.g., 0).

Perhaps you should disable the postprocessing (which by default normalises betwen 0 and 1), so that you get a feel for the range of the results? Could plot with colourbars.

[wyli]

sure @rijobro I'll try those suggestions. probably it's the batchnorm layers causing the difference for this particular demo, but eval_mode is useful in general

[wyli]

this is ready @rijobro @Nic-Ma, for the CAM I used the train() mode so that the batch norm doesn't break the visuals, occ. uses the eval() mode

[rijobro]

Ok, looks good to me.

This PR: Unify visualisation Project-MONAI/MONAI#1381 will use train for gradcam and eval for occlusion sensitivity, so we will be able to remove some of the boilerplate here. But I think we can merge and then I'll tidy up once the MONAI PR is merged.

[wyli]

Ok, looks good to me.

This PR: Unify visualisation Project-MONAI/MONAI#1381 will use train for gradcam and eval for occlusion sensitivity, so we will be able to remove some of the boilerplate here. But I think we can merge and then I'll tidy up once the MONAI PR is merged.

sure, could have another iteration later, could you approve this if it looks ok