Regarding negative cases in the luna16 dataset

[apourchot]

Hi,

I've been going through the dataset split files from the luna16 tutorial and noticed there aren't any negative cases, i.e. scans without bounding boxes. I get why they might be left out in training, but they seem to be missing in the validation datasets too. Could you help me understand a couple of things?

1. When did this approach start? It's surprising that the original challenge creators would skip negative cases, but maybe I'm missing something. I looked over the nnDetection scripts for making the splits and didn't find anything about filtering out scans based on the number of bounding boxes. But, again, I might have missed something.

2. Can you confirm if the metrics on the tutorial page include or exclude negative cases? If they're excluded, it seems like the metrics might be off, most likely over-estimated?

Thanks in advance for any info you can share!

[KumoLiu]

Hi @Can-Zhao, could you please help share some comments on this discussion?
Thanks in advance!

[Can-Zhao]

Hi @apourchot

Thank you for reaching out.
Yes, the original challenge creators skipped negative cases. (Update: The original challenge creators did not skip images without nodules.)

I do not think you will need to worry about negative samples. If I understand correctly, you were worrying that there is no negative sample, so the network tends to predict positive samples (boxes) everywhere. In detection, positive sample means a box. But in an image, most of the voxels are not associated to a box, and they are all negative samples. If during inference, a model predicts boxes everywhere on the image, e.g., 10000 boxes on an image, it will give a high false positive rate, and it is reflected in metrics.

Hope it helps. Please feel free to let me know if needed.

[Can-Zhao]

I'm confused. You seem to have two concerns. One is suspecting the images without nodules deliberately removed during evaluation. The other one is thinking that lacking of healthy subjects will cause metric to inflate. To answer your first concern, we used the exact evaluation script from the challenge. Running this evaluation script did not use our data split, and it used the annotations.csv that the organizers provide

tutorials/detection/run_luna16_offical_eval.sh

Line 28 in 3fa155b

python ./evaluation_luna16/noduleCADEvaluationLUNA16.py \

. To answer your second concern, I think your definition of positive/negative samples is not what the metrics trying to measure. For an image with 3 nodules, in your definition, it will be one positive sample; but the metrics will consider it as three positive samples and numerous of negative samples.

[Can-Zhao]

I felt I might have missed something here. I do not understand why using the original evaluation script and original annotations.csv, annotations_excluded.csv, and seriesuids.csv can be wrong. Regarding the data split json files, @dongyang0122 I was wondering if you remember how the data split json files were created from data split pkl file?

[Can-Zhao]

@apourchot You are right about original challenge creators did not skip negative cases. After I look into annotations_excluded.csv file, I realized this file did not exclude the subjects without nodules. It excludes the suspicious nodules. I'm sorry, I misunderstand how the challenge data were created. I did not find evidence that challenge creators skipped negative cases. For how data split files were created, let me find some help from nnDetection authors.
Hi Michael, @mibaumgartner, I was wondering if you remember how the data split pkl file was generated? Thank you so much!

[mibaumgartner]

Hi, the data of LUNA16 is located in 10 directories which indicate the 10 folds. The splits file is simply a reflection of that folder structure where 9/10 were used for training and 1/10 was set as the validation fold (as in https://github.com/wentaozhu/DeepLung). No additional filtering of cases was applied from my/our side (each split [train+val] contains all of the images of LUNA16 i.e. 888 images).

LUNA16 is a subset of LIDC, I would recommend to check the two papers for more specific details. From a quick glance, the LIDC paper is not quite clear about a lower boundary but mentions some soft upper boundaries.

Unfortunately, it is also not quite uncommon that datasets only have images with pathologies inside them. This certainly biases the results to some extend and does not allow for patient level evaluation. Nevertheless, for a pure object detection task, these datasets are still useful since we have an unknown number of objects which need to be localised quite precisely to be counted as correct.

[Can-Zhao]

Thank you so much Michael @mibaumgartner for helping check it! My apologies for involving you. Your insight has been invaluable.

[dongyang0122]

@apourchot Apologies for any misunderstanding caused. The initial dataset splits were generated using the annotation.csv file (from LUNA16) in conjunction with the data splits employed by nnDetection. Due to the fact that only pathological cases are enumerated in the annotation.csv file, healthy subjects were inadvertently omitted from our data splits. We will promptly revise the data list and update the relevant numbers in this tutorial to rectify this oversight. Thank you for bringing this to our attention!

[Can-Zhao]

Upon reviewing the feedback, we have initiated a thorough examination of our data processing procedures and found that the images without nodules were omitted in data split json files unintentionally. We are in consultation with my colleagues to gain a comprehensive understanding of every step involved in our data handling.

It is important to note that any errors in our data were not intentional. We are dedicated to rectifying any inaccuracies and implementing measures to prevent similar issues in the future. We will generate correct data split json files and recompute the metrics soon.

We value the feedback from our community and are grateful for the opportunity to improve. We welcome any further suggestions or insights that could assist us in this process.

[apourchot]

Thank you very much for acknowledging the problem. Looking forward to the updated splits and metrics 🙂

[Thibescobar]

Hello @apourchot,

_________ EDIT ________
I was wrong, actually, it is the opposite...
See details here: MIC-DKFZ/nnDetection#272 (comment)
______ END EDIT ______

(Maybe related to my recent post here on the GitHub of nnDetection: MIC-DKFZ/nnDetection#272)

I observe something like you I think.

When I include "healthy" patients, the FROC score is decreased. I was explaining this like you, but just by hand, based on the fact that with no object to detect, only FP are possible, thus pushing scores toward lower values for fixed FP/scan.

But a colleague challenged this explanation saying that for example if we have a score of Se 80% @ 2FP/scan in average, if we add negative healthy patients, sensitivity will remain the same, and we could expect 2FP/scan in average still on the new patients.

Sounds also good to me... So I am a bit lost since I definitly observe this phenomenum.

Do you have elements of answer or explanation of this phenomenum please.

Thank you in advance.

Best,
Thibault