Description of Testing on the GAICD dataset.

[geraltFromRivia]

Hello!

Thank you very much for sharing the code and dataset with the community. I have a questions about how would one go about testing previously published methods on the GAICD. I was wondering if you could possibly describe how do you test the VFN / VEN models on the GAICD dataset. Do you feed take the precomputed crops and then feed them each into the VFN / VEN and then rank order the crop scores?

Thank you very much.

[HuiZeng]

Hello!

Yes, we tested previous methods exactly as your description.

[HuiZeng]

Please also note that those crops in the dataset with annotated score "-2" are discarded, cause they have irregular aspect ratios.

[geraltFromRivia]

Thank you very much for the prompt reply! Is there a chance you'd be able to post a sample evaluation script for VFN or VEN?

I did notice the crops in the dataset with annotated score "-2" are discarded in the dataset definition. I am inquiring about the method of testing, as when using the pre-trained VEN model provided by the authors, the scores computed by my modified testing script computed an SRCC < 0.2, which is much lower as compared to the one reported in the paper.

Thanks!

[HuiZeng]

I uploaded the testing code here:
https://drive.google.com/file/d/1L502GpdTxFW7INQPGD5lR71uRSk6NJBh/view?usp=sharing

The attached results are obtained from VFN. Sorry for not polishing this code.

You may mess up the order of different crops if you only get an SRCC < 0.2.

[HuiZeng]

The result file is formulated as: image ID, crop ID, predicted score.

[geraltFromRivia]

Thank you very much. I will have a look at the code.