question about test

[jinxxo-j]

Hi, @Lu-Feng
Thanks you for an interesting research!

I have some question about cross image encoder.

from the question link below,
#7 (comment)

I think that cross image encoder could use prior knowledge(consecutiveness of queries) of test dataset

from supplemetary material (table 10),
there is an ablation study of cross-image encoder
the performance on pitts30k is written below,
No encoder : 90.6, 95.9, 97.2
Transformer encoder layer x 2(default) : 94.8, 97.4, 98.1

compare this performance of crica in the above link,

when test dataloader use shuffle=True,

2024-05-02 12:47:59   [0]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 92.2, R@5: 96.2, R@10: 97.2, R@100: 99.4
2024-05-02 12:49:38   [1]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 91.8, R@5: 96.1, R@10: 97.3, R@100: 99.4
2024-05-02 12:51:17   [2]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 92.0, R@5: 96.1, R@10: 97.2, R@100: 99.4
2024-05-02 12:52:57   [3]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 91.9, R@5: 96.1, R@10: 97.3, R@100: 99.4
2024-05-02 12:54:37   [4]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 92.0, R@5: 96.0, R@10: 97.3, R@100: 99.4
2024-05-02 12:56:28   [5]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 92.1, R@5: 96.0, R@10: 97.3, R@100: 99.3
2024-05-02 12:58:21   [6]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 92.1, R@5: 96.0, R@10: 97.3, R@100: 99.4
2024-05-02 13:00:23   [7]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 92.1, R@5: 96.2, R@10: 97.2, R@100: 99.4
2024-05-02 13:02:23   [8]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 91.9, R@5: 95.9, R@10: 97.2, R@100: 99.4
2024-05-02 13:04:21   [9]: Recalls on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: R@1: 92.0, R@5: 95.9, R@10: 97.2, R@100: 99.3
2024-05-02 13:04:21   Average recall on < BaseDataset, pitts30k - #database: 10000; #queries: 6816 >: 91.99090375586854

it seems cross image encoder seems to have less impact when test dataloader shuffle=True.
and i think that this issue should be solved .