The test results of the PSGTR model are incorrect

[zishengzheqiang]

I use the epoch_60.pth of PSGTR model to run test scripts

I change the detection_method='bbox'

This is the result in the log file you gave

There is a big difference between them，What is the problem behind this phenomenon?

[Jingkang50]

#51
Thanks for your question. You might refer to this discussion.

[Jingkang50]

In short, you might need to change

OpenPSG/openpsg/models/frameworks/psgtr.py

Line 16 in a0e5c8f

def triplet2Result(triplets, use_mask, eval_pan_rels=True):

to eval_pan_rels=False.

[zishengzheqiang]

Thanks for your detailed explanation! I set eval_pan_rels=False and R@20 is higher than before.
But the competition needs to submit a panseg png file, we should set eval_pan_rels=True, so the R@20 is low, Is it possible to conclude that the one-stage algorithm is worse than the two-stage algorithm?

In short, you might need to change

OpenPSG/openpsg/models/frameworks/psgtr.py

Line 16 in a0e5c8f

def triplet2Result(triplets, use_mask, eval_pan_rels=True):

to eval_pan_rels=False.

[Jingkang50]

I would not conclude in this way, as mean recall is the metric the community cares more. Also, we turn on eval_pan_rel for easy submission of the competition, but the model who has good performance with eval_pan_rel off can also greatly support the downstream task. For researchers who follows PSG but not only for competition, we also recommend them to explore a better solution when eval_pan_rel off.

[zishengzheqiang]

I get it, thank you so much!