Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

About sampling strategies of test input patch #7

Open
dituu opened this issue Jul 24, 2023 · 2 comments
Open

About sampling strategies of test input patch #7

dituu opened this issue Jul 24, 2023 · 2 comments

Comments

@dituu
Copy link

dituu commented Jul 24, 2023

Hi, thanks for releasing the code. I have a question about the test setting on PU-GAN dataset.

In original PU-GAN dataset, the test input sparse patches are generated by random downsampling form the ground truth, while this work uses Poisson downsampling in prepare_pugan.py. The former generates a realistic and non-uniform distribution, while the Poisson downsampling produces a uniform distribution.

I directly test the released model on PU-GAN test set under random downsampling setting, and find the performance degenerates greatly (CD 0.245->0.495). It seems that the method is sensitive to the input distribution pattern.

So, do you have some ideas about this phenomenon? Many thanks!

@dituu dituu changed the title About input sampling About sampling strategies of test input patch Jul 24, 2023
@yunhe20
Copy link
Owner

yunhe20 commented Jul 29, 2023

  1. As already stated in the "4.1 Experiment Setup" section, we follow PU-GCN to prepare the test set of PU-GAN dataset.
  2. In genearal, the randomly sampled test set is more difficult than possion sampled test set, so it is normal that the metrics may drop (include other methods). Have you tested other methods on your test set? And how did they perform on your test set?

@dituu
Copy link
Author

dituu commented Jul 31, 2023

Thanks for your reply! I have tested the public PU-GAN and PU-GCN model on my prepared test set (random downsampling). The results are as follows,
CD || HD || P2F
PU-GAN 0.28 4.60 3.18
PU-GCN 0.30 4.31 2.78
It seems the CD errors are consistent with the results of possion sampling, while HD and P2F are degenerated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants