Decoding large natural images

[willgdjones]

Hi Emilien! Just wanted to first say thanks for the repository. It's a very clean implementation, that I found very helpful to understand the main trick within CONCRETE.

I am using VAEs to create embeddings of natural histopathology images (over at https://github.com/willgdjones/HistoVAE), and I'm interested in the hyper-parameters you have used to create high-quality decodings. These are 128x128 pixel images, and so not unlike images that you might find in CIFAR100. I am about to try to implement vae-concrete and see if that helps, but I wanted to ask in case you had any other specific advice. The encodings that you generate as part of JointVAE are exactly what I'm looking for (so I might go ahead and try that too!).

Thanks,
Will

[EmilienDupont]

Hi Will! Thanks for reaching out, I'm glad you found the implementation helpful!

I only ran the model in this repo on MNIST (so 28x28 images) and the hyper parameters I used are the ones given as default parameters in the vae_concrete.py file. It is likely that a similar architecture with everything scaled up by a factor of 4 (e.g. number of layers etc) could work for 128x128 images but I have not had a chance to try it.

I just had a look at your repo and your project sounds very cool! As I don't know much about histopathology I am not sure what a useful embedding would be in this case. But if there is an important discrete or categorical factor in the histopathology images that you would like to embed, then I would definitely recommend trying JointVAE instead of a regular VAE concrete model. The main reason for this is that regular VAE models with a discrete and continuous latent distribution tend to ignore the discrete distribution and encode most of the information in the continuous factors (unlike JointVAE which forces information to be encoded in the discrete factors). This is why the discrete factors encoded by VAE concrete do not correspond exactly to the different categories of MNIST (as you can see here) whereas the ones by JointVAE do (as you can see here). But again, which model is best to use depends on what is desirable for the embeddings of this particular histopathology dataset.

Hope this helps and thanks again for your question!

[willgdjones]

Hi @EmilienDupont - thanks for the response, and interest! It seems that I didn't get a notification of your reply which is why it took so long for me to reply back to you.

I seem to have some general problems of training vanilla VAEs on natural images (>128x128px, 3 channel), rather than on datasets such as MNIST (28x28, 1 channel). Therefore I do think that it is likely to be some a problem with the implementation that I have. I will be testing my implementation on other datasets, just to make sure of this - especially since there doesn't seem to be any special hyper-parameter trick.

I'll keep you updated!

Best,
Will