Semisup example uses 1000 instead of 100 samples

[maschulz]

The semisupervised MNIST example uses the default value of 1000 labeled samples instead of 100 labeled samples (as stated in the README). The given hyperparameters also only work for the 1000 sample case.

Could you give details on hyperparameters for the 100 sample setting?

[takerum]

I ran the code with the default value in the README just now, and I confirmed that it achieves ~2.0 % test error rate with 100 labeled examples, which is the almost same as the published results in https://arxiv.org/abs/1507.00677.

If you mean the results published in the recent paper on arXiv https://arxiv.org/abs/1704.03976, we use different network architecture (784-1200-600-300-150-10), and also add gaussian noise( mean=0, stddev=0.5) on each hidden layer following https://arxiv.org/abs/1606.03498. The optimal epsilon on this setting is epsilon=5.0. Please see Appendix C in https://arxiv.org/abs/1704.03976.
The reprducing code for the results on CIFAR10 and SVHN are available at https://github.com/takerum/vat_tf.

I thought that many people are not interested in the results on MNIST and I did not upload the code for MNIST at https://github.com/takerum/vat_tf, but it is OK to upload the code for MNIST if you cannot reproduce by yourself.

[maschulz]

Thank you so much for your reply! And thanks for providing code examples in different frameworks! It really helps understanding your method.

As I understand the code, this happens: In the README you don't specify the number of labeled samples. The script uses the default value of 1000 as defined in train_mnist_semisup.py. If I add --num_labeled_samples=100 I get much different and worse results. Could you take a look?

[takerum]

Ah, that's right, I will take a look at the code again and I will let you know the update here.
Thanks for telling me that!

[takerum]

Hi, I ran with 100 labeled examples and I got 2.08% test error rate.
How do you run the code?
I use theano v0.9 on GPU.

[maschulz]

I run this:
python2.7 train_mnist_semisup.py --cost_type=VAT_finite_diff --epsilon=0.3 --layer_sizes=784-1200-1200-10 --num_labeled_samples=100

Here is what I get. What does your test error look like after 20 epochs?

[Epoch] -1
nll_train : 2.31259438679 error_train : 93 nll_test : 2.30697407707 error_test : 890
training...
[Epoch] 0
nll_train : 0.0443453873187 error_train : 0 nll_test : 0.597440935289 error_test : 194
[Epoch] 1
nll_train : 0.019324450636 error_train : 0 nll_test : 0.59836551081 error_test : 200
[Epoch] 2
nll_train : 0.0112554453294 error_train : 0 nll_test : 0.571296067289 error_test : 191
[Epoch] 3
nll_train : 0.00796734961175 error_train : 0 nll_test : 0.700967522354 error_test : 229
[Epoch] 4
nll_train : 0.0049996422565 error_train : 0 nll_test : 0.894565489959 error_test : 309
[Epoch] 5
nll_train : 0.0038806994592 error_train : 0 nll_test : 1.20374955258 error_test : 347
[Epoch] 6
nll_train : 0.00305372782064 error_train : 0 nll_test : 1.43310522157 error_test : 354
[Epoch] 7
nll_train : 0.00194572394747 error_train : 0 nll_test : 1.20583039377 error_test : 339
[Epoch] 8
nll_train : 0.00201217254036 error_train : 0 nll_test : 1.36642608576 error_test : 354
[Epoch] 9
nll_train : 0.00169639320708 error_train : 0 nll_test : 1.75487193393 error_test : 369
[Epoch] 10
nll_train : 0.0012498898096 error_train : 0 nll_test : 1.19984606893 error_test : 345
[Epoch] 11
nll_train : 0.00113286427302 error_train : 0 nll_test : 1.14819213092 error_test : 342
[Epoch] 12
nll_train : 0.0010653578198 error_train : 0 nll_test : 1.11596483622 error_test : 331
[Epoch] 13
nll_train : 0.000881871662422 error_train : 0 nll_test : 1.56633726305 error_test : 364
[Epoch] 14
nll_train : 0.000911863387448 error_train : 0 nll_test : 1.40311905894 error_test : 349
[Epoch] 15
nll_train : 0.000665378401212 error_train : 0 nll_test : 2.19365165642 error_test : 387
[Epoch] 16
nll_train : 0.000540239725904 error_train : 0 nll_test : 2.131644029 error_test : 388
[Epoch] 17
nll_train : 0.00057682695676 error_train : 0 nll_test : 1.70332553366 error_test : 370
[Epoch] 18
nll_train : 0.000595259032317 error_train : 0 nll_test : 1.82448186036 error_test : 407
[Epoch] 19
nll_train : 0.000476910717407 error_train : 0 nll_test : 2.22030571832 error_test : 395
[Epoch] 20
nll_train : 0.00046913241714 error_train : 0 nll_test : 1.64200101476 error_test : 373

(also on theano 0.9.0)

[maschulz]

I am reasonably confident that the epsilon is wrong.
If I increase epsilon to 4 I get better results...

[takerum]

I am sorry for the late reply.
Do you happen to run the code on CPU?
I found that the results on CPU and GPU are different, (I had never run the code on CPU before).
Actually, as you said, the the default epsilon does not work on CPU mode, while it works well on GPU mode.
However, I have not been able to find out why these happen...

[maschulz]

I did indeed run the code on CPU. Maybe there is some bug or strange behaviour in Theano?

[takerum]

I am not sure... anyway, thanks for reporting the results on CPU!
Would you tell me the final accuracy with epsilon=4 on CPU?