Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Removing the re-size for validation data, which breaking the validation accuracy of CIFAR training #12362

Merged
merged 1 commit into from
Sep 12, 2018

Conversation

juliusshufan
Copy link
Contributor

Description

The dataloader for example/image-classification resizes the imagesize of validation dataset to 256 by default, and which breaks the validation accuracy of CIFAR10, this PR trying to fix it.
Below firgure is the validation accuracy trends without this fix (model is ResNet-50).
image

Checklist

Changes

example/image-classification/common/data.py

Comments

  • If this change is a backward incompatible change, why must this change be made.
  • Interesting edge cases to note here

@juliusshufan juliusshufan requested a review from szha as a code owner August 26, 2018 11:41
@juliusshufan
Copy link
Contributor Author

@szha may I have your comments, I use this example for the regression tracking of mxnet with mkldnn backend, I'll be helpful if this issue can be addressed.
Thanks.

@szha szha requested review from zhreshold and removed request for szha September 6, 2018 03:19
@zhreshold
Copy link
Member

Does it affect the imagenet validation perf?

@juliusshufan
Copy link
Contributor Author

juliusshufan commented Sep 10, 2018

@zhreshold Thanks for the feedback, I use the pre-trained ResNet-50 model and validation-set of ImageNet-1k, and the accuracy with/w.o my changes are same: see test result:

Without my changes:
INFO:root:Finished with 126.286497 images per second
INFO:root:('accuracy', 0.753156969309463)
INFO:root:('top_k_accuracy_5', 0.9257512787723785)

With my changes:
INFO:root:Finished with 126.029153 images per second
INFO:root:('accuracy', 0.753156969309463)
INFO:root:('top_k_accuracy_5', 0.9257512787723785)

The imagenet validation perf and accuracy are same.

But without my changes, the validation accuracy trends of CIFAR10+ResNet50/VGG16 are as below, obviously it is not expected.

image
image

@TaoLv
Copy link
Member

TaoLv commented Sep 10, 2018

Seems this regression also goes into 1.3.0rc0 realese.

@juliusshufan
Copy link
Contributor Author

@zhreshold sorry for pushing, may I have you comments or approval?

Thanks.

@zhreshold
Copy link
Member

@juliusshufan Thanks for the result, it's fair enough to go through. Again, thanks for the contribution!

@zhreshold zhreshold merged commit ceabcaa into apache:master Sep 12, 2018
zhreshold added a commit that referenced this pull request Sep 12, 2018
…validation accuracy of CIFAR training (#12362)"

This reverts commit ceabcaa.
szha pushed a commit that referenced this pull request Sep 12, 2018
* Revert "Removing the re-size for validation data, which breaking the validation accuracy of CIFAR training (#12362)"

This reverts commit ceabcaa.

* Revert "[MXNET-580] Add SN-GAN example (#12419)"

This reverts commit 46a5cee.

* Revert "Remove regression checks for website links (#12507)"

This reverts commit 619bc3e.

* Revert "Revert "Fix flaky test: test_mkldnn.test_activation #12377 (#12418)" (#12516)"

This reverts commit 7ea0533.

* Revert "further bump up tolerance for sparse dot (#12527)"

This reverts commit 90599e1.

* Revert "Fix broken URLs (#12508)"

This reverts commit 3d83c89.

* Revert "Temporarily disable flaky tests (#12520)"

This reverts commit 35ca13c.

* Revert "Add support for more req patterns for bilinear sampler backward (#12386)"

This reverts commit 4ee866f.

* Revert "Change the way NDArrayIter handle the last batch (#12285)"

This reverts commit 597a637.
anirudh2290 pushed a commit to anirudh2290/mxnet that referenced this pull request Sep 19, 2018
anirudh2290 pushed a commit to anirudh2290/mxnet that referenced this pull request Sep 19, 2018
* Revert "Removing the re-size for validation data, which breaking the validation accuracy of CIFAR training (apache#12362)"

This reverts commit ceabcaa.

* Revert "[MXNET-580] Add SN-GAN example (apache#12419)"

This reverts commit 46a5cee.

* Revert "Remove regression checks for website links (apache#12507)"

This reverts commit 619bc3e.

* Revert "Revert "Fix flaky test: test_mkldnn.test_activation apache#12377 (apache#12418)" (apache#12516)"

This reverts commit 7ea0533.

* Revert "further bump up tolerance for sparse dot (apache#12527)"

This reverts commit 90599e1.

* Revert "Fix broken URLs (apache#12508)"

This reverts commit 3d83c89.

* Revert "Temporarily disable flaky tests (apache#12520)"

This reverts commit 35ca13c.

* Revert "Add support for more req patterns for bilinear sampler backward (apache#12386)"

This reverts commit 4ee866f.

* Revert "Change the way NDArrayIter handle the last batch (apache#12285)"

This reverts commit 597a637.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants