transforms: add Random Erasing for image augmentation #909

zhunzhong07 · 2019-05-16T06:02:02Z

Random Erasing randomly selects a rectangle region in an image and erases its pixels with random values. It can reduce the risk of overfitting, and improves CNN baselines in image classification, object detection and person reidentification.

I found that this augmentation method has been widely used in image classification (CIFA-10, CIFAR-100) and person re-identification.

Also, it could achieve improvements on ImageNet: +0.7% in Prec@1 for ResNet-50, +0.33% in Prec@1 for ResNet-34.

Therefore, I think it would be valuable to users.

'Random Erasing Data Augmentation' by Zhong et.al. https://arxiv.org/pdf/1708.04896.pdf

A parallel work is "Improved Regularization of Convolutional Neural Networks with Cutout" proposed by DeVries. https://arxiv.org/pdf/1708.04552.pdf

Previous pull request and issues #335 #226 #420

codecov-io · 2019-05-16T06:49:31Z

Codecov Report

Merging #909 into master will increase coverage by 2.82%.
The diff coverage is 82.05%.

@@            Coverage Diff             @@
##           master     #909      +/-   ##
==========================================
+ Coverage   60.03%   62.85%   +2.82%     
==========================================
  Files          64       65       +1     
  Lines        5054     5140      +86     
  Branches      754      773      +19     
==========================================
+ Hits         3034     3231     +197     
+ Misses       1817     1683     -134     
- Partials      203      226      +23

Impacted Files	Coverage Δ
torchvision/transforms/functional.py	`70.95% <60%> (-0.17%)`	⬇️
torchvision/transforms/transforms.py	`82.12% <85.29%> (-0.42%)`	⬇️
torchvision/datasets/stl10.py	`26.59% <0%> (-3.26%)`	⬇️
torchvision/models/mobilenet.py	`89.7% <0%> (-2.61%)`	⬇️
torchvision/ops/roi_pool.py	`67.44% <0%> (-0.86%)`	⬇️
torchvision/ops/roi_align.py	`65.95% <0%> (-0.71%)`	⬇️
torchvision/models/detection/faster_rcnn.py	`74.39% <0%> (ø)`	⬆️
torchvision/extension.py	`38.09% <0%> (ø)`
torchvision/ops/boxes.py	`94.73% <0%> (+0.14%)`	⬆️
... and 9 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b3a1b6...429a55e. Read the comment docs.

zhunzhong07 · 2019-05-16T10:01:11Z

@alykhantejani Could you pay attention to this method and advice the member to add it into the transform. I think random erasing is a useful augmentation method that will often be used in vision tasks.

ekagra-ranjan · 2019-05-16T17:25:39Z

This transform can be handy during self-supervision training.

rwightman · 2019-05-19T05:29:07Z

I've used random erasing successfully in metric loss (triplet) training and larger image (224x224+ imagenet like) training with good results. I think it would be a worthwhile addition.

@zhunzhong07 .. I found that the per-pixel version was quite useful for the problems but you didn't include it in your github impl or this PR? In my experiments, normally distributed pixels, after image normalization worked well. Uniform dist caused convergence issues later in training. I perform the RE operation once the tensors are on the GPU as part of a GPU prefetching loader/collate/normalize.

Feel free to copy any ideas: https://github.com/rwightman/pytorch-image-models/blob/master/data/random_erasing.py

zhunzhong07 · 2019-05-19T07:13:49Z

@rwightman Thanks for your advice. In PR, I have included per-pixel mode by v = torch.rand(img.size()[0], h, w). I have checked your code and notice that the per-pixel value should be normally distributed. I will fix this bug. Thank you!

One request. I don't have enough GPUs to train a model on imagenet right now. So, if you already have the results, could you also provide some results of training w/ or w/o random erasing on imagenet? Thank you!

rwightman · 2019-05-19T18:29:04Z

@zhunzhong07 I'll run some imagenet trainings to support this. I don't think I have two historical runs with all the hyper-params and results recorded that didn't have some sort of change in library versions, other hyper params, machines etc... I'll let you know how it goes

ekagra-ranjan

The following proposed changes will be appreciable.

torchvision/transforms/functional.py

ekagra-ranjan · 2019-05-21T20:00:55Z

test/test_transforms.py

@@ -1317,6 +1317,23 @@ def test_random_grayscale(self):
        # Checking if RandomGrayscale can be printed as string
        trans3.__repr__()

+    def test_random_erasing(self):


Can you make the test a bit more stronger by checking if the region around the erased patch is equal to the original image?

Addressed by #1060.

surgan12 · 2019-05-18T12:03:47Z

torchvision/transforms/transforms.py

+    def __call__(self, img):
+        """
+        Args:
+            img (Tensor): Image to be erased.


IMO , having input and output as PIL Image should be good and consistent with the other transforms
e.g In order to apply RandomOrder with the list of other transforms.

If you don't do this operation after Normalization (which is a Tensor based transform), you have to duplicate argument passing and pass your dataset stats to both Normalize and the RandomErasing transform. Post norm, you can assume a mean of 0 and consistent std dev. Also, if you do it before norm, mixed up with other transforms, it's much easier to skew the statistics of your data and cause divergence between train and validation.

In my experience, using it on a few projects now, it's generally cleaner, less fussy, and more efficient (integrated with moving data to the GPU) if done after normalization as tensor ops.

surgan12 · 2019-05-18T12:06:43Z

torchvision/transforms/transforms.py

+            w = int(round(math.sqrt(target_area / aspect_ratio)))
+
+            if w < img.size()[2] and h < img.size()[1]:
+                x = random.randint(0, img.size()[1] - h)


@zhunzhong07 , just a minor optimisation, can't we store img.size()[1] kinda values instead of computing again and again.

Addressed by #1087.

rwightman · 2019-05-29T18:59:51Z

FWIW, I finished several training sessions with different random-erasing settings. I ran no RE, constant (0) RE, and normally distributed (0 mean, 1 std) per-pixel RE. I did not do a random color (solid) run. I'm still running some other tests in this series to validate some other impl and hyper-params for personal interest....

All training with cosine LR decay, 5 epoch warmup, 120 epochs until a 10 epoch constant cooldown, label smoothing @ 0.1
Standard imagenet preprocessing & augmentation otherwise
Thanks to cosine + smoothing, all 3 models I trained are pretty darn good results for a resnet-34 (see torchvision pretrained), including the one without random-erasing
There is an improvement with each level of RE applied
The improvement is still evident when validating against the 'ImagenetV2 matched-frequency' validation set, where both RE models show an advantage

ImageNet 1K validation

training	results
torchvision	Prec@1 73.312 (26.688) Prec@5 91.426 (8.574)
me w/ no RE	Prec@1 74.772 (25.228) Prec@5 91.988 (8.012)
me w/ const 0 RE	Prec@1 74.838 (25.162) Prec@5 92.212 (7.788)
me w/ per-pixel, normal RE	Prec@1 75.110 (24.890) Prec@5 92.284 (7.716)

ImagenetV2-matched-frequency validation (https://github.com/modestyachts/ImageNetV2)

training	results
torchvision	Prec@1 61.190 (38.810) Prec@5 82.710 (17.290)
me w/ no RE	Prec@1 62.300 (37.700) Prec@5 83.870 (16.130)
me w/ const 0 RE	Prec@1 62.790 (37.210) Prec@5 84.060 (15.940)
me w/ per-pixel, normal RE	Prec@1 62.870 (37.130) Prec@5 84.140 (15.860)

zhunzhong07 · 2019-05-30T02:45:37Z

@rwightman Thanks for your experimental results. It is great to see that random erasing could improve the performance of ImageNet.

Did you run these results on your impl https://github.com/rwightman/pytorch-image-models? If so, could you also provide the running shell (i.e., parameters of distributed_train.sh), so that we can accurately reproduce results?

Thank you for your time and efforts on implementing these results.

rwightman · 2019-05-31T19:02:02Z

Yeah, using the train script in image-models. I was only running single GPU for these runs and did them in parallel. I had a local mod experimenting with the warmup and changing it's overlap behaviour with the main schedule but differences is minor, I extended the epochs here by 5 to compensate. These should reproduce results closely enough:

No RE:
python train.py /imagenet/ --model resnet34 -b 256 --epochs 125 --warmup-epochs 5 --sched cosine --lr 0.1 --weight-decay 1e-4 --reprob 0.

RE constant:
python train.py /imagenet/ --model resnet34 -b 256 --epochs 125 --warmup-epochs 5 --sched cosine --lr 0.1 --weight-decay 1e-4 --reprob 0.4

RE per-pixel normal:
python train.py /imagenet/ --model resnet34 -b 256 --epochs 125 --warmup-epochs 5 --sched cosine --lr 0.1 --weight-decay 1e-4 --reprob 0.4 --remode pixel

fmassa · 2019-06-04T15:24:42Z

@rwightman thanks a lot for the feedback wrt the usefulness of RandomErasing.

I'll have a closer look at the implementation today

zhunzhong07 · 2019-06-06T00:16:21Z

@rwightman Thanks! With your provided scripts, I have obtained similar results for ResNet34.

@fmassa Thank you for your attention. I also run RandomErasing for ResNet50 and ResNet101, and achieve an improvement (+0.7 in Prec@1 for ResNet50 and +0.55 in Prec@1 for ResNet101).

Results on ImageNet 1K validation

training	results
torchvision ResNet50	Prec@1 76.15 (23.85) Prec@5 92.87 (7.13)
me ResNet50 w/ no RE	Prec@1 76.33 (23.67) Prec@5 92.96 (7.04)
me ResNet50 w/ per-pixel, normal RE	Prec@1 77.08 (22.92) Prec@5 93.27 (6.73)
torchvision ResNet101	Prec@1 77.37 (22.63) Prec@5 93.56 (6.44)
me ResNet101 w/ no RE	Prec@1 79.02 (20.98) Prec@5 94.27 (5.73)
me ResNet101 w/ per-pixel, normal RE	Prec@1 79.57 (20.43) Prec@5 94.7 (5.3)

fmassa

Thanks fir the PR!

I have a few comments, let me know what you think.

Also, this is the transform that you used to obtain better results, with value='random', is that right?

Can you also add an entry to the documentation in https://github.com/pytorch/vision/blob/master/docs/source/transforms.rst

torchvision/transforms/functional.py

torchvision/transforms/transforms.py

zhunzhong07 · 2019-06-08T05:07:20Z

@fmassa Thank you for your improved comments. I have modified the PR and add an entry to the documentation, according to your suggestions.

Yes, using the mode of value='random' would achieve better results. I would like to know your suggestion: only using 'random' mode, or, including multi modes (random and constant)?

zhunzhong07 · 2019-06-14T09:39:37Z

Hi @fmassa. I've modified the PR according to your comments. I also add the results of ResNet101 above, consistent improvement is obtained by RandomErasing.

fmassa

Thanks a lot!

Zhaoyi-Yan · 2019-07-03T07:34:22Z

@zhunzhong07 Hi, have you tried it for detection?

zhunzhong07 · 2019-07-03T10:46:08Z

@Zhaoyi-Yan Yes. Random Erasing can improve the results of Fast RCNN on VOC17. Please refer to our paper.

zhunzhong07 added 2 commits May 16, 2019 15:59

add erase function

294c99a

add Random Erasing

6d34f5e

zhunzhong07 changed the title ~~add erase function~~ transforms: add Random Erasing for image augmentation May 16, 2019

zhunzhong07 marked this pull request as ready for review May 16, 2019 06:23

zhunzhong07 mentioned this pull request May 16, 2019

transforms: add Random Erasing for image augmentation #335

Closed

zhunzhong07 added 2 commits May 16, 2019 16:32

Update transforms.py

caecccd

Update transforms.py

219efbb

zhunzhong07 added 6 commits May 16, 2019 18:17

add test for random erasing

511282a

Update test_transforms.py

a5251dc

fix flake8

bce27fb

Update test_transforms.py

9b3773a

Update functional.py

53e12d4

Update test_transforms.py

bf98b39

zhunzhong07 added 2 commits May 19, 2019 17:48

fix bug for per-pixel erasing

6b59f26

Update transforms.py

51d0cc9

ekagra-ranjan suggested changes May 21, 2019

View reviewed changes

surgan12 reviewed May 23, 2019

View reviewed changes

specific for coordinate (x, y)

b92f21e

fmassa requested changes Jun 7, 2019

View reviewed changes

zhunzhong07 added 3 commits June 8, 2019 14:44

add raise TypeError for img

987cc3c

Update transforms.py

d241524

Update transforms.rst

429a55e

rwightman mentioned this pull request Jun 17, 2019

default training hyper-parameters huggingface/pytorch-image-models#11

Closed

fmassa approved these changes Jun 24, 2019

View reviewed changes

fmassa merged commit 3254560 into pytorch:master Jun 24, 2019

fmassa mentioned this pull request Feb 3, 2021

bugfix aspect ratio sampling in transforms.RandomErasing #3344

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

transforms: add Random Erasing for image augmentation #909

transforms: add Random Erasing for image augmentation #909

zhunzhong07 commented May 16, 2019 •

edited

Loading

codecov-io commented May 16, 2019 •

edited

Loading

zhunzhong07 commented May 16, 2019

ekagra-ranjan commented May 16, 2019

rwightman commented May 19, 2019

zhunzhong07 commented May 19, 2019 •

edited

Loading

rwightman commented May 19, 2019

ekagra-ranjan left a comment

ekagra-ranjan May 21, 2019

ekagra-ranjan Jul 4, 2019

surgan12 May 18, 2019

rwightman May 25, 2019

surgan12 May 18, 2019

ekagra-ranjan Jul 4, 2019

rwightman commented May 29, 2019

zhunzhong07 commented May 30, 2019

rwightman commented May 31, 2019

fmassa commented Jun 4, 2019

zhunzhong07 commented Jun 6, 2019 •

edited

Loading

fmassa left a comment

zhunzhong07 commented Jun 8, 2019 •

edited

Loading

zhunzhong07 commented Jun 14, 2019

fmassa left a comment

Zhaoyi-Yan commented Jul 3, 2019

zhunzhong07 commented Jul 3, 2019

transforms: add Random Erasing for image augmentation #909

transforms: add Random Erasing for image augmentation #909

Conversation

zhunzhong07 commented May 16, 2019 • edited Loading

codecov-io commented May 16, 2019 • edited Loading

Codecov Report

zhunzhong07 commented May 16, 2019

ekagra-ranjan commented May 16, 2019

rwightman commented May 19, 2019

zhunzhong07 commented May 19, 2019 • edited Loading

rwightman commented May 19, 2019

ekagra-ranjan left a comment

Choose a reason for hiding this comment

ekagra-ranjan May 21, 2019

Choose a reason for hiding this comment

ekagra-ranjan Jul 4, 2019

Choose a reason for hiding this comment

surgan12 May 18, 2019

Choose a reason for hiding this comment

rwightman May 25, 2019

Choose a reason for hiding this comment

surgan12 May 18, 2019

Choose a reason for hiding this comment

ekagra-ranjan Jul 4, 2019

Choose a reason for hiding this comment

rwightman commented May 29, 2019

zhunzhong07 commented May 30, 2019

rwightman commented May 31, 2019

fmassa commented Jun 4, 2019

zhunzhong07 commented Jun 6, 2019 • edited Loading

fmassa left a comment

Choose a reason for hiding this comment

zhunzhong07 commented Jun 8, 2019 • edited Loading

zhunzhong07 commented Jun 14, 2019

fmassa left a comment

Choose a reason for hiding this comment

Zhaoyi-Yan commented Jul 3, 2019

zhunzhong07 commented Jul 3, 2019

zhunzhong07 commented May 16, 2019 •

edited

Loading

codecov-io commented May 16, 2019 •

edited

Loading

zhunzhong07 commented May 19, 2019 •

edited

Loading

zhunzhong07 commented Jun 6, 2019 •

edited

Loading

zhunzhong07 commented Jun 8, 2019 •

edited

Loading