Implement all AutoAugment transforms + Policies

[datumbox]

Fixes #3050

This PR implements the necessary transforms for AutoAugment:

• posterize
• solarize
• equalize
• sharpness
• autocontrast
• invert

It also implements the following policies:

• ImageNet AutoAugment Policy
• CIFAR10 AutoAugment Policy
• SVHN AutoAugment Policy

[codecov]

Codecov Report

Merging #3123 (8bf0f34) into master (4eab7a6) will increase coverage by 0.22%.
The diff coverage is 83.90%.

@@            Coverage Diff             @@
##           master    #3123      +/-   ##
==========================================
+ Coverage   72.79%   73.01%   +0.22%     
==========================================
  Files          98       99       +1     
  Lines        8964     9256     +292     
  Branches     1430     1493      +63     
==========================================
+ Hits         6525     6758     +233     
- Misses       1992     2025      +33     
- Partials      447      473      +26     

Impacted Files	Coverage Δ
torchvision/transforms/functional_pil.py	67.61% <62.50%> (-0.77%)	⬇️
torchvision/transforms/functional_tensor.py	72.28% <67.05%> (-1.22%)	⬇️
torchvision/transforms/functional.py	81.52% <92.30%> (+0.68%)	⬆️
torchvision/transforms/autoaugment.py	94.11% <94.11%> (ø)
torchvision/transforms/__init__.py	100.00% <100.00%> (ø)
torchvision/transforms/transforms.py	83.69% <100.00%> (+1.61%)	⬆️
torchvision/extension.py	54.09% <0.00%> (-13.12%)	⬇️
torchvision/ops/deform_conv.py	69.23% <0.00%> (-3.08%)	⬇️
torchvision/models/detection/anchor_utils.py	93.33% <0.00%> (-1.34%)	⬇️
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4eab7a6...8bf0f34. Read the comment docs.

[vfdev-5]

@datumbox great PR !

I didn't yet review everything and thought about spliting it into several ones:

• invert, posterize, etc ops (F and T)
• auto augment
  what do you think ?

I left some comments about the code.

[datumbox]

@vfdev-5 Thanks for the comments.

I'm writing here what we discussed on Slack for future reference. This PR is composed of sub-PRs for each operator and policy (see commit messages). Just note that follow up PRs make changes to past (clean ups, bug fixes, deduplications etc).

I addressed as many of your comments as I could. I agree with all of your points and that's the API I was going for originally but I had to adapt it because it was not compatible with JIT.

Let me know what you think for the rest. Thanks!

[vfdev-5]

Few more comments :)

[datumbox]

The failing test on onnx are unrelated.

[fmassa]

Thanks a lot for the PR @datumbox !

I've made a few comments, just so that I understand a bit better if the .clamp are needed everywhere.

Let me know what you think

[fmassa]

for information: one potential way of performing the operations in a batched fashion (while trading on memory) is to do something like

# img is a ...xCxHxW tensor
# temp is ...xCxHxWx256
temp =  img[..., None] == torch.arange(0, 256, dtype=torch.uint8, device=img.device)
hist_batched = temp.sum(dim=[-3, -2])

We avoid converting to float32, but we expand the image to have 256 more uint8 elements (so 64x more memory, probably not worth it here except if running on CUDA for batched inputs)

[datumbox]

Great idea. As this PR is getting lengthy, I raised a separate issue for this #3173.

[fmassa]

Thanks a lot!

[ain-soph]

Hi, thanks a lot for implementing AutoAugment. But I have some trouble using it and find no docs about this.
My previous transform on CIFAR10 is

transforms.Compose([
    transforms.RandomCrop(32, padding=4),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.49139968, 0.48215827, 0.44653124), (0.24703233, 0.24348505, 0.26158768)),
    Cutout(cutout_length=16),
])

ImageNet

transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)),
])

I wonder what my transform should be if I want to apply AutoAugment.

[NicolasHug]

@ain-soph does this help? https://pytorch.org/vision/master/auto_examples/plot_transforms.html#autoaugment (These are the latest docs, they will be released "officially" soon).

You should be able to use AutoAugment as any other transform.

[datumbox]

@ain-soph I think Nicolas reply has the best and most up-to-date references.

To echo what he said, you should be able to use it similar to any other transform. This is the example we provided on the release notes:

from torchvision import transforms

t = transforms.AutoAugment()
transformed = t(image)

transform=transforms.Compose([
    transforms.Resize(256),
    transforms.AutoAugment(),
    transforms.ToTensor()])

Note that our reference scripts also show you how you combine AutoAugment with RandomErasing which is a very typical setup because on the AutoAugment paper this is how it was used:

vision/references/classification/presets.py

Lines 7 to 20 in 1703e4c

	trans = [transforms.RandomResizedCrop(crop_size)]
	if hflip_prob > 0:
	trans.append(transforms.RandomHorizontalFlip(hflip_prob))
	if auto_augment_policy is not None:
	aa_policy = autoaugment.AutoAugmentPolicy(auto_augment_policy)
	trans.append(autoaugment.AutoAugment(policy=aa_policy))
	trans.extend([
	transforms.ToTensor(),
	transforms.Normalize(mean=mean, std=std),
	])
	if random_erase_prob > 0:
	trans.append(transforms.RandomErasing(p=random_erase_prob))
	self.transforms = transforms.Compose(trans)

@NicolasHug given the common combination of the two above transforms, I wonder if it would make sense to add 1 example of that in the gallery? The AutoAugment needs to go first before RandomErasing.

[NicolasHug]

Yes, for now the example only showcases individual transforms, but perhaps a new section like "common transformation pipelines" would make sense? If the only candidate is AutoAugment+RandomErasing, maybe we can just add this one too

[datumbox]

I'm not aware of any other candidate. Those two go together because that's how the paper introduces them, but I could be wrong.

Please note that RandomErasing is implemented only for Tensors not PIL and thus the gallery code might need to rescale and convert to PIL before showing the image to the user.

[ain-soph]

Thanks for your guys’ explanation! Based on CIFAR scenario, it seems the AutoAugment will be inserted before ToTensor. I’ll take a try. Hope ResNet18 acc could get a boost to 98% from 97%.