Adding more ConvNeXt variants + Speed optimizations

[datumbox]

Adding convnext_small, convnext_base and convnext_large.

Small

torchrun --nproc_per_node=1 train.py --test-only --prototype --weights ConvNeXt_Small_Weights.IMAGENET1K_V1 --model convnext_small -b 1
Acc@1 83.616 Acc@5 96.650

Base

torchrun --nproc_per_node=1 train.py --test-only --prototype --weights ConvNeXt_Base_Weights.IMAGENET1K_V1 --model convnext_base -b 1
Acc@1 84.062 Acc@5 96.870

Large

torchrun --nproc_per_node=1 train.py --test-only --prototype --weights ConvNeXt_Large_Weights.IMAGENET1K_V1 --model convnext_large -b 1
Acc@1 84.414 Acc@5 96.976

Also switch from channels first + Conv1x1 to channels last + Linear layers as it gets a 20% speed boost.

[facebook-github-bot]

💊 CI failures summary and remediations

As of commit d708d06 (more details on the Dr. CI page):

---

• 2/2 failures introduced in this PR

---

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

CodeQL / build (1/1)

Step: "Build TorchVision" (full log | diagnosis details | 🔁 rerun)

2022-02-01T12:52:57.2791929Z ##[error]Process completed with exit code 1.

2022-02-01T12:52:57.1309743Z     self.finalize_options()
2022-02-01T12:52:57.1310807Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/develop.py", line 52, in finalize_options
2022-02-01T12:52:57.1311252Z     easy_install.finalize_options(self)
2022-02-01T12:52:57.1312152Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 276, in finalize_options
2022-02-01T12:52:57.1312603Z     self._fix_install_dir_for_user_site()
2022-02-01T12:52:57.1313539Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 382, in _fix_install_dir_for_user_site
2022-02-01T12:52:57.1313999Z     self.create_home_path()
2022-02-01T12:52:57.1315068Z   File "/home/runner/.local/lib/python3.8/site-packages/setuptools/command/easy_install.py", line 1338, in create_home_path
2022-02-01T12:52:57.1315590Z     if path.startswith(home) and not os.path.isdir(path):
2022-02-01T12:52:57.1316096Z AttributeError: 'int' object has no attribute 'startswith'
2022-02-01T12:52:57.2791929Z ##[error]Process completed with exit code 1.
2022-02-01T12:52:57.2848483Z Post job cleanup.
2022-02-01T12:52:57.4203399Z [command]/usr/bin/git version
2022-02-01T12:52:57.4259234Z git version 2.34.1
2022-02-01T12:52:57.4302184Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-02-01T12:52:57.4349471Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-02-01T12:52:57.4733281Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-02-01T12:52:57.4761732Z http.https://github.com/.extraheader
2022-02-01T12:52:57.4771586Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-02-01T12:52:57.4831437Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-02-01T12:52:57.5538642Z Cleaning up orphan processes

---

1 failure not recognized by patterns:

Job	Step	Action
cmake_macos_cpu	curl -o conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
sh conda.sh -b
source $HOME/miniconda3/bin/activate
conda install -yq conda-build cmake
packaging/build_cmake.sh
	🔁 rerun

---

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

[datumbox]

@s9xie @liuzhuang13 as discussed at #5197 (comment) here are some benchmarks on our side. This is the speed of TorchVision's current implementation 2bbb112 VS after applying this patch 290440b. I understand that your numbers were different but I acknowledge that my benchmarks focus only on a specific batch-size and hardware combination. The difference is big but I was hoping that we could fix the speed difference by optimizing the underlying kernel and avoid needing workarounds on TorchVision. Have you spoken with Core about it? Overall what are your thoughts?

[liuzhuang13]

@datumbox Thanks! It seems after your patch our implementations are really similar, both using permuting and linear layers. If I understand your results correctly, yours is 15~20% faster than ours somehow. Any idea why? We will benchmark your current patched version against ours too.

[datumbox]

@liuzhuang13 I've assumed it's because I don't use a custom LayerNorm. I would love to know if you can confirm the above with your benchmarks. I've also notified the PyTorch performance team to see if that's something known and work on or something worth improving on the future (I've tagged you and Saining to the post for FYI).

[NicolasHug]

Thanks @datumbox

[datumbox]

Thanks for the review @NicolasHug. I'm going to move now the class to main TorchVision so that we can include it in the upcoming release.

Edit: On second thought, I'll move the graduation to core area on a separate PR.