[Feature] Support TIMMBackbone #998

Junjun2016 · 2021-10-26T19:38:42Z

Motivation

TIMM (https://github.com/rwightman/pytorch-image-models) contains a lot of backbones, which are not available in this repository. But mmsegmentation code is more flexible for training and adding new components for semantic segmentation. It would be convenient to have a wrapper here to use TIMM backbones.

Related PR：open-mmlab/mmpretrain#427

Modification

This PR adds TIMMBackbone wrapper to use timm backbones.

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
No

Use cases (Optional)

To use resnet18 from timm with pretrained weights and output stride 8, you need to use the following backbone config:

backbone=dict(
    type="TIMMBackbone",
    model_name="resnet18",
    features_only=True,
    pretrained=True,   
    output_stride=8)

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
The documentation has been modified accordingly, like docstring or example tutorials.

codecov · 2021-10-26T19:56:29Z

Codecov Report

Merging #998 (0db1bd1) into master (93f8b14) will decrease coverage by 0.02%.
The diff coverage is 84.00%.

@@            Coverage Diff             @@
##           master     #998      +/-   ##
==========================================
- Coverage   89.85%   89.83%   -0.03%     
==========================================
  Files         118      119       +1     
  Lines        6558     6583      +25     
  Branches     1019     1023       +4     
==========================================
+ Hits         5893     5914      +21     
- Misses        464      467       +3     
- Partials      201      202       +1

Flag	Coverage Δ
unittests	`89.83% <84.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/models/backbones/timm_backbone.py	`83.33% <83.33%> (ø)`
mmseg/models/backbones/__init__.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 93f8b14...0db1bd1. Read the comment docs.

xvjiarui · 2021-10-26T21:28:35Z

.github/workflows/build.yml

+        if: ${{matrix.torch >= '1.6.0'}}
+      - name: Run unittests and generate coverage report and skip timm unittests
+        run: |
+          coverage run --branch --source mmseg -m pytest tests/ --ignore tests/test_models/test_backbones/test_timm_backbone.py


Could CI work with TIMM?

https://github.com/open-mmlab/mmsegmentation/runs/4016108790?check_suite_focus=true

Can work with PyTorch >= 1.6.0, but failed with PyTorch 1.5.1
Please take a look at these two commits:
b75c201
bc7e80e

xvjiarui · 2021-10-27T02:30:21Z

tests/test_models/test_backbones/test_timm_backbone.py

+    model = TIMMBackbone(
+        model_name='resnetv2_152x2_bitm',
+        features_only=True,
+        pretrained=False,
+        output_stride=8)
+    imgs = torch.randn(1, 3, 16, 16)
+    feats = model(imgs)
+    feats = [feat.shape for feat in feats]
+    assert len(feats) == 5
+    assert feats[0] == torch.Size((1, 128, 8, 8))
+    assert feats[1] == torch.Size((1, 512, 4, 4))
+    assert feats[2] == torch.Size((1, 1024, 2, 2))
+    assert feats[3] == torch.Size((1, 2048, 2, 2))
+    assert feats[4] == torch.Size((1, 4096, 2, 2))
+
+    # Test resnetv2_152x4_bitm from timm, features_only=True, output_stride=8
+    model = TIMMBackbone(
+        model_name='resnetv2_152x4_bitm',
+        features_only=True,
+        pretrained=False,
+        output_stride=8)


We ignore large models

68bf65f

Remove unittests of large models of timm backbone and include pt 1.5.1 for timm unittest

xvjiarui

#1004 is merge. Maybe we could try if TIMM works with 1.5.0

Junjun2016 · 2021-11-01T15:31:34Z

#1004 is merge. Maybe we could try if TIMM works with 1.5.0

It works with pt1.5 after removing the unittests for resnetv2_101x3, resnetv2_152x2, and resnetv2_152x4.

500f22e 0db1bd1

Junjun2016 · 2021-11-01T15:45:00Z

xvjiarui · 2021-11-01T15:57:19Z

.github/workflows/build.yml

+        if: ${{matrix.torch >= '1.5.0'}}
+      - name: Skip timm unittests and generate coverage report
+        run: |
+          coverage run --branch --source mmseg -m pytest tests/ --ignore tests/test_models/test_backbones/test_timm_backbone.py


Maybe we could undo the ignore

TIMM does not support pt1.3

if: ${{matrix.torch < '1.5.0'}} will skip the timm unittests.

based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998

based on open-mmlab/mmsegmentation#998

* Support features_only in TIMMBackbone based on open-mmlab/mmsegmentation#998 * update test for mmdet * fix unit test for build_without_timm * Update docstring Co-authored-by: mzr1996 <[email protected]>

* add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs

* Support features_only in TIMMBackbone based on open-mmlab/mmsegmentation#998 * update test for mmdet * fix unit test for build_without_timm * Update docstring Co-authored-by: mzr1996 <[email protected]>

* add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs

* add TIMMBackbone and unittests * add timm to tests requirements * deprecate pt1.3.1 * reduce the unittests input of timm backbone * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * fix ci * remove unittests of large models of timm backbone * generate coverage report for all unittests env * reduce the unittests input of timm backbone * reduce the unittests input of timm backbone

* [Enhancement] Upgrade isort in pre-commit hook (#7130) * upgrade isort to v5.10.1 * replace known_standard_library with extra_standard_library * upgrade isort to v5.10.1 replace known_standard_library with extra_standard_library * imports order changes * [Fix] cannot to save the best checkpoint when the key_score is None (#7101) * [Fix] Fix MixUp transform filter boxes failing case. Added test case (#7080) * [Fix] Update the version limitation of mmcv-full and pytorch in CI. (#7133) * Update * Update build.yml * Update build.yml * [Feature] Support TIMMBackbone (#7020) * add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs * Create 2_new_data_model.md (#6476) fix some typo Co-authored-by: PJLAB\huanghaian <[email protected]> * [FIX] add Ci of pytorch 1.10 and comments for bbox clamp (#7081) (#7083) * add comments for bbox clamp * add CI of pytorch1.10 * add ci of pytorch1.10.1 * mmcv1.9.0->mmcv1.9 * add ci of pytorch1.10 * Add daily issue owners (#7163) * Add code owners Signed-off-by: del-zhenwu <[email protected]> * Update code owners Signed-off-by: del-zhenwu <[email protected]> * [Feature] Support visualization for Panoptic Segmentation (#7041) * First commit of v2 * split the functions * Support to show panoptic result * temp * Support to show gt * support show gt * fix lint * Support to browse datasets * Fix unit tests * Fix findContours * fix comments * Fix pre-commit * fix lint * Add the type of an argument * [Fix] confusion_matrix.py analysis tool handling NaNs (#7147) * [Fix] Added missing property in SABLHead (#7091) * Added missing property in SABLHead * set pre-commit-hooks to v0.1.0 * set maskdownlint to v0.11.0 * pre-commit-hooks Co-authored-by: Cedric Luo <[email protected]> * Update config.md (#7215) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Update config.md fix some typos Co-authored-by: Jamie <[email protected]> Co-authored-by: BigDong <[email protected]> * [Feature] Support DyHead (#6823) * add DyHead * move and update DYReLU * update * replace stack with sum to reduce memory * clean and update * update to align inference accuracy (incomplete) * fix pad * update to align training accuracy and pick #6867 * add README and metafile * update docs * resolve comments * revert picking 6867 * update README.md * update metafile.yml * resolve comments and update urls * Fix broken colab link (#7218) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Fix broken colab link Co-authored-by: Jamie <[email protected]> Co-authored-by: BigDong <[email protected]> * Remove the inplace addition in `FPN` (#7175) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Remove the inplace addition in `FPN` * update Co-authored-by: Jamie <[email protected]> Co-authored-by: BigDong <[email protected]> Co-authored-by: PJLAB\huanghaian <[email protected]> * [Feature] Support OpenImages Dataset (#6331) * [Feature] support openimage group of eval * [Feature] support openimage group of eval * support openimage dataset * support openimage challenge dataset * fully support OpenImages-V6 and OpenImages Challenge 2019 * Fix some logic error * update config file * fix get data_infos error * fully support OpenImages evaluation * update OpenImages config files * [Feature] support OpenImages datasets * fix bug * support load image metas from pipeline * fix bug * fix get classes logic error * update code * support get image metas * support openimags * support collect image metas * support Open Images * fix openimages logic * minor fix * add a new function to compute openimages tpfp * minor fix * fix ci error * minor fix * fix indication * minor fix * fix returns * fix returns * fix returns * fix returns * fix returns * minor fix * update readme * support loading image level labels and fix some logic * minor fix * minor fix * add class names * minor fix * minor fix * minor fix * add openimages test unit * minor fix * minor fix * fix test unit * minor fix * fix logic error * minor fix * fully support openimages * minor fix * fix docstring * fix docstrings in readthedocs * update get image metas script * label_description_file -> label_file * update openimages readme * fix test unit * fix test unit * minor fix * update readme file * Update get_image_metas.py * [Enhance] Speed up SimOTA matching. (#7098) * [Feature] Add Maskformer to mmdet (#7212) * first commit * add README * move model description from config to readme add description for binary_input add description for dice loss add a independent panoptic gt processing function add a independent panoptic gt processing function remove compatibility of pretrain in maskformer * update comments in maskformer_head * update docs format * Add deprecation message for deploy tools (#7242) * Add CI for windows (#7228) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Add mmrotate (#7216) * fix description for args in CSPDarknet (#7187) * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * Update build.yml * fix test_find_latest_checkpoint * fix data_infos__default_db_directories * fix test_custom_classes_override_default * Update test_custom_dataset.py * Update test_common.py * Update assign_result.py * use os.path.join * fix bug * Update test_common.py * Update assign_result.py * Update sampling_result.py * os.path -> osp * os.path -> osp Co-authored-by: Jamie <[email protected]> Co-authored-by: BigDong <[email protected]> Co-authored-by: Hyeokjoon Kwon <[email protected]> * add Chinese version of init_cfg (#7188) * [Fix] Fix wrong img name in onnx2tensorrt.py (#7157) * [Docs] fix albumentations installed way (#7143) * Create init_cfg.md * Update docs/zh_cn/tutorials/init_cfg.md Co-authored-by: Haian Huang(深度眸) <[email protected]> * update init_cfg.md * update init_cfg.md * update init_cfg.md * update init_cfg.md Co-authored-by: Jamie <[email protected]> Co-authored-by: BigDong <[email protected]> Co-authored-by: Haian Huang(深度眸) <[email protected]> * update MaskFormer readme and docs (#7241) * update docs for maskformer * update readme * update readme format * update link * update json link * update format of ConfigDict * update format of function returns * uncomment main in deployment/test.py * [Feature] ResNet Strikes Back. (#7001) * [Feature] ResNet Strikes Back. * add more cfg * add readme * update * update * update * update * update * update * Maskformer Visualization (#7247) * maskformer visualization * compatible with models that do not contain arg of pretrained * compatible with models that do not contain arg of pretrained * Bump versions to v2.22.0 (#7240) * Bump versions to v2.22.0 * Fix comments and add the latest PRs * fix the id of contributor * relax the version of mmcv * Add ResNet Strikes Back * Update README_zh-CN.md * Update README.md * fix typo * Update README_zh-CN.md Co-authored-by: Wenwei Zhang <[email protected]> * Maskformer metafile and rsb readme format (#7250) * [Fix] Fix Open Images testunit to avoid error in Windows CI (#7252) * [Feature] support openimage group of eval * [Feature] support openimage group of eval * support openimage dataset * support openimage challenge dataset * fully support OpenImages-V6 and OpenImages Challenge 2019 * Fix some logic error * update config file * fix get data_infos error * fully support OpenImages evaluation * update OpenImages config files * [Feature] support OpenImages datasets * fix bug * support load image metas from pipeline * fix bug * fix get classes logic error * update code * support get image metas * support openimags * support collect image metas * support Open Images * fix openimages logic * minor fix * add a new function to compute openimages tpfp * minor fix * fix ci error * minor fix * fix indication * minor fix * fix returns * fix returns * fix returns * fix returns * fix returns * minor fix * update readme * support loading image level labels and fix some logic * minor fix * minor fix * add class names * minor fix * minor fix * minor fix * add openimages test unit * minor fix * minor fix * fix test unit * minor fix * fix logic error * minor fix * fully support openimages * minor fix * fix docstring * fix docstrings in readthedocs * update get image metas script * label_description_file -> label_file * update openimages readme * fix test unit * fix test unit * minor fix * update readme file * Update get_image_metas.py * fix oid testunit to avoid some error in windows * minor fix to avoid some error in windows * minor fix * add comments in oid test unit * minor fix Co-authored-by: Cedric Luo <[email protected]> Co-authored-by: LuooChen <[email protected]> Co-authored-by: Daniel van Sabben Alsina <[email protected]> Co-authored-by: jbwang1997 <[email protected]> Co-authored-by: Yosuke Shinya <[email protected]> Co-authored-by: siatwangmin <[email protected]> Co-authored-by: PJLAB\huanghaian <[email protected]> Co-authored-by: del-zhenwu <[email protected]> Co-authored-by: Guangchen Lin <[email protected]> Co-authored-by: VIKASH RANJAN <[email protected]> Co-authored-by: Range King <[email protected]> Co-authored-by: Jamie <[email protected]> Co-authored-by: BigDong <[email protected]> Co-authored-by: Haofan Wang <[email protected]> Co-authored-by: Zhijian Liu <[email protected]> Co-authored-by: BigDong <[email protected]> Co-authored-by: RangiLyu <[email protected]> Co-authored-by: Yue Zhou <[email protected]> Co-authored-by: Hyeokjoon Kwon <[email protected]> Co-authored-by: Kevin Ye <[email protected]>

* add TIMMBackbone based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998 * update and clean * fix unit test * Revert * add example configs

* Support features_only in TIMMBackbone based on open-mmlab/mmsegmentation#998 * update test for mmdet * fix unit test for build_without_timm * Update docstring Co-authored-by: mzr1996 <[email protected]>

…xample (open-mmlab#998) tensorboard import in readme, otherwise accelerator.trackers[0] out of range Co-authored-by: lukovnikov <[email protected]>

Junjun2016 added 2 commits October 27, 2021 03:31

add TIMMBackbone and unittests

064e1f9

add timm to tests requirements

ec8ffae

Junjun2016 added 4 commits October 27, 2021 04:18

deprecate pt1.3.1

57e3983

reduce the unittests input of timm backbone

3fa7650

fix ci

2a8503c

fix ci

eeca97a

xvjiarui reviewed Oct 26, 2021

View reviewed changes

Junjun2016 added 7 commits October 27, 2021 05:30

fix ci

831f6ef

fix ci

e4b2cb0

fix ci

f25eea5

fix ci

b75c201

fix ci

bc7e80e

fix ci

c4cbe44

fix ci

a09d948

xvjiarui reviewed Oct 27, 2021

View reviewed changes

Junjun2016 added 4 commits October 27, 2021 16:28

remove unittests of large models of timm backbone

68bf65f

generate coverage report for all unittests env

5eced88

reduce the unittests input of timm backbone

500f22e

reduce the unittests input of timm backbone

0db1bd1

xvjiarui approved these changes Nov 1, 2021

View reviewed changes

xvjiarui reviewed Nov 1, 2021

View reviewed changes

Junjun2016 merged commit 54435fb into open-mmlab:master Nov 2, 2021

shinya7y added a commit to shinya7y/mmdetection that referenced this pull request Jan 16, 2022

add TIMMBackbone

4450e11

based on open-mmlab/mmpretrain#427 open-mmlab/mmsegmentation#998

shinya7y mentioned this pull request Jan 17, 2022

[Feature] Support TIMMBackbone open-mmlab/mmdetection#7020

Merged

shinya7y added a commit to shinya7y/mmclassification that referenced this pull request Jan 23, 2022

Support features_only in TIMMBackbone

b0a38ad

based on open-mmlab/mmsegmentation#998

shinya7y mentioned this pull request Jan 23, 2022

[Feature] Support features_only in TIMMBackbone open-mmlab/mmpretrain#668

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support TIMMBackbone #998

[Feature] Support TIMMBackbone #998

Junjun2016 commented Oct 26, 2021 •

edited

Loading

codecov bot commented Oct 26, 2021 •

edited

Loading

xvjiarui Oct 26, 2021

Junjun2016 Oct 26, 2021

xvjiarui Oct 27, 2021

Junjun2016 Oct 27, 2021

xvjiarui left a comment

Junjun2016 commented Nov 1, 2021 •

edited

Loading

Junjun2016 commented Nov 1, 2021

xvjiarui Nov 1, 2021

Junjun2016 Nov 1, 2021

Junjun2016 Nov 1, 2021

[Feature] Support TIMMBackbone #998

[Feature] Support TIMMBackbone #998

Conversation

Junjun2016 commented Oct 26, 2021 • edited Loading

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

codecov bot commented Oct 26, 2021 • edited Loading

Codecov Report

xvjiarui Oct 26, 2021

Choose a reason for hiding this comment

Junjun2016 Oct 26, 2021

Choose a reason for hiding this comment

xvjiarui Oct 27, 2021

Choose a reason for hiding this comment

Junjun2016 Oct 27, 2021

Choose a reason for hiding this comment

xvjiarui left a comment

Choose a reason for hiding this comment

Junjun2016 commented Nov 1, 2021 • edited Loading

Junjun2016 commented Nov 1, 2021

xvjiarui Nov 1, 2021

Choose a reason for hiding this comment

Junjun2016 Nov 1, 2021

Choose a reason for hiding this comment

Junjun2016 Nov 1, 2021

Choose a reason for hiding this comment

Junjun2016 commented Oct 26, 2021 •

edited

Loading

codecov bot commented Oct 26, 2021 •

edited

Loading

Junjun2016 commented Nov 1, 2021 •

edited

Loading