[Feature] Support TIMMBackbone

[shinya7y]

Motivation

timm (https://github.com/rwightman/pytorch-image-models) contains a lot of backbones, which are not available in this repository. It would be convenient to add examples to use timm backbones.

Related to the features of mmcls and mmseg.
open-mmlab/mmpretrain#427
open-mmlab/mmsegmentation#998

Modification

This PR adds example configs and a document to use timm backbones via the TIMMBackbone wrapper in mmcls.

Use cases (Optional)

Please see config files.

We can use ConvNeXt with timm>=0.5.4. However, other efforts are needed to support the settings of the official ConvNeXt (e.g., LearningRateDecayOptimizerConstructor). Related to open-mmlab/mmsegmentation#1216

[codecov]

Codecov Report

Merging #7020 (414c73a) into dev (4bdb312) will decrease coverage by 0.27%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##              dev    #7020      +/-   ##
==========================================
- Coverage   62.41%   62.14%   -0.28%     
==========================================
  Files         330      330              
  Lines       26199    26018     -181     
  Branches     4436     4436              
==========================================
- Hits        16353    16168     -185     
- Misses       8976     8977       +1     
- Partials      870      873       +3     

Flag	Coverage Δ
unittests	62.12% <ø> (-0.28%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmdet/utils/misc.py	95.23% <0.00%> (-4.77%)	⬇️
mmdet/models/dense_heads/base_mask_head.py	40.00% <0.00%> (-4.45%)	⬇️
mmdet/models/seg_heads/base_semantic_head.py	34.37% <0.00%> (-3.87%)	⬇️
mmdet/models/roi_heads/base_roi_head.py	76.47% <0.00%> (-3.53%)	⬇️
mmdet/core/bbox/coder/yolo_bbox_coder.py	50.00% <0.00%> (-2.95%)	⬇️
mmdet/models/detectors/base.py	50.78% <0.00%> (-2.56%)	⬇️
...panoptic_fusion_heads/base_panoptic_fusion_head.py	52.63% <0.00%> (-2.37%)	⬇️
mmdet/datasets/pipelines/formatting.py	65.78% <0.00%> (-1.98%)	⬇️
mmdet/models/detectors/lad.py	32.35% <0.00%> (-1.94%)	⬇️
mmdet/core/anchor/point_generator.py	52.08% <0.00%> (-1.92%)	⬇️
... and 61 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4bdb312...414c73a. Read the comment docs.

[ZwwWayne]

Can we directly use the TIMMBackbone in MMCls? I think we can directly use the config like

_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/coco_detection.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
model = dict(
    backbone=dict(
        _delete_=True,
        type='mmcls.TIMMBackbone',
        model_name='resnet18',
        out_indices=(1, 2, 3, 4)),
    neck=dict(in_channels=[64, 128, 256, 512]))

# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

After we installed mmcls/timm

[shinya7y]

We need to update mmcls's TIMMBackbone for mmdet and mmseg because mmcls's TIMMBackbone does not support features_only=True.
After updating, the config will be as follows.

_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/coco_detection.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
model = dict(
    backbone=dict(
        _delete_=True,
        type='mmcls.TIMMBackbone',
        model_name='resnet18',
        features_only=True,  # mmcls assumes False only, causing error
        pretrained=True,
        out_indices=(1, 2, 3, 4)),
    neck=dict(in_channels=[64, 128, 256, 512]))

# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

# import to trigger register_module in mmcls
custom_imports = dict(imports=['mmcls.models'], allow_failed_imports=False)

# disable NumClassCheckHook
custom_hooks = []

[ZwwWayne]

We need to update mmcls's TIMMBackbone for mmdet and mmseg because mmcls's TIMMBackbone does not support features_only=True. After updating, the config will be as follows.

_base_ = [
    '../_base_/models/retinanet_r50_fpn.py',
    '../_base_/datasets/coco_detection.py',
    '../_base_/schedules/schedule_1x.py', '../_base_/default_runtime.py'
]
model = dict(
    backbone=dict(
        _delete_=True,
        type='mmcls.TIMMBackbone',
        model_name='resnet18',
        features_only=True,  # mmcls assumes False only, causing error
        pretrained=True,
        out_indices=(1, 2, 3, 4)),
    neck=dict(in_channels=[64, 128, 256, 512]))

# optimizer
optimizer = dict(type='SGD', lr=0.01, momentum=0.9, weight_decay=0.0001)

# import to trigger register_module in mmcls
custom_imports = dict(imports=['mmcls.models'], allow_failed_imports=False)

# disable NumClassCheckHook
custom_hooks = []

Great! Thank you @shinya7y , would you like to directly create a PR in mmcls to support that? We can also create an issue there to let them fix this in this version.

[ZwwWayne]

Hi @shinya7y ,
It seems that mmcls already support that. We can add comments in the config to indicate that it requires the newest mmcls (should be v0.20.0) and clean the code. Hopefully this PR can be merged by this week as a feature in the next release of mmdet.

[shinya7y]

I will do them and put example configs in configs/timm_example like configs/albu_example.
If other directories (e.g., configs/timm_backbone, docs/*/tutorials) are preferred, please let me know.

[ZwwWayne]

I will do them and put example configs in configs/timm_example like configs/albu_example. If other directories (e.g., configs/timm_backbone, docs/*/tutorials) are preferred, please let me know.

Hi @shinya7y,
This idea is great! I think we can do that~

[hhaAndroid]

@shinya7y Great. LGTM.

[hhaAndroid]

@shinya7y The timm of mmcls cannot support frozen stages. Is there any way to do this?

[shinya7y]

@shinya7y The timm of mmcls cannot support frozen stages. Is there any way to do this?

It will be difficult. Related discussion: open-mmlab/mmpretrain#605

[hhaAndroid]

@shinya7y The timm of mmcls cannot support frozen stages. Is there any way to do this?

It will be difficult. Related discussion: open-mmlab/mmclassification#605

OK.

[RangiLyu]

LGTM