[FEATURE] Integrate multi-gpu training for semi-supervised learning and self-supervised learning

[supersoob]

Summary

This PR is to enable multi-gpu training for semi-supervised learning and self-supervised learning.

• Detection SEMI
• Segmentation SEMI/SELF
• Classification SEMI/SELF/SUPCON

Main Issues

• All tensors in losses has to have its local rank num because it is broadcasted or ring-scattered to other gpu. In det semisl, there were some tensors (ps_recall, ps_) without device num
• In self-sl classification, ckpt loading incurs issues that RuntimeError: storage has wrong size. Because two models are needed to be built. After the first model(online_model) is loaded and saved to cache, the second model(target_model) will bring the ckpt from the cache because pretrained url is the same since they are same model. Such access to the cache can be done concurrently and make it corrupt during serialization between two processes.

How to test

• Run e2e or integration test for each task

Checklist

• I submit my changes into the develop branch
• I have added description of my changes into CHANGELOG
• I have updated the documentation accordingly
• I have added tests to cover my changes
• I have linked related issues

License

• I submit my code changes under the same MIT License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below)

# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

[codecov-commenter]

Codecov Report

Patch coverage: 44.44% and project coverage change: -0.01 ⚠️

Comparison is base (6116639) 80.68% compared to head (1c21c29) 80.68%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1534      +/-   ##
===========================================
- Coverage    80.68%   80.68%   -0.01%     
===========================================
  Files          486      486              
  Lines        33249    33253       +4     
===========================================
+ Hits         26826    26829       +3     
- Misses        6423     6424       +1     

Impacted Files	Coverage Δ
...x/mpa/modules/models/detectors/unbiased_teacher.py	19.84% <0.00%> (-0.16%)	⬇️
...fication/adapters/mmcls/models/classifiers/byol.py	89.24% <100.00%> (+0.35%)	⬆️

... and 3 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[jaegukhyun]

Generally LGTM, I left a comment for question

[eunwoosh]

Could you add Test Case for this with skip flag? I'll re-enable multi GPU TCs after checking.

[supersoob]

@jaegukhyun @eunwoosh could you check again?