[Feature/OTX] Rebase develop to feature/otx before MPA refactoring (#…

…1284) * Update submodule branch (#1222) * Enhance training schedule for multi-label classification (#1212) * [CVS-88098] Remove initialize from export functions (#1226) * Train graph added (#1211) Co-authored-by: Lee, Soobee <[email protected]> * Add @attrs decorator for base configs (#1229) Signed-off-by: Songki Choi <[email protected]> Co-authored-by: Harim Kang <[email protected]> * Pretrained weight download error in MobilenetV3-large-1 of deep-object-reid in SC (#1233) * [Anomaly Task] Revert hpo template (#1230) * 🐞 [Anomaly Task] Fix progress bar (#1223) * [CVS-90555] Fix NaN value in classification (#1244) * update hpo_config.yaml (#1240) * [CVS-90400, CVS-91015] NNCF pruning supported tweaks (#1248) * Turned off pruning_support visibility for anomaly models (CVS-91015) * Disabled pruning for EfficientNet-V2-S (CVS-90400) * [Anomaly Task] 🐞 Fix inference when model backbone changes (#1242) * Fix CVS-91469 sseg compatibility issue * [CVS-91472] Add pruning_supported value (#1263) * Pruning supported tweaks (#1256) * [CVS-90400, CVS-91015] NNCF pruning supported tweaks (#1248) * Turned off pruning_support visibility for anomaly models (CVS-91015) * Disabled pruning for EfficientNet-V2-S (CVS-90400) * Revert "[CVS-90400, CVS-91015] NNCF pruning supported tweaks (#1248)" (#1269) * [OTE-TEST] Disable obsolete test cases (#1220) * [OTE-TEST] hot-fix for MPA performance tests (#1273) * Expose early stopping hyper-parameters for all tasks (#1241) * Resolve pre-commit issues (#1272) * Remove LazyEarlyStopHook in model_multilabel.py (#1281) * Removed xfail (#1239) Signed-off-by: Songki Choi <[email protected]> Co-authored-by: Ashwin Vaidya <[email protected]> Co-authored-by: Jaeguk Hyun <[email protected]> Co-authored-by: Nikita Savelyev <[email protected]> Co-authored-by: Vladisalv Sovrasov <[email protected]> Co-authored-by: Jihwan Eom <[email protected]> Co-authored-by: Songki Choi <[email protected]> Co-authored-by: Soobee Lee <[email protected]> Co-authored-by: Lee, Soobee <[email protected]> Co-authored-by: Eugene Liu <[email protected]> Co-authored-by: Emily Chun <[email protected]> Co-authored-by: ljcornel <[email protected]> Co-authored-by: Eunwoo Shin <[email protected]>
openvinotoolkit · Sep 26, 2022 · caf7492 · caf7492
1 parent d7e7618
commit caf7492
Show file tree

Hide file tree

Showing 154 changed files with 5,053 additions and 3,907 deletions.
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -19,7 +19,7 @@ repos:
       - id: isort
         alias: isort_rest
         name: "isort - legacy (ote_cli|external)"
-        files: '^(ote_cli|external/anomaly)/.*\.py'
+        files: '^(ote_cli|external/anomaly|external/model-preparation-algorithm)/.*\.py'
         exclude: "tests/"
 
   - repo: https://github.com/psf/black
@@ -39,7 +39,7 @@ repos:
       - id: black
         name: "black - legacy (rest)"
         args: [--line-length, "120"]
-        files: '^external/anomaly/.*\.py'
+        files: '^(external/anomaly|external/model-preparation-algorithm)/.*\.py'
 
   - repo: https://github.com/PyCQA/flake8
     rev: "5.0.3"
@@ -56,7 +56,7 @@ repos:
       # is to be removed.
       - id: flake8
         name: "flake8 - legacy "
-        files: '^(ote_sdk|ote_cli|external/anomaly)/.*\.py'
+        files: '^(ote_sdk|ote_cli|external/anomaly|external/model-preparation-algorithm)/.*\.py'
         args: ["--config", ".flake8", "--max-complexity", "20"]
         exclude: ".*/protobuf"
 

diff --git a/QUICK_START_GUIDE.md b/QUICK_START_GUIDE.md
@@ -173,6 +173,9 @@ usage: ote train template params [-h]
                                  [--learning_parameters.learning_rate LEARNING_RATE]
                                  [--learning_parameters.learning_rate_warmup_iters LEARNING_RATE_WARMUP_ITERS]
                                  [--learning_parameters.num_iters NUM_ITERS]
+                                 [--learning_parameters.enable_early_stopping ENABLE_EARLY_STOPPING]
+                                 [--learning_parameters.early_stop_patience EARLY_STOP_PATIENCE]
+                                 [--learning_parameters.early_stop_iteration_patience EARLY_STOP_ITERATION_PATIENCE]
                                  [--postprocessing.confidence_threshold CONFIDENCE_THRESHOLD]
                                  [--postprocessing.result_based_confidence_threshold RESULT_BASED_CONFIDENCE_THRESHOLD]
                                  [--nncf_optimization.enable_quantization ENABLE_QUANTIZATION]
@@ -205,6 +208,22 @@ optional arguments:
                         default_value: 300
                         max_value: 100000
                         min_value: 1
+  --learning_parameters.enable_early_stopping ENABLE_EARLY_STOPPING
+                        header: Enable early stopping of the training
+                        type: BOOLEAN
+                        default_value: True
+  --learning_parameters.early_stop_patience EARLY_STOP_PATIENCE
+                        header: Patience for early stopping
+                        type: INTEGER
+                        default_value: 10
+                        max_value: 50
+                        min_value: 0
+  --learning_parameters.early_stop_iteration_patience EARLY_STOP_ITERATION_PATIENCE
+                        header: Iteration patience for early stopping
+                        type: INTEGER
+                        default_value: 0
+                        max_value: 1000
+                        min_value: 0
   --postprocessing.confidence_threshold CONFIDENCE_THRESHOLD
                         header: Confidence threshold
                         type: FLOAT

diff --git a/external/README.md b/external/README.md
@@ -5,68 +5,60 @@ Every sub-project is fully indepedent from each other, and each of them has its
 
 ## Anomaly Classification
 
-| ID                               | Name  | Complexity (GFlops) | Model size (MB) | Path                                                 |
-| -------------------------------- | ----- | ------------------- | --------------- | ---------------------------------------------------- |
-| ote_anomaly_classification_padim | PADIM | 3.9                 | 168.4           | anomaly/templates/classification/padim/template.yaml |
-| ote_anomaly_classification_stfpm | STFPM | 5.6                 | 21.1            | anomaly/templates/classification/stfpm/template.yaml |
+| ID                               | Name  | Complexity (GFlops) | Model size (MB) | Path                                               |
+| -------------------------------- | ----- | ------------------- | --------------- | -------------------------------------------------- |
+| ote_anomaly_classification_padim | PADIM | 3.9                 | 168.4           | anomaly/configs/classification/padim/template.yaml |
+| ote_anomaly_classification_stfpm | STFPM | 5.6                 | 21.1            | anomaly/configs/classification/stfpm/template.yaml |
 
 ## Anomaly Detection
 
-| ID                          | Name  | Complexity (GFlops) | Model size (MB) | Path                                            |
-| --------------------------- | ----- | ------------------- | --------------- | ----------------------------------------------- |
-| ote_anomaly_detection_padim | PADIM | 3.9                 | 168.4           | anomaly/templates/detection/padim/template.yaml |
-| ote_anomaly_detection_stfpm | STFPM | 5.6                 | 21.1            | anomaly/templates/detection/stfpm/template.yaml |
+| ID                          | Name  | Complexity (GFlops) | Model size (MB) | Path                                          |
+| --------------------------- | ----- | ------------------- | --------------- | --------------------------------------------- |
+| ote_anomaly_detection_padim | PADIM | 3.9                 | 168.4           | anomaly/configs/detection/padim/template.yaml |
+| ote_anomaly_detection_stfpm | STFPM | 5.6                 | 21.1            | anomaly/configs/detection/stfpm/template.yaml |
 
 ## Anomaly Segmentation
 
-| ID                             | Name  | Complexity (GFlops) | Model size (MB) | Path                                               |
-| ------------------------------ | ----- | ------------------- | --------------- | -------------------------------------------------- |
-| ote_anomaly_segmentation_padim | PADIM | 3.9                 | 168.4           | anomaly/templates/segmentation/padim/template.yaml |
-| ote_anomaly_segmentation_stfpm | STFPM | 5.6                 | 21.1            | anomaly/templates/segmentation/stfpm/template.yaml |
+| ID                             | Name  | Complexity (GFlops) | Model size (MB) | Path                                             |
+| ------------------------------ | ----- | ------------------- | --------------- | ------------------------------------------------ |
+| ote_anomaly_segmentation_padim | PADIM | 3.9                 | 168.4           | anomaly/configs/segmentation/padim/template.yaml |
+| ote_anomaly_segmentation_stfpm | STFPM | 5.6                 | 21.1            | anomaly/configs/segmentation/stfpm/template.yaml |
 
 ## Image Classification
 
-| ID                                                             | Name                             | Complexity (GFlops) | Model size (MB) | Path                                                                                               |
-| -------------------------------------------------------------- | -------------------------------- | ------------------- | --------------- | -------------------------------------------------------------------------------------------------- |
-| ClassIncremental_Image_Classification_MobileNet-V3-small       | MobileNet-V3-small-ClsIncr       | 0.12                | 1.56            | model-preparation-algorithm/configs/classification/mobilenet_v3_small_cls_incr/template.yaml       |
-| ClassIncremental_Image_Classification_MobileNet-V3-large-0.75x | MobileNet-V3-large-0.75x-ClsIncr | 0.32                | 2.76            | model-preparation-algorithm/configs/classification/mobilenet_v3_large_075_cls_incr/template.yaml   |
-| ClassIncremental_Image_Classification_MobileNet-V3-large-1x    | MobileNet-V3-large-1x-ClsIncr    | 0.44                | 4.29            | model-preparation-algorithm/configs/classification/mobilenet_v3_large_1_cls_incr/template.yaml     |
-| Custom_Image_Classification_MobileNet-V3-large-1x              | MobileNet-V3-large-1x            | 0.44                | 4.29            | deep-object-reid/configs/ote_custom_classification/mobilenet_v3_large_1/template_experimental.yaml |
-| ClassIncremental_Image_Classification_EfficinetNet-B0          | EfficientNet-B0-ClsIncr          | 0.81                | 4.09            | model-preparation-algorithm/configs/classification/efficientnet_b0_cls_incr/template.yaml          |
-| Custom_Image_Classification_EfficinetNet-B0                    | EfficientNet-B0                  | 0.81                | 4.09            | deep-object-reid/configs/ote_custom_classification/efficientnet_b0/template_experimental.yaml      |
-| ClassIncremental_Image_Classification_EfficinetNet-V2-S        | EfficientNet-V2-S-ClsIncr        | 5.76                | 20.23           | model-preparation-algorithm/configs/classification/efficientnet_v2_s_cls_incr/template.yaml        |
-| Custom_Image_Classification_EfficientNet-V2-S                  | EfficientNet-V2-S                | 5.76                | 20.23           | deep-object-reid/configs/ote_custom_classification/efficientnet_v2_s/template_experimental.yaml    |
+| ID                                                | Name                  | Complexity (GFlops) | Model size (MB) | Path                                                                                           |
+| ------------------------------------------------- | --------------------- | ------------------- | --------------- | ---------------------------------------------------------------------------------------------- |
+| Custom_Image_Classification_MobileNet-V3-large-1x | MobileNet-V3-large-1x | 0.44                | 4.29            | model-preparation-algorithm/configs/classification/mobilenet_v3_large_1_cls_incr/template.yaml |
+| Custom_Image_Classification_EfficinetNet-B0       | EfficientNet-B0       | 0.81                | 4.09            | model-preparation-algorithm/configs/classification/efficientnet_b0_cls_incr/template.yaml      |
+| Custom_Image_Classification_EfficientNet-V2-S     | EfficientNet-V2-S     | 5.76                | 20.23           | model-preparation-algorithm/configs/classification/efficientnet_v2_s_cls_incr/template.yaml    |
 
 ## Object Detection
 
-| ID                                           | Name          | Complexity (GFlops) | Model size (MB) | Path                                                                                         |
-| -------------------------------------------- | ------------- | ------------------- | --------------- | -------------------------------------------------------------------------------------------- |
-| Custom_Object_Detection_YOLOX                | YOLOX         | 6.5                 | 20.4            | mmdetection/configs/custom-object-detection/cspdarknet_YOLOX/template_experimental.yaml      |
-| Custom_Object_Detection_Gen3_SSD             | SSD           | 9.4                 | 7.6             | mmdetection/configs/custom-object-detection/gen3_mobilenetV2_SSD/template_experimental.yaml  |
-| ClassIncremental_Object_Detection_Gen3_ATSS  | ATSS-ClsIncr  | 20.6                | 9.1             | model-preparation-algorithm/configs/detection/mobilenetv2_atss_cls_incr/template.yaml        |
-| Custom_Object_Detection_Gen3_ATSS            | ATSS          | 20.6                | 9.1             | mmdetection/configs/custom-object-detection/gen3_mobilenetV2_ATSS/template_experimental.yaml |
-| ClassIncremental_Object_Detection_Gen3_VFNet | VFNet-ClsIncr | 457.4               | 126.0           | model-preparation-algorithm/configs/detection/resnet50_vfnet_cls_incr/template.yaml          |
+| ID                                | Name  | Complexity (GFlops) | Model size (MB) | Path                                                                                  |
+| --------------------------------- | ----- | ------------------- | --------------- | ------------------------------------------------------------------------------------- |
+| Custom_Object_Detection_YOLOX     | YOLOX | 6.5                 | 20.4            | model-preparation-algorithm/configs/detection/cspdarknet_yolox_cls_incr/template.yaml |
+| Custom_Object_Detection_Gen3_SSD  | SSD   | 9.4                 | 7.6             | model-preparation-algorithm/configs/detection/mobilenetv2_ssd_cls_incr/template.yaml  |
+| Custom_Object_Detection_Gen3_ATSS | ATSS  | 20.6                | 9.1             | model-preparation-algorithm/configs/detection/mobilenetv2_atss_cls_incr/template.yaml |
 
-## Object Counting
+## Instance Segmentation (Object Counting)
 
-| ID                                                             | Name                     | Complexity (GFlops) | Model size (MB) | Path                                                                                    |
-| -------------------------------------------------------------- | ------------------------ | ------------------- | --------------- | --------------------------------------------------------------------------------------- |
-| Custom_Counting_Instance_Segmentation_MaskRCNN_EfficientNetB2B | MaskRCNN-EfficientNetB2B | 68.48               | 13.27           | mmdetection/configs/custom-counting-instance-seg/efficientnetb2b_maskrcnn/template.yaml |
-| Custom_Counting_Instance_Segmentation_MaskRCNN_ResNet50        | MaskRCNN-ResNet50        | 533.8               | 177.9           | mmdetection/configs/custom-counting-instance-seg/resnet50_maskrcnn/template.yaml        |
+| ID                                                             | Name                     | Complexity (GFlops) | Model size (MB) | Path                                                                                             |
+| -------------------------------------------------------------- | ------------------------ | ------------------- | --------------- | ------------------------------------------------------------------------------------------------ |
+| Custom_Counting_Instance_Segmentation_MaskRCNN_EfficientNetB2B | MaskRCNN-EfficientNetB2B | 68.48               | 13.27           | model-preparation-algorithm/configs/instance-segmentation/efficientnetb2b_maskrcnn/template.yaml |
+| Custom_Counting_Instance_Segmentation_MaskRCNN_ResNet50        | MaskRCNN-ResNet50        | 533.8               | 177.9           | model-preparation-algorithm/configs/instance-segmentation/resnet50_maskrcnn/template.yaml        |
 
 ## Rotated Object Detection
 
-| ID                                                                          | Name                     | Complexity (GFlops) | Model size (MB) | Path                                                                         |
-| --------------------------------------------------------------------------- | ------------------------ | ------------------- | --------------- | ---------------------------------------------------------------------------- |
-| Custom_Rotated_Detection_via_Instance_Segmentation_MaskRCNN_EfficientNetB2B | MaskRCNN-EfficientNetB2B | 68.48               | 13.27           | mmdetection/configs/rotated_detection/efficientnetb2b_maskrcnn/template.yaml |
-| Custom_Rotated_Detection_via_Instance_Segmentation_MaskRCNN_ResNet50        | MaskRCNN-ResNet50        | 533.8               | 177.9           | mmdetection/configs/rotated_detection/resnet50_maskrcnn/template.yaml        |
+| ID                                                                          | Name                     | Complexity (GFlops) | Model size (MB) | Path                                                                                         |
+| --------------------------------------------------------------------------- | ------------------------ | ------------------- | --------------- | -------------------------------------------------------------------------------------------- |
+| Custom_Rotated_Detection_via_Instance_Segmentation_MaskRCNN_EfficientNetB2B | MaskRCNN-EfficientNetB2B | 68.48               | 13.27           | model-preparation-algorithm/configs/rotated-detection/efficientnetb2b_maskrcnn/template.yaml |
+| Custom_Rotated_Detection_via_Instance_Segmentation_MaskRCNN_ResNet50        | MaskRCNN-ResNet50        | 533.8               | 177.9           | model-preparation-algorithm/configs/rotated-detection/resnet50_maskrcnn/template.yaml        |
 
 ## Semantic Segmentation
 
-| ID                                                       | Name                      | Complexity (GFlops) | Model size (MB) | Path                                                                                      |
-| -------------------------------------------------------- | ------------------------- | ------------------- | --------------- | ----------------------------------------------------------------------------------------- |
-| Custom_Semantic_Segmentation_Lite-HRNet-s-mod2_OCR       | Lite-HRNet-s-mod2 OCR     | 1.82                | 3.5             | mmsegmentation/configs/custom-sematic-segmentation/ocr-lite-hrnet-s-mod2/template.yaml    |
-| ClassIncremental_Semantic_Segmentation_Lite-HRNet-18_OCR | Lite-HRNet-18 OCR-ClsIncr | 3.45                | 4.5             | model-preparation-algorithm/configs/segmentation/ocr-lite-hrnet-18-cls-incr/template.yaml |
-| Custom_Semantic_Segmentation_Lite-HRNet-18_OCR           | Lite-HRNet-18 OCR         | 3.45                | 4.5             | mmsegmentation/configs/custom-sematic-segmentation/ocr-lite-hrnet-18/template.yaml        |
-| Custom_Semantic_Segmentation_Lite-HRNet-18-mod2_OCR      | Lite-HRNet-18-mod2 OCR    | 3.63                | 4.8             | mmsegmentation/configs/custom-sematic-segmentation/ocr-lite-hrnet-18-mod2/template.yaml   |
-| Custom_Semantic_Segmentation_Lite-HRNet-x-mod3_OCR       | Lite-HRNet-x-mod3 OCR     | 13.97               | 6.4             | mmsegmentation/configs/custom-sematic-segmentation/ocr-lite-hrnet-x-mod3/template.yaml    |
+| ID                                                  | Name               | Complexity (GFlops) | Model size (MB) | Path                                                                                  |
+| --------------------------------------------------- | ------------------ | ------------------- | --------------- | ------------------------------------------------------------------------------------- |
+| Custom_Semantic_Segmentation_Lite-HRNet-s-mod2_OCR  | Lite-HRNet-s-mod2  | 1.82                | 3.5             | model-preparation-algorithm/configs/segmentation/ocr-lite-hrnet-s-mod2/template.yaml  |
+| Custom_Semantic_Segmentation_Lite-HRNet-18_OCR      | Lite-HRNet-18      | 3.45                | 4.5             | model-preparation-algorithm/configs/segmentation/ocr-lite-hrnet-18/template.yaml      |
+| Custom_Semantic_Segmentation_Lite-HRNet-18-mod2_OCR | Lite-HRNet-18-mod2 | 3.63                | 4.8             | model-preparation-algorithm/configs/segmentation/ocr-lite-hrnet-18-mod2/template.yaml |
+| Custom_Semantic_Segmentation_Lite-HRNet-x-mod3_OCR  | Lite-HRNet-x-mod3  | 13.97               | 6.4             | model-preparation-algorithm/configs/segmentation/ocr-lite-hrnet-x-mod3/template.yaml  |
diff --git a/...p-object-reid/configs/ote_custom_classification/efficientnet_v2_s/compression_config.json b/...p-object-reid/configs/ote_custom_classification/efficientnet_v2_s/compression_config.json
@@ -44,16 +44,6 @@
         },
         "nncf_config": {
             "compression": [
-                {
-                    "algorithm": "filter_pruning",
-                    "pruning_init": 0.1,
-                    "params": {
-                        "schedule": "baseline",
-                        "pruning_flops_target": 0.1,
-                        "filter_importance": "geometric_median",
-                        "prune_downsample_convs": true
-                    }
-                },
                 {
                     "algorithm": "quantization",
                     "preset": "mixed",

diff --git a/...bject-reid/configs/ote_custom_classification/efficientnet_v2_s/template_experimental.yaml b/...bject-reid/configs/ote_custom_classification/efficientnet_v2_s/template_experimental.yaml
@@ -43,7 +43,7 @@ hyper_parameters:
       enable_pruning:
         default_value: false
       pruning_supported:
-        default_value: true
+        default_value: false
       maximal_accuracy_degradation:
         default_value: 1.0
 

diff --git a/external/deep-object-reid/constraints.txt b/external/deep-object-reid/constraints.txt
@@ -1 +1,2 @@
 opencv-python==4.5.5.64  # remedy for fixed opencv-python-headless version in e2e-test-framework
+optuna==2.10.1 # remedy for fixed optuna version incompatible in OTE CI
diff --git a/external/deep-object-reid/submodule b/external/deep-object-reid/submodule
Original file line number	Diff line number	Diff line change
		@@ -1 +1,2 @@
		opencv-python==4.5.5.64 # remedy for fixed opencv-python-headless version in e2e-test-framework
		optuna==2.10.1 # remedy for fixed optuna version incompatible in OTE CI
+2 −1		.github/workflows/main.yml
+2 −2		openvino-requirements.txt
+0 −1		optional-requirements.txt
+1 −1		requirements.txt
+1 −1		tools/auxiliary/optuna_optim.py
+1 −1		tools/auxiliary/run_classification.py
+1 −1		tools/auxiliary/run_multilabel.py
+63 −0		tools/check_classes_mlc.py
+2 −2		torchreid/engine/engine.py
+4 −7		torchreid/losses/asl.py
+0 −6		torchreid/models/timm_wrapper.py
+3 −1		torchreid/utils/torchtools.py