Skip to content
Permalink

Comparing changes

This is a direct comparison between two commits made in this repository or its related repositories. View the default comparison for this range or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: openvinotoolkit/training_extensions
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 32d851b334c2dbc7cba393ddc4eab632bc7223a3
Choose a base ref
..
head repository: openvinotoolkit/training_extensions
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: be685cba81b62580bcc21da7f45ef95916eb62e7
Choose a head ref
Showing 665 changed files with 8,314 additions and 6,745 deletions.
18 changes: 9 additions & 9 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -22,20 +22,20 @@ not fully covered by unit tests or manual testing can be complicated. -->

<!-- Put an 'x' in all the boxes that apply -->

- [ ] I submit my changes into the `develop` branch
- [ ] I have added description of my changes into [CHANGELOG](https://github.com/openvinotoolkit/training_extensions/blob/develop/CHANGELOG.md)
- [ ] I have updated the [documentation](https://github.com/openvinotoolkit/training_extensions/tree/develop/docs) accordingly
- [ ] I have added tests to cover my changes
- [ ] I have [linked related issues](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword)
- [ ] I have added unit tests to cover my changes.​
- [ ] I have added integration tests to cover my changes.​
- [ ] I have added e2e tests for validation.
- [ ] I have added the description of my changes into CHANGELOG in my target branch (e.g., [CHANGELOG](https://github.com/openvinotoolkit/training_extensions/blob/develop/CHANGELOG.md) in develop).​
- [ ] I have updated the documentation in my target branch accordingly (e.g., [documentation](https://github.com/openvinotoolkit/training_extensions/tree/develop/docs) in develop).
- [ ] I have [linked related issues](https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword).

### License

- [ ] I submit _my code changes_ under the same [MIT License](https://github.com/openvinotoolkit/training_extensions/blob/develop/LICENSE) that covers the project.
- [ ] I submit _my code changes_ under the same [Apache License](https://github.com/openvinotoolkit/training_extensions/blob/develop/LICENSE) that covers the project.
Feel free to contact the maintainers if that's a concern.
- [ ] I have updated the license header for each file (see an example below)
- [ ] I have updated the license header for each file (see an example below).

```python
# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT
# SPDX-License-Identifier: Apache-2.0
```
14 changes: 7 additions & 7 deletions .github/workflows/pre_merge.yml
Original file line number Diff line number Diff line change
@@ -43,7 +43,7 @@ jobs:
- name: Install dependencies
run: python -m pip install -r requirements/dev.txt
- name: Unit-testing
run: tox -re pre-merge -- tests/unit
run: tox -e pre-merge -- tests/unit
- name: Upload coverage reports to Codecov
run: |
# If the workflow is triggered from PR then it gets the commit id from the PR.
@@ -72,7 +72,7 @@ jobs:
- name: Install dependencies
run: python -m pip install -r requirements/dev.txt
- name: Integration-testing
run: tox -re pre-merge -- tests/integration/cli/test_cli.py
run: tox -e pre-merge -- tests/integration/cli/test_cli.py
Pre-Merge-Integration-Cls-Test:
runs-on: [self-hosted, linux, x64, dev]
needs: Pre-Merge-Unit-Test
@@ -83,7 +83,7 @@ jobs:
- name: Install dependencies
run: python -m pip install -r requirements/dev.txt
- name: Integration-testing
run: tox -re pre-merge-cls
run: tox -e pre-merge-cls
Pre-Merge-Integration-Det-Test:
runs-on: [self-hosted, linux, x64, dev]
needs: Pre-Merge-Unit-Test
@@ -94,7 +94,7 @@ jobs:
- name: Install dependencies
run: python -m pip install -r requirements/dev.txt
- name: Integration-testing
run: tox -re pre-merge-det
run: tox -e pre-merge-det
Pre-Merge-Integration-Seg-Test:
runs-on: [self-hosted, linux, x64, dev]
needs: Pre-Merge-Unit-Test
@@ -105,7 +105,7 @@ jobs:
- name: Install dependencies
run: python -m pip install -r requirements/dev.txt
- name: Integration-testing
run: tox -re pre-merge-seg
run: tox -e pre-merge-seg
Pre-Merge-Integration-Action-Test:
runs-on: [self-hosted, linux, x64, dev]
needs: Pre-Merge-Unit-Test
@@ -116,7 +116,7 @@ jobs:
- name: Install dependencies
run: python -m pip install -r requirements/dev.txt
- name: Integration-testing
run: tox -re pre-merge-action
run: tox -e pre-merge-action
Pre-Merge-Integration-Anomaly-Test:
runs-on: [self-hosted, linux, x64, dev]
needs: Pre-Merge-Unit-Test
@@ -127,4 +127,4 @@ jobs:
- name: Install dependencies
run: python -m pip install -r requirements/dev.txt
- name: Integration-testing
run: tox -re pre-merge-anomaly
run: tox -e pre-merge-anomaly
54 changes: 54 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -2,6 +2,60 @@

All notable changes to this project will be documented in this file.

## \[v1.2.0\]

### New features

-

### Enhancements

-

### Bug fixes

-

### Known issues

- OpenVINO(==2022.3) IR inference is not working well on 2-stage models (e.g. Mask-RCNN) exported from torch==1.13.1
(working well up to torch==1.12.1) (<https://github.com/openvinotoolkit/training_extensions/issues/1906>)

## \[v1.1.0\]

### New features

- Add FP16 IR export support (<https://github.com/openvinotoolkit/training_extensions/pull/1683>)
- Add in-memory caching in dataloader (<https://github.com/openvinotoolkit/training_extensions/pull/1694>)
- Add MoViNet template for action classification (<https://github.com/openvinotoolkit/training_extensions/pull/1742>)
- Add Semi-SL multilabel classification algorithm (<https://github.com/openvinotoolkit/training_extensions/pull/1805>)
- Integrate multi-gpu training for semi-supervised learning and self-supervised learning (<https://github.com/openvinotoolkit/training_extensions/pull/1534>)
- Add train-type parameter to otx train (<https://github.com/openvinotoolkit/training_extensions/pull/1874>)
- Add embedding of inference configuration to IR for classification (<https://github.com/openvinotoolkit/training_extensions/pull/1842>)
- Enable VOC dataset in OTX (<https://github.com/openvinotoolkit/training_extensions/pull/1862>)
- Add mmcls.VisionTransformer backbone support (<https://github.com/openvinotoolkit/training_extensions/pull/1908>)

### Enhancements

- Parametrize saliency maps dumping in export (<https://github.com/openvinotoolkit/training_extensions/pull/1888>)
- Bring mmdeploy to action recognition model export & Test optimization of action tasks (<https://github.com/openvinotoolkit/training_extensions/pull/1848>)
- Update backbone lists (<https://github.com/openvinotoolkit/training_extensions/pull/1835>)
- Add explanation for XAI & minor doc fixes (<https://github.com/openvinotoolkit/training_extensions/pull/1923>)
- Refactor phase#1: MPA modules

### Bug fixes

- Handle unpickable update_progress_callback (<https://github.com/openvinotoolkit/training_extensions/pull/1892>)
- Dataset Adapter: Avoid duplicated annotation and permit empty image (<https://github.com/openvinotoolkit/training_extensions/pull/1873>)
- Arrange scale between bbox preds and bbox targets in ATSS (<https://github.com/openvinotoolkit/training_extensions/pull/1880>)
- Fix label mismatch of evaluation and validation with large dataset in semantic segmentation (<https://github.com/openvinotoolkit/training_extensions/pull/1851>)
- Fix packaging errors including cython module build / import issues (<https://github.com/openvinotoolkit/training_extensions/pull/1936>)

### Known issues

- OpenVINO(==2022.3) IR inference is not working well on 2-stage models (e.g. Mask-RCNN) exported from torch==1.13.1
(working well up to torch==1.12.1) (<https://github.com/openvinotoolkit/training_extensions/issues/1906>)

## \[v1.0.1\]

### Enhancements
3 changes: 3 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -2,3 +2,6 @@ recursive-include requirements *
recursive-include otx *.pyx
recursive-include otx *.yaml
recursive-include otx *.json
recursive-exclude otx *.c
graft tests
global-exclude *.py[cod]
26 changes: 13 additions & 13 deletions README.md
Original file line number Diff line number Diff line change
@@ -89,19 +89,19 @@ You can find more details with examples in the [CLI command intro](https://openv

## Updates

### v1.0.0 (1Q23)

- Package Installation via PyPI
- OpenVINO™ Training Extensions installation will be supported via PyPI
- CLI update
- Update `find` command to find configurations of tasks/algorithms
- Introduce `build` command to customize task or model configurations
- Automatic algorihm selection for the `train` command using the given input dataset
- Adaptation of [Datumaro](https://github.com/openvinotoolkit/datumaro) component as a dataset interface
- Integrate hyper-parameter optimizations
- Support action recognition task

### v1.1+ (2Q23)
### v1.1.0 (1Q23)

- Add FP16 IR export support (<https://github.com/openvinotoolkit/training_extensions/pull/1683>)
- Add in-memory caching in dataloader (<https://github.com/openvinotoolkit/training_extensions/pull/1694>)
- Add MoViNet template for action classification (<https://github.com/openvinotoolkit/training_extensions/pull/1742>)
- Add Semi-SL multilabel classification algorithm (<https://github.com/openvinotoolkit/training_extensions/pull/1805>)
- Integrate multi-gpu training for semi-supervised learning and self-supervised learning (<https://github.com/openvinotoolkit/training_extensions/pull/1534>)
- Add train-type parameter to otx train (<https://github.com/openvinotoolkit/training_extensions/pull/1874>)
- Add embedding of inference configuration to IR for classification (<https://github.com/openvinotoolkit/training_extensions/pull/1842>)
- Enable VOC dataset in OTX (<https://github.com/openvinotoolkit/training_extensions/pull/1862>)
- Add mmcls.VisionTransformer backbone support (<https://github.com/openvinotoolkit/training_extensions/pull/1908>)

### v1.2+ (2Q23)

- In planning

Binary file added docs/source/_static/logos/github_icon.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
10 changes: 9 additions & 1 deletion docs/source/conf.py
Original file line number Diff line number Diff line change
@@ -52,7 +52,15 @@
"logo": {
"image_light": 'logos/otx-logo.png',
"image_dark": 'logos/otx-logo.png',
}
},
"icon_links": [
{
"name": "GitHub",
"url": "https://github.com/openvinotoolkit/training_extensions",
"icon": "_static/logos/github_icon.png",
"type": "local",
},
],
}
html_css_files = [
'css/custom.css',
Original file line number Diff line number Diff line change
@@ -9,3 +9,4 @@ Additional Features
models_optimization
hpo
auto_configuration
xai
95 changes: 95 additions & 0 deletions docs/source/guide/explanation/additional_features/xai.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
Explainable AI (XAI)
====================

**Explainable AI (XAI)** is a field of research that aims to make machine learning models more transparent and interpretable to humans.
The goal is to help users understand how and why AI systems make decisions and provide insight into their inner workings. It allows us to detect, analyze, and prevent common mistakes, for example, when the model uses irrelevant features to make a prediction.
XAI can help to build trust in AI, make sure that the model is safe for development and increase its adoption in various domains.

Most XAI methods generate **saliency maps** as a result. Saliency map is a visual representation, suitable for human comprehension, that highlights the most important parts of the image from the model point of view.
It looks like a heatmap, where warm-colored areas represent the areas with main focus.


.. figure:: ../../../../utils/images/xai_example.jpg
:width: 600
:alt: this image shows the result of XAI algorithm

These images are taken from `D-RISE paper <https://arxiv.org/abs/2006.03204>`_.


We can generate saliency maps for a certain model that was trained in OpenVINO™ Training Extensions, using ``otx explain`` command line. Learn more about its usage in :doc:`../../tutorials/base/explain` tutorial.

*********************************
XAI algorithms for classification
*********************************

.. image:: ../../../../utils/images/xai_cls.jpg
:width: 600
:align: center
:alt: this image shows the comparison of XAI classification algorithms


For classification networks these algorithms are used to generate saliency maps:

- **Activation Map​** - this is the most basic and naive approach. It takes the outputs of the model's feature extractor (backbone) and averages it in channel dimension. The results highly rely on the backbone and ignore neck and head computations. Basically, it gives a relatively good and fast result.

- `Eigen-Cam <https://arxiv.org/abs/2008.00299​>`_ uses Principal Component Analysis (PCA). It returns the first principal component of the feature extractor output, which most of the time corresponds to the dominant object. The results highly rely on the backbone as well and ignore neck and head computations.

- `Recipro-CAM​ <https://arxiv.org/pdf/2209.14074>`_ uses Class Activation Mapping (CAM) to weigh the activation map for each class, so it can generate different saliency per class. Recipro-CAM is a fast gradient-free Reciprocal CAM method. The method involves spatially masking the extracted feature maps to exploit the correlation between activation maps and network predictions for target classes.


Below we show the comparison of described algorithms. ``Access to the model internal state`` means the necessity to modify the model's outputs and dump inner features.
``Per-class explanation support`` means generation different saliency maps for different classes.

+-------------------------------------------+----------------+----------------+-------------------------------------------------------------------------+
| Classification algorithm | Activation Map | Eigen-Cam | Recipro-CAM |
+===========================================+================+================+=========================================================================+
| Need access to model internal state | Yes | Yes | Yes |
+-------------------------------------------+----------------+----------------+-------------------------------------------------------------------------+
| Gradient-free | Yes | Yes | Yes |
+-------------------------------------------+----------------+----------------+-------------------------------------------------------------------------+
| Single-shot | Yes | Yes | No (re-infer neck + head H*W times, where HxW – feature map size) |
+-------------------------------------------+----------------+----------------+-------------------------------------------------------------------------+
| Per-class explanation support | No | No | Yes |
+-------------------------------------------+----------------+----------------+-------------------------------------------------------------------------+
| Execution speed | Fast | Fast | Medium |
+-------------------------------------------+----------------+----------------+-------------------------------------------------------------------------+


****************************
XAI algorithms for detection
****************************

For detection networks these algorithms are used to generate saliency maps:

- **Activation Map​** - the same approach as for classification networks, which uses the outputs from feature extractor. This is an algorithm is used to generate saliency maps for two-stage detectors.

- **DetClassProbabilityMap** - this approach takes the raw classification head output and uses class probability maps to calculate regions of interest for each class. So, it creates different salience maps for each class. This algorithm is implemented for single-stage detectors only.

.. image:: ../../../../utils/images/xai_det.jpg
:width: 600
:align: center
:alt: this image shows the detailed description of XAI detection algorithm


The main limitation of this method is that, due to training loss design of most single-stage detectors, activation values drift towards the center of the object while propagating through the network.
This prevents from getting clear explanation in the input image space using intermediate activations.

Below we show the comparison of described algorithms. ``Access to the model internal state`` means the necessity to modify the model's outputs and dump inner features.
``Per-class explanation support`` means generation different saliency maps for different classes. ``Per-box explanation support`` means generation standalone saliency maps for each detected prediction.


+-------------------------------------------+----------------------------+--------------------------------------------+
| Detection algorithm | Activation Map | DetClassProbabilityMap |
+===========================================+============================+============================================+
| Need access to model internal state | Yes | Yes |
+-------------------------------------------+----------------------------+--------------------------------------------+
| Gradient-free | Yes | Yes |
+-------------------------------------------+----------------------------+--------------------------------------------+
| Single-shot | Yes | Yes |
+-------------------------------------------+----------------------------+--------------------------------------------+
| Per-class explanation support | No | Yes |
+-------------------------------------------+----------------------------+--------------------------------------------+
| Per-box explanation support | No | No |
+-------------------------------------------+----------------------------+--------------------------------------------+
| Execution speed | Fast | Fast |
+-------------------------------------------+----------------------------+--------------------------------------------+
Loading