Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add openvino interpreter samples #159

Merged
merged 7 commits into from
Mar 17, 2021
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 40 additions & 32 deletions datumaro/plugins/openvino/README.md
Original file line number Diff line number Diff line change
@@ -1,36 +1,43 @@
# OpenVINO Inference Interpreter
Interpreter samples to parse OpenVINO inference outputs.
# OpenVINO Inference Interpreter
Interpreter samples to parse OpenVINO inference outputs.

## Models supported from interpreter samples
There are 5 SSD and 1 image classification interpreter samples.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
There are 5 SSD and 1 image classification interpreter samples.
There are detection and image classification examples.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


- 5 SSD interpreter samples support the following models.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- 5 SSD interpreter samples support the following models.
- Detection (SSD-based)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

- Intel Pre-trained Models > Object Detection
- [face-detection-0200](https://docs.openvinotoolkit.org/latest/omz_models_intel_face_detection_0200_description_face_detection_0200.html)
- [face-detection-0202](https://docs.openvinotoolkit.org/latest/omz_models_intel_face_detection_0202_description_face_detection_0202.html)
- [face-detection-0204](https://docs.openvinotoolkit.org/latest/omz_models_intel_face_detection_0204_description_face_detection_0204.html)
- [person-detection-0200](https://docs.openvinotoolkit.org/latest/omz_models_intel_person_detection_0200_description_person_detection_0200.html)
- [person-detection-0201](https://docs.openvinotoolkit.org/latest/omz_models_intel_person_detection_0201_description_person_detection_0201.html)
- [person-detection-0202](https://docs.openvinotoolkit.org/latest/omz_models_intel_person_detection_0202_description_person_detection_0202.html)
- [person-vehicle-bike-detection-2000](https://docs.openvinotoolkit.org/latest/omz_models_intel_person_vehicle_bike_detection_2000_description_person_vehicle_bike_detection_2000.html)
- [person-vehicle-bike-detection-2001](https://docs.openvinotoolkit.org/latest/omz_models_intel_person_vehicle_bike_detection_2001_description_person_vehicle_bike_detection_2001.html)
- [person-vehicle-bike-detection-2002](https://docs.openvinotoolkit.org/latest/omz_models_intel_person_vehicle_bike_detection_2002_description_person_vehicle_bike_detection_2002.html)
- [vehicle-detection-0200](https://docs.openvinotoolkit.org/latest/omz_models_intel_vehicle_detection_0200_description_vehicle_detection_0200.html)
- [vehicle-detection-0201](https://docs.openvinotoolkit.org/latest/omz_models_intel_vehicle_detection_0201_description_vehicle_detection_0201.html)
- [vehicle-detection-0202](https://docs.openvinotoolkit.org/latest/omz_models_intel_vehicle_detection_0202_description_vehicle_detection_0202.html)

- Public Pre-Trained Models(OMZ) > Object Detection
- [ssd_mobilenet_v1_coco](https://docs.openvinotoolkit.org/latest/omz_models_public_ssd_mobilenet_v1_coco_ssd_mobilenet_v1_coco.html)
- [ssd_mobilenet_v2_coco](https://docs.openvinotoolkit.org/latest/omz_models_public_ssd_mobilenet_v2_coco_ssd_mobilenet_v2_coco.html)

- Intel Pre-trained Models
- Object Detection
- face-detection-0200 (https://docs.openvinotoolkit.org/latest/omz_models_intel_face_detection_0200_description_face_detection_0200.html)
- face-detection-0202 (https://docs.openvinotoolkit.org/latest/omz_models_intel_face_detection_0202_description_face_detection_0202.html)
- face-detection-0204 (https://docs.openvinotoolkit.org/latest/omz_models_intel_face_detection_0204_description_face_detection_0204.html)
- person-detection-0200 (https://docs.openvinotoolkit.org/latest/omz_models_intel_person_detection_0200_description_person_detection_0200.html)
- person-detection-0201 (https://docs.openvinotoolkit.org/latest/omz_models_intel_person_detection_0201_description_person_detection_0201.html)
- person-detection-0202 (https://docs.openvinotoolkit.org/latest/omz_models_intel_person_detection_0202_description_person_detection_0202.html)
- person-vehicle-bike-detection-2000 (https://docs.openvinotoolkit.org/latest/omz_models_intel_person_vehicle_bike_detection_2000_description_person_vehicle_bike_detection_2000.html)
- person-vehicle-bike-detection-2001 (https://docs.openvinotoolkit.org/latest/omz_models_intel_person_vehicle_bike_detection_2001_description_person_vehicle_bike_detection_2001.html)
- person-vehicle-bike-detection-2002 (https://docs.openvinotoolkit.org/latest/omz_models_intel_person_vehicle_bike_detection_2002_description_person_vehicle_bike_detection_2002.html)
- vehicle-detection-0200 (https://docs.openvinotoolkit.org/latest/omz_models_intel_vehicle_detection_0200_description_vehicle_detection_0200.html)
- vehicle-detection-0201 (https://docs.openvinotoolkit.org/latest/omz_models_intel_vehicle_detection_0201_description_vehicle_detection_0201.html)
- vehicle-detection-0202 (https://docs.openvinotoolkit.org/latest/omz_models_intel_vehicle_detection_0202_description_vehicle_detection_0202.html)
- Image classification interpreter sample supports the following model.
- Public Pre-Trained Models(OMZ) > Classification
- [mobilenet-v2-pytorch](https://docs.openvinotoolkit.org/latest/omz_models_public_mobilenet_v2_pytorch_mobilenet_v2_pytorch.html)

- Public Pre-Trained Models(OMZ)
- Classification
- mobilenet-v2-pytorch (https://docs.openvinotoolkit.org/latest/omz_models_public_mobilenet_v2_pytorch_mobilenet_v2_pytorch.html)
You can find more OpenVINO™ Trained Models [here](https://docs.openvinotoolkit.org/latest/omz_models_intel_index.html)
To run the inference with OpenVINO™, the model format should be Intermediate Representation(IR).
For the Caffe/TensorFlow/MXNet/Kaldi/ONNX models, please see the [Model Conversion Instruction](https://docs.openvinotoolkit.org/latest/openvino_docs_MO_DG_prepare_model_convert_model_Converting_Model.html)

- Object Detection
- ssd_mobilenet_v1_coco (https://docs.openvinotoolkit.org/latest/omz_models_public_ssd_mobilenet_v1_coco_ssd_mobilenet_v1_coco.html)
- ssd_mobilenet_v2_coco (https://docs.openvinotoolkit.org/latest/omz_models_public_ssd_mobilenet_v2_coco_ssd_mobilenet_v2_coco.html)
You need to implement your own interpreter samples to support the other OpenVINO™ Trained Models.

## Model download
- Prerequisites
- OpenVINO (To install OpenVINO, please see https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html)
- OpenVINO models (To download OpenVINO models, please go https://docs.openvinotoolkit.org/latest/omz_tools_downloader_README.html)
- VOCdevkit dataset (To download VOC2012 dataset, please go http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit)
- OpenVINO (To install OpenVINO, please see the [OpenVINO™ Installation Instruction](https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- OpenVINO™ (To install OpenVINO™, please see the [OpenVINO™ Installation Instruction](https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html)
- OpenVINO™ (To install OpenVINO™, please see the [OpenVINO™ Installation Instruction](https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

- OpenVINO models (To download OpenVINO models, please see the [Model Downloader Instruction] https://docs.openvinotoolkit.org/latest/omz_tools_downloader_README.html)
Copy link
Contributor

@zhiltsov-max zhiltsov-max Mar 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing )

Suggested change
- OpenVINO™ models (To download OpenVINO™ models, please see the [Model Downloader Instruction] https://docs.openvinotoolkit.org/latest/omz_tools_downloader_README.html)
- OpenVINO™ models (To download OpenVINO™ models, please see the [Model Downloader Instruction] https://docs.openvinotoolkit.org/latest/omz_tools_downloader_README.html))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

- PASCAL VOC 2012 dataset (To download VOC 2012 dataset, please go [VOC2012 download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit)
Copy link
Contributor

@zhiltsov-max zhiltsov-max Mar 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing )

Suggested change
- PASCAL VOC 2012 dataset (To download VOC 2012 dataset, please go [VOC2012 download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit)
- PASCAL VOC 2012 dataset (To download VOC 2012 dataset, please go [VOC2012 download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


```bash
# cd <openvino_dir>/deployment_tools/open_model_zoo/tools/downloader
@@ -43,30 +50,31 @@ Interpreter samples to parse OpenVINO inference outputs.

## Model inference
- Prerequisites:
- Put the models downloaded in <datumaro_root_dir>/datumaro/plugins/openvino/models
- Put the VOCdevkit dataset downloaded in <datumaro_root_dir>
- OpenVINO™ (To install OpenVINO™, please see the [OpenVINO™ Installation Instruction](https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html)
Copy link
Contributor

@zhiltsov-max zhiltsov-max Mar 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing ) (+3 next)

Suggested change
- OpenVINO™ (To install OpenVINO™, please see the [OpenVINO™ Installation Instruction](https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html)
- OpenVINO™ (To install OpenVINO™, please see the [OpenVINO™ Installation Instruction](https://docs.openvinotoolkit.org/latest/openvino_docs_install_guides_installing_openvino_linux.html))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

- Datumaro (To install Datumaro, please see the [User Manual](docs/user_manual.md)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- Datumaro (To install Datumaro, please see the [User Manual](docs/user_manual.md)
- Datumaro (To install Datumaro, please see the [User Manual](docs/user_manual.md))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

- OpenVINO™ models (To download OpenVINO™ models, please see the [Model Downloader Instruction] https://docs.openvinotoolkit.org/latest/omz_tools_downloader_README.html)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- OpenVINO™ models (To download OpenVINO™ models, please see the [Model Downloader Instruction] https://docs.openvinotoolkit.org/latest/omz_tools_downloader_README.html)
- OpenVINO™ models (To download OpenVINO™ models, please see the [Model Downloader Instruction] (https://docs.openvinotoolkit.org/latest/omz_tools_downloader_README.html))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

- PASCAL VOC 2012 dataset (To download VOC 2012 dataset, please go [VOC2012 download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- PASCAL VOC 2012 dataset (To download VOC 2012 dataset, please go [VOC2012 download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit)
- PASCAL VOC 2012 dataset (To download VOC 2012 dataset, please go [VOC2012 download](http://host.robots.ox.ac.uk/pascal/VOC/voc2012/#devkit))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


- To run the inference with OpenVINO models and the interpreter samples, please follow the instructions below.
- To run the inference with OpenVINO models and the interpreter samples, please follow the instructions below.

```bash
# . <openvino_dir>/bin/setupvars.sh
# cd <datumaro_root_dir>/datumaro/plugins/openvino/
# <openvino_dir>/bin/setupvars.sh
# datum create -o <proj_dir>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why these commands have different indents?

# datum model add -l <launcher> -p <proj_dir> --copy -- -d <path_to_xml> -w <path_to_bin> -i <path_to_interpreter_script>
# datum add path -p <proj_dir> -f <format> <path_to_dataset>
# datum model run -p <proj_dir> -m model-0
#
# Examples
# Detection> ssd_mobilenet_v2_coco
. /opt/intel/openvino_2021/bin/setupvars.sh
/opt/intel/openvino_2021/bin/setupvars.sh
Copy link
Contributor

@zhiltsov-max zhiltsov-max Mar 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

. or source in the beginning is required

Suggested change
/opt/intel/openvino_2021/bin/setupvars.sh
. /opt/intel/openvino/bin/setupvars.sh

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we install openvino 2021.r1, the default path is /opt/intel/openvino_2021, not /opt/intel/openvino. The default path like openvino_2021 had been using since openvino 2021.r1. Do you still want me to change it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to openvino because we don't know user which Openvino version will install. openvino is more general. So I changed it.

cd /home/cvalgo/workspace/datumaro/datumaro/plugins/openvino/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clean /home/cvalgo/workspace. Make the examples as simple and ready to use, as possible.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cd /home/cvalgo/workspace/datumaro/datumaro/plugins/openvino/
cd datumaro/plugins/openvino/

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

datum create -o proj_ssd_mobilenet_v2_coco_detection
datum model add -l openvino -p proj_ssd_mobilenet_v2_coco_detection --copy -- --output-layers=do_ExpandDims_conf/sigmoid -d model/ssd_mobilenet_v2_coco.xml -w model/ssd_mobilenet_v2_coco.bin -i samples/ssd_mobilenet_coco_detection_interp.py
datum add path -p proj_ssd_mobilenet_v2_coco_detection -f voc /home/cvalgo/workspace/datumaro/VOCdevkit
datum model run -p proj_ssd_mobilenet_v2_coco_detection -m model-0

# Classification> mobilenet-v2-pytorch
. /opt/intel/openvino_2021/bin/setupvars.sh
/opt/intel/openvino_2021/bin/setupvars.sh
cd /home/cvalgo/workspace/datumaro/datumaro/plugins/openvino/
datum create -o proj_mobilenet_v2_classification
datum model add -l openvino -p proj_mobilenet_v2_classification --copy -- -d model/mobilenet-v2-pytorch.xml -w model/mobilenet-v2-pytorch.bin -i samples/mobilenet_v2_pytorch_interp.py
Original file line number Diff line number Diff line change
@@ -18,7 +18,7 @@ def process_outputs(inputs, outputs):
image_results = []
output = softmax(output).tolist()
label = output.index(max(output))
image_results.append(Label(label=label, attributes={"confidences": output}))
image_results.append(Label(label=label, attributes={"scores": output}))

results.append(image_results[:])

@@ -30,9 +30,7 @@ def get_categories():

label_categories = LabelCategories()

with open(
"samples/imagenet.class", "r"
) as file:
with open("samples/imagenet.class", "r") as file:
for line in file.readlines():
label = line.strip()
label_categories.add(label)
14 changes: 1 addition & 13 deletions datumaro/plugins/openvino/samples/ssd_face_detection_interp.py
Original file line number Diff line number Diff line change
@@ -65,19 +65,7 @@ def process_outputs(inputs, outputs):
w = min(int(det[5] * input_width - x), input_width)
h = min(int(det[6] * input_height - y), input_height)

image_results.append(
Bbox(
x,
y,
w,
h,
label=label,
attributes={
"confidence": conf,
"confidences": list(map(float, det_confs)),
},
)
)
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
image_results.append(Bbox(x, y, w, h, label=label,
attributes={ 'score': conf, 'scores': list(map(float, det_confs)) }
))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


results.append(image_results)

Original file line number Diff line number Diff line change
@@ -68,19 +68,7 @@ def process_outputs(inputs, outputs):
w = min(int(det[5] * input_width - x), input_width)
h = min(int(det[6] * input_height - y), input_height)

image_results.append(
Bbox(
x,
y,
w,
h,
label=label,
attributes={
"confidence": conf,
"confidences": list(map(float, det_confs)),
},
)
)
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
image_results.append(Bbox(x, y, w, h, label=label,
attributes={ 'score': conf, 'scores': list(map(float, det_confs)) }
))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


results.append(image_results)

@@ -92,9 +80,7 @@ def get_categories():

label_categories = LabelCategories()

with open(
"samples/coco.class", "r"
) as file:
with open("samples/coco.class", "r") as file:
for line in file.readlines():
label = line.strip()
label_categories.add(label)
Original file line number Diff line number Diff line change
@@ -65,19 +65,7 @@ def process_outputs(inputs, outputs):
w = min(int(det[5] * input_width - x), input_width)
h = min(int(det[6] * input_height - y), input_height)

image_results.append(
Bbox(
x,
y,
w,
h,
label=label,
attributes={
"confidence": conf,
"confidences": list(map(float, det_confs)),
},
)
)
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
image_results.append(Bbox(x, y, w, h, label=label,
attributes={ 'score': conf, 'scores': list(map(float, det_confs)) }
))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


results.append(image_results)

Original file line number Diff line number Diff line change
@@ -65,19 +65,7 @@ def process_outputs(inputs, outputs):
w = min(int(det[5] * input_width - x), input_width)
h = min(int(det[6] * input_height - y), input_height)

image_results.append(
Bbox(
x,
y,
w,
h,
label=label,
attributes={
"confidence": conf,
"confidences": list(map(float, det_confs)),
},
)
)
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
image_results.append(Bbox(x, y, w, h, label=label,
attributes={ 'score': conf, 'scores': list(map(float, det_confs)) }
))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


results.append(image_results)

Original file line number Diff line number Diff line change
@@ -65,19 +65,7 @@ def process_outputs(inputs, outputs):
w = min(int(det[5] * input_width - x), input_width)
h = min(int(det[6] * input_height - y), input_height)

image_results.append(
Bbox(
x,
y,
w,
h,
label=label,
attributes={
"confidence": conf,
"confidences": list(map(float, det_confs)),
},
)
)
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
image_results.append(Bbox(x, y, w, h, label=label, attributes={"score": conf, "scores": list(map(float, det_confs)),},))
image_results.append(Bbox(x, y, w, h, label=label,
attributes={ 'score': conf, 'scores': list(map(float, det_confs)) }
))

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


results.append(image_results)

6 changes: 3 additions & 3 deletions datumaro/plugins/sampler/sampler.py
Original file line number Diff line number Diff line change
@@ -123,10 +123,10 @@ def _load_inference_from_subset(extractor, subset_name):
raise Exception(f"Item {item.id} does not have annotations")

for annotation in item.annotations:
if 'confidences' not in annotation.attributes:
if 'scores' not in annotation.attributes:
raise Exception(f"Item {item.id} - an annotation "
"does not have 'confidences' attribute")
probs = annotation.attributes['confidences']
"does not have 'scores' attribute")
probs = annotation.attributes['scores']

infer_df['ImageID'].append(item.id)

1 change: 1 addition & 0 deletions docs/user_manual.md
Original file line number Diff line number Diff line change
@@ -850,6 +850,7 @@ datum model add \
```

Interpretation script for an OpenVINO detection model (`convert.py`):
You can find OpenVINO™ model interpreter samples in datumaro/plugins/openvino/samples. [Instruction](datumaro/plugins/openvino/README.md)

``` python
from datumaro.components.extractor import *
24 changes: 12 additions & 12 deletions tests/test_sampler.py
Original file line number Diff line number Diff line change
@@ -42,7 +42,7 @@ def _get_probs(out_range=False):
return probs

def _generate_classification_dataset(self, config, subset=None,
empty_confidences=False, out_range=False, no_attr=False, no_img=False):
empty_scores=False, out_range=False, no_attr=False, no_img=False):
probs = self._get_probs(out_range)
if subset is None:
self.subset = ["train", "val", "test"]
@@ -56,11 +56,11 @@ def _generate_classification_dataset(self, config, subset=None,
num_item = config[label]
label_cat.add(label, attributes=None)
for _ in range(num_item):
confidences = probs[idx]
scores = probs[idx]
idx += 1
if empty_confidences:
confidences = []
attr = {"confidences": confidences}
if empty_scores:
scores = []
attr = {"scores": scores}
if no_attr:
attr = {}
img = Image(path=f"test/dataset/{idx}.jpg", size=(90, 90))
@@ -341,7 +341,7 @@ def test_sampler_gives_error(self):
data_df["ImagePath"].append(data.image.path)

for annotation in data.annotations:
probs = annotation.attributes["confidences"]
probs = annotation.attributes["scores"]
infer_df["ImageID"].append(data.id)

for prob_idx, prob in enumerate(probs):
@@ -368,7 +368,7 @@ def test_sampler_gives_error(self):
data_df["ImagePath"].append(data.image.path)

for annotation in data.annotations:
probs = annotation.attributes["confidences"]
probs = annotation.attributes["scores"]

for prob_idx, prob in enumerate(probs):
infer_df[f"ClassProbability{prob_idx+1}"].append(prob)
@@ -401,14 +401,14 @@ def test_sampler_get_invalid_data(self):
result = iter(result)
next(result)

with self.subTest("Dataset without confidences (Probability)"):
with self.subTest("Dataset without Scores (Probability)"):
config = {
"label1": 10,
"label2": 10,
"label3": 10,
}

source = self._generate_classification_dataset(config, empty_confidences=True)
source = self._generate_classification_dataset(config, empty_scores=True)
with self.assertRaisesRegex(Exception, "ClassProbability"):
result = Sampler(
source,
@@ -431,7 +431,7 @@ def test_sampler_get_invalid_data(self):
}

source = self._generate_classification_dataset(
config, empty_confidences=False, out_range=True
config, empty_scores=False, out_range=True
)
with self.assertRaisesRegex(Exception, "Invalid data"):
result = Sampler(
@@ -447,15 +447,15 @@ def test_sampler_get_invalid_data(self):
result = iter(result)
next(result)

with self.subTest("No confidences Attribute Data"):
with self.subTest("No Scores Attribute Data"):
config = {
"label1": 10,
"label2": 10,
"label3": 10,
}

source = self._generate_classification_dataset(config, no_attr=True)
with self.assertRaisesRegex(Exception, "does not have 'confidences'"):
with self.assertRaisesRegex(Exception, "does not have 'scores'"):
result = Sampler(
source,
algorithm="entropy",