Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use convert_model in tests #2102

Merged

Conversation

AlexanderDokuchaev
Copy link
Collaborator

@AlexanderDokuchaev AlexanderDokuchaev commented Sep 3, 2023

Changes

Use mo.convert_model instead of converting via onnx in tests.

  • Update version to 2.6.0, mo.convert_model contains check nncf.__version__ >= 2.6.0
  • Add dump fp32 models
  • Disable POT backend for inception_resnet_v2, levit_128, vgg11 in test_quantize_conformance.py (120012)
  • Update metrics of test_quantize_conformance.py
Model 2023.0.1 2023.1.0 Diff
timm/efficientnet_b0_backend_OLD_TORCH 0.74976 0.75538 0.0056
timm/efficientnet_b0_BC_backend_POT 0.76878 0.76872 -0.0001
timm/levit_128_backend_ONNX 0.73082 0.73184 0.0010
timm/mobilenetv2_050_backend_POT 0.64234 0.64362 0.0013
timm/mobilenetv2_050_BC_backend_POT 0.64376 0.6431 -0.0007
timm/mobilenetv3_small_050_backend_POT 0.27062 0.54104 0.2704
timm/swin_base_patch4_window7_224_backend_OLD_TORCH 0.79586 0.81376 0.0179
timm/swin_base_patch4_window7_224_backend_TORCH 0.78026 0.80696 0.0267

Reason for changes

New api in openvino to convert PT models

Related tickets

117885

@AlexanderDokuchaev AlexanderDokuchaev requested a review from a team as a code owner September 3, 2023 21:14
@github-actions github-actions bot added NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PTQ Pull requests that updates NNCF PTQ labels Sep 3, 2023
@codecov
Copy link

codecov bot commented Sep 3, 2023

Codecov Report

Merging #2102 (b2f7947) into develop (19d7260) will increase coverage by 36.16%.
Report is 2 commits behind head on develop.
The diff coverage is 100.00%.

Additional details and impacted files
@@             Coverage Diff              @@
##           develop    #2102       +/-   ##
============================================
+ Coverage         0   36.16%   +36.16%     
============================================
  Files            0      476      +476     
  Lines            0    42494    +42494     
============================================
+ Hits             0    15369    +15369     
- Misses           0    27125    +27125     
Files Changed Coverage Δ
nncf/version.py 100.00% <100.00%> (ø)

... and 475 files with indirect coverage changes

@@ -7,7 +7,7 @@ onnx==1.13.1
onnxruntime==1.14.1
pytest
pytest-cov
openvino-dev==2023.0.1
openvino-dev==2023.1.0.dev20230811
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why was it changed? Do we have any proof that this package is not buggy?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2023.1.0.dev20230811 have some bug that was fixed on current release branch.
But with 2023.0.1 mo.convert_model does not support quantized models.
Until dump new openvino package it's the best option.

@AlexanderDokuchaev AlexanderDokuchaev changed the title convert_model [WIP] convert_model Sep 4, 2023
@AlexanderDokuchaev
Copy link
Collaborator Author

post_training_quantization/149/

@AlexanderDokuchaev
Copy link
Collaborator Author

Update e2e test will be done in another PR.

@AlexanderDokuchaev AlexanderDokuchaev changed the title [WIP] convert_model Use convert_model in tests Sep 15, 2023
@AlexanderDokuchaev
Copy link
Collaborator Author

post_training_quantization/152 - with dump fp32 models

Copy link
Contributor

@alexsu52 alexsu52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about PyTorch and TensorFlow backends?

@@ -9,7 +9,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.

__version__ = "2.5.0"
__version__ = "2.6.0"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@KodiaqQ, I suggest to merge this PR after NNCF 2.6.0 release.

@alexsu52 alexsu52 merged commit 50feea4 into openvinotoolkit:develop Sep 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NNCF OpenVINO Pull requests that updates NNCF OpenVINO NNCF PTQ Pull requests that updates NNCF PTQ
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants