Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update e2e tests to use convert_model #2152

Merged

Conversation

AlexanderDokuchaev
Copy link
Collaborator

@AlexanderDokuchaev AlexanderDokuchaev commented Sep 21, 2023

Changes

  • Update torch examples, added argument --export-model-path instead of --to-onnx that select output type by suffix '.xml' or '.onnx'.
  • Update tests to use Command to run examples.
  • Remove tests of q_dq export path.
  • Remove code to make html report, now metrics dumps in results.csv.
  • Added "build url" in results.csv.
  • Actualize reference metrics.
  • Move normalization of input to accuracy cheker configs.
  • Use target_ov and target_pt to define reference metrics for backends.
  • Skip test unet_mapillary_int8 and unet_mapillary_magnitude_sparsity_int8 models by 123448.
  • Skip resnet18_imagenet_binarization_dorefa model by 22543.

Updates in CI:

  • Update script to report final e2e_result.html for e2e tests.
  • Now if any test falls trigger_job will be fails.

After merge requires update ci-pipelines.

Related tickets

117885

Tests

tests/torch/test_sota_checkpoints.py

TODO

@github-actions github-actions bot added documentation Improvements or additions to documentation NNCF PT Pull requests that updates NNCF PyTorch labels Sep 21, 2023
@codecov
Copy link

codecov bot commented Sep 21, 2023

Codecov Report

Merging #2152 (ae3f3cd) into develop (b4b2e19) will increase coverage by 53.92%.
Report is 13 commits behind head on develop.
The diff coverage is n/a.

Additional details and impacted files
@@             Coverage Diff              @@
##           develop    #2152       +/-   ##
============================================
+ Coverage    36.60%   90.53%   +53.92%     
============================================
  Files          485      486        +1     
  Lines        43722    43862      +140     
============================================
+ Hits         16006    39709    +23703     
+ Misses       27716     4153    -23563     
Flag Coverage Δ
COMMON 15.82% <ø> (+<0.01%) ⬆️
ONNX 33.83% <ø> (+0.01%) ⬆️
OPENVINO 38.66% <ø> (?)
TENSORFLOW 30.02% <ø> (?)
TORCH 62.68% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 347 files with indirect coverage changes

@AlexanderDokuchaev AlexanderDokuchaev marked this pull request as ready for review October 24, 2023 00:49
@AlexanderDokuchaev AlexanderDokuchaev requested a review from a team as a code owner October 24, 2023 00:49
Copy link
Contributor

@vshampor vshampor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to see usage of dataclasses, test fixtures and type hints - don't forget to take opportunities to set up type hints even if they weren't therer before.

examples/torch/common/argparser.py Outdated Show resolved Hide resolved
examples/torch/common/argparser.py Outdated Show resolved Hide resolved
examples/torch/common/export.py Show resolved Hide resolved
examples/torch/common/export.py Outdated Show resolved Hide resolved
examples/torch/common/export.py Outdated Show resolved Hide resolved
tests/torch/test_sota_checkpoints.py Outdated Show resolved Hide resolved
tests/torch/test_sota_checkpoints.py Outdated Show resolved Hide resolved
tests/torch/test_sota_checkpoints.py Outdated Show resolved Hide resolved
tests/torch/sota_checkpoints_eval.json Outdated Show resolved Hide resolved
tests/torch/sota_checkpoints_eval.json Outdated Show resolved Hide resolved
@AlexanderDokuchaev
Copy link
Collaborator Author

view/all/job/ci-maintenance/job/pt_e2e_tests/job/NNCF/job/nightly/job/TriggerBetta/55
view/all/job/ci-maintenance/job/pt_e2e_tests/job/NNCF/job/nightly/job/TriggerBetta/57 (with fixed reference)

examples/torch/common/export.py Outdated Show resolved Hide resolved
tests/torch/test_sota_checkpoints.py Show resolved Hide resolved
Copy link
Contributor

@alexsu52 alexsu52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you provide summary results and builds?

@AlexanderDokuchaev
Copy link
Collaborator Author

@alexsu52 ci-maintenance/job/pt_e2e_tests/job/NNCF/job/nightly/job/TriggerBetta/75/

@alexsu52 alexsu52 dismissed vshampor’s stale review November 29, 2023 10:48

All comments were reviewed

Copy link
Contributor

@alexsu52 alexsu52 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff!!!

@KodiaqQ KodiaqQ merged commit ce061bb into openvinotoolkit:develop Nov 30, 2023
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation NNCF PT Pull requests that updates NNCF PyTorch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants