Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conformance test for weight compression #2425

Merged

Conversation

ljaljushkin
Copy link
Contributor

@ljaljushkin ljaljushkin commented Jan 25, 2024

Changes

Added tests for accuracy and duration of weight compression on tiny-llama-1B

Reason for changes

catch bugs with accuracy (e.g. difference after changing rounding - #2420) and with duration of weight compression

Related tickets

130630

Tests

  • local run of weight compression conformance tests
  • PTQ conformance tests in CI (build 273)
  • CI scripts should be adopted for a new location of reference.yaml
  • html generation should accept new names in the table.

Total execution time for tests w/ or w/o FP32 option is ~20/~40 minutes.
Model download takes extra 20 minutes, but with caching models on share #2427
the download won't happen.

image

@github-actions github-actions bot added documentation Improvements or additions to documentation NNCF PTQ Pull requests that updates NNCF PTQ labels Jan 25, 2024
Copy link

codecov bot commented Jan 25, 2024

Codecov Report

Attention: 1 lines in your changes are missing coverage. Please review.

Comparison is base (c0e51f8) 91.02% compared to head (837d695) 84.86%.
Report is 8 commits behind head on develop.

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #2425      +/-   ##
===========================================
- Coverage    91.02%   84.86%   -6.17%     
===========================================
  Files          496      497       +1     
  Lines        45079    45222     +143     
===========================================
- Hits         41035    38379    -2656     
- Misses        4044     6843    +2799     
Files Coverage Δ
...n/algorithms/weight_compression/mixed_precision.py 46.66% <0.00%> (-51.43%) ⬇️

... and 64 files with indirect coverage changes

Flag Coverage Δ
COMMON 44.23% <0.00%> (-0.37%) ⬇️
ONNX 35.01% <0.00%> (-0.08%) ⬇️
OPENVINO ∅ <ø> (∅)
TENSORFLOW 30.19% <0.00%> (-0.07%) ⬇️
TORCH 66.16% <0.00%> (+2.48%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
common 92.71% <ø> (-0.66%) ⬇️
torch 93.38% <ø> (+0.20%) ⬆️
tensorflow 93.99% <ø> (ø)
onnx 98.01% <ø> (ø)
openvino 26.03% <ø> (-68.42%) ⬇️
ptq 68.40% <0.00%> (-20.68%) ⬇️

@github-actions github-actions bot removed the documentation Improvements or additions to documentation label Jan 31, 2024
@ljaljushkin ljaljushkin marked this pull request as ready for review January 31, 2024 12:14
@ljaljushkin ljaljushkin requested a review from a team as a code owner January 31, 2024 12:14
@ljaljushkin
Copy link
Contributor Author

job 4
image
report looks good.
accuracy results for data-aware methods are not stable. will investigate and fix later

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Feb 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation NNCF PTQ Pull requests that updates NNCF PTQ
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants