Skip to content

Commit

Permalink
Returned OV 2024.3 references for OV 2024.5 in Weight Compression tes…
Browse files Browse the repository at this point in the history
…ts (#3065)

### Changes

Revert of #2954
Use [references for the OV
2024.3](https://github.com/ljaljushkin/nncf_pytorch/blob/873fbb80ab07bd41a3b1a2e21f68bc1a7a394810/tests/post_training/data/wc_reference_data.yaml)
when run conformance tests with OV 2024.5.

### Reason for changes

According to the CPU team: 
> 1) In 24.4 time frame, RMS impl has been enabled
(openvinotoolkit/openvino#26147), which
introduce some accuracy change due to vrsqrtss which is done by
approximation.
> 2) In 24.5 time frame, fix has been applied
(openvinotoolkit/openvino#26817) to use vsqrtss
instead. With that we can see the result can be covered to that prior to
openvinotoolkit/openvino#26147
> With that (in 24.5), the accuracy result is recovered to that prior to
RMS impl openvinotoolkit/openvino#26147. Let's
say that 24.4 may has some accuracy issue in some case. 24.5 fix that
issue.

### Related tickets

156605
156776
156237
151260

### Tests

- [x] 47 build of
weekly/job/openvino-nightly/job/post_training_weight_compression

![image](https://github.com/user-attachments/assets/10271e74-4e0b-4506-917a-306481b500e3)
- [x] 48 build for xfail tests

![image](https://github.com/user-attachments/assets/dc5a9424-35fa-4015-965a-ca08fa46e4b7)
  • Loading branch information
ljaljushkin authored Nov 11, 2024
1 parent 2536849 commit 68983a7
Showing 1 changed file with 27 additions and 1 deletion.
28 changes: 27 additions & 1 deletion tests/post_training/data/wc_reference_data_2024.5.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,34 @@
tinyllama_NF4_scale_estimation_stateful_per_channel_backend_OV:
metric_value: 0.87132
metric_value: 0.88663
num_int4: 11
num_int8: 290
metrics_xfail_reason: "Issue-148819"
tinyllama_int4_data_free_backend_TORCH:
metric_value: 0.73873
num_int4: 114
num_int8: 84
tinyllama_awq_backup_mode_none_backend_OV:
metric_value: 0.84783
num_int4: 208
num_int8: 0
tinyllama_data_aware_awq_scale_estimation_backend_OV:
metric_value: 0.85502
num_int4: 94
num_int8: 124
tinyllama_data_aware_awq_scale_estimation_stateful_backend_OV:
metric_value: 0.85502
num_int4: 94
num_int8: 124
tinyllama_data_aware_awq_stateful_backend_OV:
metric_value: 0.85616
num_int4: 94
num_int8: 124
tinyllama_data_aware_gptq_scale_estimation_stateful_backend_OV:
metric_value: 0.86503
num_int4: 94
num_int8: 124
metrics_xfail_reason: "Issue-148819"
tinyllama_scale_estimation_per_channel_backend_OV:
metric_value: 0.81389
num_int4: 188
num_int8: 124

0 comments on commit 68983a7

Please sign in to comment.