LP Transformation tests use API 2.0 #21677

evkotov · 2023-12-14T19:11:40Z

Details:

use SubgraphBaseTest as the base class
rename Run -> run
add init_input_shapes

Tickets:

111374

...shared/include/low_precision_transformations/concat_with_different_precision_on_children.hpp

.../plugin/shared/src/low_precision_transformations/concat_with_intermediate_transformation.cpp

src/tests/functional/plugin/shared/src/low_precision_transformations/mat_mul_transformation.cpp

...ional/plugin/shared/src/low_precision_transformations/output_layers_concat_multi_channel.cpp

...sses/include/shared_test_classes/base/low_precision_transformations/layer_transformation.hpp

...nctional/shared_test_classes/src/base/low_precision_transformations/layer_transformation.cpp

eshoguli · 2023-12-29T15:29:47Z

@iefode, note, please, in some test cases we need to set rel_threshold more then 100%. Is it expected? We need to do this because current tensor comparision has specific implementation for zero values.

src/tests/functional/plugin/shared/src/low_precision_transformations/add_transformation.cpp

praasz · 2024-01-05T06:56:00Z

src/tests/functional/plugin/shared/src/low_precision_transformations/add_transformation.cpp

+        inputShape2[3] = 1;
+    }
+    init_input_shapes({ inputShape, inputShape2 });
+
    function = ngraph::builder::subgraph::AddFunction::getOriginal(


Don't use ngraph. Check similar usage of ngraph functions in modified files.

@itikhono,
There are helper functions in src/tests/ov_helpers/ov_lpt_models. Should it be fixed ngraph -> ov namespace in this PR or it could be done in next separate PR? There are already 200+ files in this PR

Talked offline. This will be done in a next separate PR since there are already too many files in this PR

created ticket

...l/plugin/shared/src/low_precision_transformations/eliminate_fake_quantize_transformation.cpp

...hared/src/low_precision_transformations/fake_quantize_with_dq_not_optimal_transformation.cpp

...ts/functional/plugin/shared/src/low_precision_transformations/interpolate_transformation.cpp

...nal/plugin/shared/src/low_precision_transformations/mat_mul_with_constant_transformation.cpp

...tests/functional/plugin/shared/src/low_precision_transformations/multiply_transformation.cpp

.../plugin/shared/src/low_precision_transformations/multiply_with_one_parent_transformation.cpp

...functional/plugin/shared/src/low_precision_transformations/recurrent_cell_transformation.cpp

...sts/functional/plugin/shared/src/low_precision_transformations/reduce_max_transformation.cpp

...sses/include/shared_test_classes/base/low_precision_transformations/layer_transformation.hpp

...nctional/shared_test_classes/src/base/low_precision_transformations/layer_transformation.cpp

src/tests/functional/shared_test_classes/src/base/utils/generate_inputs.cpp

src/tests/test_utils/common_test_utils/include/common_test_utils/data_utils.hpp

iefode · 2024-01-09T08:32:45Z

src/tests/test_utils/common_test_utils/include/common_test_utils/data_utils.hpp

+                                            const float min,
+                                            const float max,
+                                            const int seed) {
+    std::mt19937 gen(seed);


Just minor open question: to use such generator for instead of gtest::Random?
@evkotov Have you checked which generator is using in gtest::Random?

the main problem to generate within [min, max] range with float min and max. It is simple to get the way I used. gtest::Random operates uint32 range

Minor for the future: unite two functions into one. Because float numbers could be generate by resolution using

src/tests/test_utils/common_test_utils/src/ov_tensor_utils.cpp

src/tests/functional/shared_test_classes/src/base/utils/generate_inputs.cpp

sbalandi · 2024-01-10T14:11:46Z

@iefode, note, please, in some test cases we need to set rel_threshold more then 100%. Is it expected? We need to do this because current tensor comparision has specific implementation for zero values.

Hi @eshoguli , if you mean, that tests fail in case threshold and max error are equal, this behavior is fixed in PR: #21783 (function less_or_equal in src/tests/test_utils/common_test_utils/src/ov_tensor_utils.cpp ). Also, please, not, that after this PR the meaning of rel_threshold will change and we are checking, that accumulated error is not greater than the sum of the maximum allowed errors for all shapes

…-> ov

.../shared_tests_instances/low_precision_transformations/fuse_subtract_to_fq_transformation.cpp

iefode

In general LGTM. I have left several comments for the future.

iefode · 2024-01-18T07:28:21Z

...sses/include/shared_test_classes/base/low_precision_transformations/layer_transformation.hpp


-    static std::pair<float, float> getQuantizationInterval(const ngraph::element::Type precision);
+    static std::string to_string(const ov::pass::low_precision::LayerTransformation::Params& params);


Minor: so strange naming... What does it mean?

This looks strange in diff. If you open file you can see that getQuantizationInterval renamed to get_quantization_interval

iefode · 2024-01-18T07:29:15Z

...nctional/shared_test_classes/src/base/low_precision_transformations/layer_transformation.cpp

+}
+
+void LayerTransformation::init_input_shapes(const ov::PartialShape& shape) {
+    std::pair<ov::PartialShape, std::vector<ov::Shape>> input_shapes(shape, { shape.to_shape() });


I did not get any response to the comment. Why should it be resolved?

iefode · 2024-01-18T07:30:34Z

src/tests/test_utils/common_test_utils/include/common_test_utils/data_utils.hpp

+                                            const float min,
+                                            const float max,
+                                            const int seed) {
+    std::mt19937 gen(seed);


Minor for the future: unite two functions into one. Because float numbers could be generate by resolution using

jane-intel

No significant comments from my side. You did a great job cleaning out ngraph namespace usage. I'm looking forward for the future clearing of API 1.0 headers and methods.
Please elaborate why some tests are skipped and some tests are given huge tolerance in the description of this PR. It seems like we are losing the coverage just because we are switching namespaces.
Overall, looks ok to me.

...ared_tests_instances/low_precision_transformations/fq_with_dq_not_optimal_transformation.cpp

.../shared_tests_instances/low_precision_transformations/fuse_multiply_to_fq_transformation.cpp

.../shared_tests_instances/low_precision_transformations/fuse_subtract_to_fq_transformation.cpp

jane-intel · 2024-01-18T07:44:58Z

...red/src/low_precision_transformations/pull_reshape_through_dequantization_transformation.cpp

-    // threshold = 0.1f;
+    abs_threshold = 1.0e-3;


is it desirable behavior? did we lose coverage? did we soften the testing?

jane-intel · 2024-01-18T07:46:27Z

...sts/functional/plugin/shared/src/low_precision_transformations/reduce_max_transformation.cpp

@@ -31,12 +30,15 @@ std::string ReduceMaxTransformation::getTestCaseName(const testing::TestParamInf
 }

 void ReduceMaxTransformation::SetUp() {
-    ngraph::element::Type netPrecision;
-    ngraph::PartialShape inputShape;
+    abs_threshold = 1.1;


it looks like a big threshold. could you elaborate what was the original threshold for this test before refactoring.

.../shared_tests_instances/low_precision_transformations/fuse_subtract_to_fq_transformation.cpp

Co-authored-by: Evgenya Nugmanova <[email protected]>

…low_precision_transformations/fuse_multiply_to_fq_transformation.cpp Co-authored-by: Evgenya Nugmanova <[email protected]>

…low_precision_transformations/fuse_subtract_to_fq_transformation.cpp Co-authored-by: Evgenya Nugmanova <[email protected]>

itikhono · 2024-01-18T08:19:23Z

Looks good!
Let's address the remaining comments in the next PR and merge this one today.

evkotov added category: IE Tests OpenVINO Test: plugins and common do not merge do_not_review do_not_merge labels Dec 14, 2023

evkotov self-assigned this Dec 14, 2023

evkotov requested review from a team as code owners December 14, 2023 19:11

github-actions bot added the category: CPU OpenVINO CPU plugin label Dec 14, 2023

evkotov requested review from a team as code owners December 18, 2023 17:31

github-actions bot added the category: GPU OpenVINO GPU plugin label Dec 18, 2023

evkotov removed do not merge do_not_review do_not_merge labels Dec 20, 2023

evkotov requested review from itikhono, eshoguli and iefode December 20, 2023 15:36

evkotov changed the title ~~[NO REVIEW] [NO MERGE] LP Transformation tests use API 2.0~~ LP Transformation tests use API 2.0 Dec 20, 2023

vurusovs self-assigned this Dec 25, 2023

vurusovs suggested changes Dec 25, 2023

View reviewed changes

praasz reviewed Jan 5, 2024

View reviewed changes

iefode self-assigned this Jan 8, 2024

iefode added the conformance label Jan 9, 2024

iefode reviewed Jan 9, 2024

View reviewed changes

evkotov force-pushed the cvs_111374_dev branch from 97fca4a to aa0d490 Compare January 9, 2024 14:55

vurusovs suggested changes Jan 10, 2024

View reviewed changes

src/tests/functional/shared_test_classes/src/base/utils/generate_inputs.cpp Outdated Show resolved Hide resolved

evkotov added 14 commits January 17, 2024 11:21

fix gpu func tests InferenceEngine

5879c86

getTestCaseNameByParams -> get_test_case_name_by_params; some ngraph …

84d8760

…-> ov

code review: cleanups

c22afbd

code review: fix headers

849c10e

fix ngraph -> ov

3eca1f7

rename camel style to snake

8738749

remove/fix ngraph includes

4af671d

use static_partial_shapes_to_test_representation

a589658

move generate_inputs.cpp aux functions to anon namespace

af2cf3f

code review fix

11d9b90

ci fixes

ab541a2

code review: refactor get_runtime_precision methods

a483cce

refactor get_fq_scalar_range

bf9967a

fixes after rebase

2822678

evkotov force-pushed the cvs_111374_dev branch from 8a1cdf5 to 2822678 Compare January 17, 2024 14:37

add windows failures to skip list

7a15cb8

vurusovs approved these changes Jan 17, 2024

View reviewed changes

praasz reviewed Jan 18, 2024

View reviewed changes

.../shared_tests_instances/low_precision_transformations/fuse_subtract_to_fq_transformation.cpp Outdated Show resolved Hide resolved

iefode approved these changes Jan 18, 2024

View reviewed changes

jane-intel approved these changes Jan 18, 2024

View reviewed changes

Merge branch 'master' into cvs_111374_dev

58685db

itikhono reviewed Jan 18, 2024

View reviewed changes

.../shared_tests_instances/low_precision_transformations/fuse_subtract_to_fq_transformation.cpp Outdated Show resolved Hide resolved

itikhono and others added 3 commits January 18, 2024 12:17

Apply suggestions from code review

47313f6

Co-authored-by: Evgenya Nugmanova <[email protected]>

Update src/plugins/intel_gpu/tests/functional/shared_tests_instances/…

a7a2337

…low_precision_transformations/fuse_multiply_to_fq_transformation.cpp Co-authored-by: Evgenya Nugmanova <[email protected]>

Update src/plugins/intel_gpu/tests/functional/shared_tests_instances/…

11166b7

…low_precision_transformations/fuse_subtract_to_fq_transformation.cpp Co-authored-by: Evgenya Nugmanova <[email protected]>

itikhono approved these changes Jan 18, 2024

View reviewed changes

itikhono enabled auto-merge (squash) January 18, 2024 08:20

p-durandin approved these changes Jan 18, 2024

View reviewed changes

itikhono merged commit 0dc2dc5 into openvinotoolkit:master Jan 18, 2024
99 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LP Transformation tests use API 2.0 #21677

LP Transformation tests use API 2.0 #21677

evkotov commented Dec 14, 2023

eshoguli commented Dec 29, 2023

praasz Jan 5, 2024

evkotov Jan 10, 2024

evkotov Jan 11, 2024

evkotov Jan 17, 2024

iefode Jan 9, 2024

evkotov Jan 11, 2024

iefode Jan 18, 2024

sbalandi commented Jan 10, 2024

iefode left a comment

iefode Jan 18, 2024

evkotov Jan 30, 2024

iefode Jan 18, 2024

iefode Jan 18, 2024

jane-intel left a comment

jane-intel Jan 18, 2024

jane-intel Jan 18, 2024

itikhono commented Jan 18, 2024


		static std::pair<float, float> getQuantizationInterval(const ngraph::element::Type precision);
		static std::string to_string(const ov::pass::low_precision::LayerTransformation::Params& params);

LP Transformation tests use API 2.0 #21677

LP Transformation tests use API 2.0 #21677

Conversation

evkotov commented Dec 14, 2023

Details:

Tickets:

eshoguli commented Dec 29, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbalandi commented Jan 10, 2024

iefode left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jane-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

itikhono commented Jan 18, 2024