[OpTest] support prim test in OpTest #50509

Charles-hit · 2023-02-14T14:08:01Z

PR types

New features

PR changes

Others

Describe

任务背景：
组合算子+编译器协同任务需要对算子精度进行保障，为了复用框架原有的算子单测case，本PR在OpTest中进行改造，新增了组合测试功能对算子精度进行保障。
PR改动：
1.对OpTest新增组合测试功能。
2.修复OpTest框架如果某个单测挂了导致动态图静态图运行模式不正确的问题，对静态图代码利用guard进行保护。
3.对softmax、expand、reduce_sum三个算子添加组合单测。

paddle-bot · 2023-02-14T14:08:08Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

… prim_test_frame

xiaoguoguo626807 · 2023-02-20T09:29:32Z

python/paddle/fluid/tests/unittests/prim_op_test.py

+        return kernel_sig
+
+    def is_only_check_prim(self):
+        return self.only_prim


why only test prim

用户在新增一些组合单测会触发Op测试，但是部分Op规定不可以测试fp32类型，导致单测挂掉。对于组合测试并没有这个要求，这个开关用来跳过非组合的测试。

xiaoguoguo626807 · 2023-02-20T09:33:52Z

python/paddle/fluid/tests/unittests/eager_op_test.py

            )
            if not kernel_sig:
                return None
            assert hasattr(self, "python_api"), (
                "Detect there is KernelSignature for `%s` op, please set the `self.python_api` if you set check_dygraph = True"
                % self.op_type
            )
-            args = prepare_python_api_arguments(
+            args = OpTestUtils.prepare_python_api_arguments(


add todo: those code change will recover after delete legacy dygraph

xiaoguoguo626807 · 2023-02-20T09:36:07Z

python/paddle/fluid/tests/unittests/eager_op_test.py

        inplace_atol=None,
    ):
+        core._set_prim_all_enabled(False)
+


_set_prim_forward_enabled enough?

在跑这个前向case中会有反向inplace的测试，这儿是修复当某个单测挂掉导致前反向开关没有关，走到组合inplace逻辑中导致段错误。

cyber-pioneer · 2023-02-20T11:09:08Z

python/paddle/fluid/tests/unittests/config.py

+        "rev_comp": {"rtol": 1e-2, "atol": 1e-2},
+        "cinn": {"rtol": 1e-1, "atol": 1e-1},
+    },
+}


是否需要支持bfloat16？

numpy没有bfloat16数据类型，这儿用unit16来表示bfloat16，目前python api都是这样做的

cyber-pioneer · 2023-02-20T11:41:52Z

python/paddle/fluid/tests/unittests/eager_op_test.py

+        if check_prim:
+            prim_checker = PrimForwardChecker(self, place)
+            prim_checker.check()
+            # Support operators which not in the NO_FP64_CHECK_GRAD_OP_LIST list can be test prim with fp32


用户会感知这里的dtype逻辑么？

cyber-pioneer · 2023-02-20T11:43:11Z

python/paddle/fluid/tests/unittests/test_reduce_op.py

@@ -66,6 +55,8 @@ def setUp(self):
            'Out': self.inputs['X'].sum(axis=tuple(self.attrs['dim']))
        }
        self.gradient = self.calc_gradient()
+        # error occurred in cinn


未来我们会用代码扫一遍看看哪些case没开cinn，最后会集中处理。

XiaoguangHu01 · 2023-02-21T07:44:37Z

python/paddle/tensor/math.py

@@ -1265,6 +1265,7 @@ def sum(x, axis=None, dtype=None, keepdim=False, name=None):
            'x',
            [
                'bool',
+                'uint16',


为什么需要新增这个数据类型呢？

因为这个在python端拿来表示bfloat16数据类型（numpy没有bfloat16，所以用这个来表示），静态图目前bfloat16数据类型流程没有完全测试过，这儿新增uint16来测试sum算子静态图下bfloat16数据类型。

xiaoguoguo626807

LGTM

cyber-pioneer · 2023-02-21T09:47:38Z

LGTM

XiaoguangHu01

LGTM

JiabinYang

some comments for now

JiabinYang · 2023-02-20T05:00:40Z

python/paddle/fluid/tests/unittests/config.py

+
+import numpy as np
+
+TOLERANCE = {


dose this config used only for op test?

JiabinYang · 2023-02-20T05:01:41Z

python/paddle/fluid/tests/unittests/eager_op_test.py

@@ -401,6 +401,7 @@ def is_custom_device_op_test():
                and not is_npu_op_test()
                and not is_mlu_op_test()
                and not is_custom_device_op_test()
+                and not cls.check_prim


JiabinYang · 2023-02-20T05:03:06Z

python/paddle/fluid/tests/unittests/eager_op_test.py


 sys.path.append(os.path.abspath(os.path.dirname(__file__)))
+from prim_op_test import OpTestUtils, PrimForwardChecker, PrimGradChecker


Make OpTestUtils in an independent file instead of in prim_op_test, since it's not only for prim op test

JiabinYang · 2023-02-20T05:03:39Z

python/paddle/fluid/tests/unittests/eager_op_test.py

-        program = Program()
-        block = program.global_block()
-        op = self._append_ops(block)
+        with paddle.fluid.framework._dygraph_guard(None):


Do not use fluid api?

Why dygraph_guard, there are all static operations below

JiabinYang · 2023-02-20T05:18:43Z

python/paddle/fluid/tests/unittests/eager_op_test.py

-                grad_program
-            ).with_data_parallel(
-                loss_name="", build_strategy=build_strategy, places=place
+        with paddle.fluid.framework._dygraph_guard(None):


same...using program_guard

JiabinYang · 2023-02-20T05:20:57Z

python/paddle/fluid/tests/unittests/prim_op_test.py

+        self.checker_name = "PrimForwardChecker"
+        self.place = place
+        self.op_test = op_test
+        self.save_eager_or_static_status()


JiabinYang · 2023-02-20T05:21:35Z

python/paddle/fluid/tests/unittests/prim_op_test.py

+    def init_checker(self):
+        assert hasattr(
+            self.op_test, 'prim_op_type'
+        ), "if you want to test comp op, please set prim_op_type in setUp function."


more comments for prim_op_type, what is it?

JiabinYang · 2023-02-20T05:31:12Z

python/paddle/fluid/tests/unittests/prim_op_test.py

+        assert hasattr(
+            self.op_test, 'dtype'
+        ), "Please set dtype in setUp function."
+        self.op_type = self.op_test.op_type


why not use self.op_test.xxx directly

support prim test in OpTest

0324a82

Charles-hit added 26 commits February 14, 2023 14:46

fix cmake

2f4b577

fix op test

f8c067e

fix test_input_spec

92705f5

disable cinn in reduce_sum unit test

10aa52f

add bfloat16 dtype for sum

739e146

polish code

71fc6a3

add clear jit program function

82f2057

convert grad out from tensor to numpy

eaf8176

remove unnecessary code

f8e4fa2

add only_prim flag

0ed31a1

fix flag

596b77c

fix op test

db1ba08

fix optest comp inplace error

5b141d3

fix op test

c04cbf2

fix op test with guard

23d978f

add initialization of check_comp flag

016ee35

fix comp inplace error in op test

12ab8cb

rename check_comp with check_prim and add bfloat16 dtype convert

4b2d222

rename comp_op_type to prim_op_type

8b6dda7

rename comp to prim

889146c

remove useless code

7913e27

skip ci check for only prim

a0210f3

add no_grad_vars and grad_outputs in prim test

c3ffa48

fix var_dict

e48efee

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

0d375a0

… prim_test_frame

fix op test for only_prim

dbddace

xiaoguoguo626807 reviewed Feb 20, 2023

View reviewed changes

cyber-pioneer reviewed Feb 20, 2023

View reviewed changes

Charles-hit added 2 commits February 20, 2023 12:20

fix dy2static bugs

e825902

polish some code

d19cf71

XiaoguangHu01 reviewed Feb 21, 2023

View reviewed changes

xiaoguoguo626807 approved these changes Feb 21, 2023

View reviewed changes

cyber-pioneer approved these changes Feb 21, 2023

View reviewed changes

zhiqiu approved these changes Feb 21, 2023

View reviewed changes

XiaoguangHu01 approved these changes Feb 21, 2023

View reviewed changes

Charles-hit merged commit 457defe into PaddlePaddle:develop Feb 21, 2023

JiabinYang reviewed Feb 21, 2023

View reviewed changes

Charles-hit deleted the prim_test_frame branch February 28, 2023 07:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[OpTest] support prim test in OpTest #50509

[OpTest] support prim test in OpTest #50509

Charles-hit commented Feb 14, 2023 •

edited

Loading

paddle-bot bot commented Feb 14, 2023

xiaoguoguo626807 Feb 20, 2023

Charles-hit Feb 20, 2023

xiaoguoguo626807 Feb 20, 2023

Charles-hit Feb 20, 2023

xiaoguoguo626807 Feb 20, 2023

Charles-hit Feb 20, 2023

cyber-pioneer Feb 20, 2023

Charles-hit Feb 20, 2023

cyber-pioneer Feb 20, 2023

Charles-hit Feb 20, 2023

cyber-pioneer Feb 20, 2023

Charles-hit Feb 20, 2023

XiaoguangHu01 Feb 21, 2023

Charles-hit Feb 21, 2023

xiaoguoguo626807 left a comment

cyber-pioneer commented Feb 21, 2023

XiaoguangHu01 left a comment

JiabinYang left a comment

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023

JiabinYang Feb 20, 2023


		sys.path.append(os.path.abspath(os.path.dirname(__file__)))
		from prim_op_test import OpTestUtils, PrimForwardChecker, PrimGradChecker

[OpTest] support prim test in OpTest #50509

[OpTest] support prim test in OpTest #50509

Conversation

Charles-hit commented Feb 14, 2023 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented Feb 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiaoguoguo626807 left a comment

Choose a reason for hiding this comment

cyber-pioneer commented Feb 21, 2023

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

JiabinYang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Charles-hit commented Feb 14, 2023 •

edited

Loading