Print the forward's stack when backward op has nan/inf and FLAGS_check_nan_inf_level = 0 #52639

AnnaTrainingG · 2023-04-07T08:36:20Z

PR types

Others

PR changes

Others

Describe

新增nan_inf检查工具反向op调用栈打印的支持，效果如下：

paddle-bot · 2023-04-07T08:36:25Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

Xreki

需要添加单测

Xreki · 2023-04-12T06:07:11Z

paddle/fluid/eager/api/manual/eager_manual/forwards/add_n_fwd_func.cc

+        egr::Controller::Instance().GetOpPythonStackStr() + forward_trace;
+    try {
+      PADDLE_ENFORCE(
+          false,


这个PADDLE_ENFORCE有什么用呢？

之前是通过catch的形式获取当前函数的名称和行数现在不需要了我删除一下

Xreki · 2023-04-12T06:10:32Z

paddle/fluid/eager/api/manual/eager_manual/forwards/conv2d_fwd_function.cc

  // Check NaN and Inf if needed
  if (FLAGS_check_nan_inf) {
    egr::CheckTensorHasNanOrInf("conv2d", api_result);
+    std::string filename = __FILE__;


每个manual实现的API都需要添加，重复代码太多了

Xreki · 2023-04-12T06:15:42Z

paddle/fluid/eager/api/utils/global_utils.h

@@ -74,6 +74,12 @@ class Controller {

  void EnableLayoutAutoTune() { tracer_->EnableLayoutAutoTune(); }

+  void SetOpPythonStackStr(std::string stack_str) {


SetOpPythonStackStr -> SetPythonStack

Xreki · 2023-04-12T06:20:09Z

paddle/fluid/eager/auto_code_generator/generator/python_c_gen.py

+       std::string filename = __FILE__;
+       std::string line= std::to_string(__LINE__);
+       std::string function_name = __FUNCTION__;
+       pybind11::gil_scoped_acquire gil;


获取python代码调用栈的pybind实现，还是建议放在pybind正式代码里面，尽量不要全部都用代码生成的方式。

Xreki · 2023-04-12T06:21:45Z

paddle/fluid/eager/grad_node_info.h

+    backward_trace_ = backward_trace;
+  }
+
+  std::string GetForwardTrace() { return backward_trace_; }


backward_trace_ -> forward_trace_

zhangting2020

LGTM

Xreki · 2023-04-18T09:05:22Z

paddle/fluid/eager/auto_code_generator/generator/eager_gen.py

+CHECK_NAN_AND_INF_TEMPLATE_BACKWARD = """
+  if (FLAGS_check_nan_inf) {{
+     try{{
+       egr::CheckTensorHasNanOrInf("{}", {});


forward_trace可以直接传入CheckTensorHasNanOrInf，在CheckTensorHasNanOrInf里面打印？

那样就需要修改CheckTensorHasNanOrInf了，代码改的比较多了

改CheckTensorHasNanOrInf是合理的啊，因为打印调用栈本身就是CheckTensorHasNanOrInf功能的一部分，后面静态图也能服用部分逻辑，可以后续PR再考虑。

Xreki · 2023-04-18T09:07:00Z

paddle/fluid/eager/auto_code_generator/generator/eager_gen.py

+       std::cout<<forward_trace<<std::endl;
+       std::rethrow_exception(std::current_exception());
+     }}
+  }}


注意字符串模板中的代码格式，另外为啥要用两个花括号？

这是自动代码生成需要用两层花括号

Xreki · 2023-04-18T09:11:35Z

paddle/fluid/eager/auto_code_generator/generator/python_c_gen.py

@@ -121,7 +121,10 @@ def FindParsingFunctionFromAttributeType(atype):
 NOAMP_DYGRAPH_FUNCTION_TEMPLATE = "decltype({}({})) out = {}({});"


-FUNCTION_SET_DEVICE_TEMPLATE = """{}    if (paddle::platform::is_gpu_place(place)) {{
+FUNCTION_SET_DEVICE_TEMPLATE = """{}
+    LOG(INFO)<<"this is SetPythonStack";


这个模板是用于哪里的？LOG(INFO)会一直输出吧

是的这个我下个PR再修改

Xreki · 2023-04-18T09:12:09Z

paddle/fluid/eager/auto_code_generator/generator/python_c_gen.py

-FUNCTION_SET_DEVICE_TEMPLATE = """{}    if (paddle::platform::is_gpu_place(place)) {{
+FUNCTION_SET_DEVICE_TEMPLATE = """{}
+    LOG(INFO)<<"this is SetPythonStack";
+    SetPythonStack();


这里需要加if (FLAGS_check_nan_inf)判断来限制一下吗？

不需要，这个调用里面我加了判断

Xreki · 2023-04-18T09:18:11Z

python/paddle/fluid/tests/unittests/test_nan_inf.py

@@ -141,6 +148,25 @@ def _check_num_nan_inf(use_cuda):
        if paddle.fluid.core.is_compiled_with_cuda():
            _check_num_nan_inf(use_cuda=True)

+    def test_check_stack(self):


这个测试其实没有必要再费劲加到这个单测里面了。之所以构造这么一个复杂的测试方式，是因为之前CUDA抛出的异常捕获不到。调用栈报错，应该可以直接用try ... except捕获到。

不行呀，前向的调用栈是print出来的，无法try... except捕获

JiabinYang

2comments

JiabinYang · 2023-04-18T09:32:10Z

paddle/fluid/eager/auto_code_generator/generator/eager_gen.py

@@ -1047,11 +1069,15 @@ def GenerateNodeCreationCodes(self, for_backward=False):

        node_event_name = forward_api_name + " node_creation"
        node_creation_event_str = f"{indent}paddle::platform::RecordEvent node_creation_record_event(\"{node_event_name}\", paddle::platform::TracerEventType::OperatorInner, 1);\n"
+        set_forward_trace = (


能否设置FLAGS_check_nan_inf为false时不设置？

可以的，后续和上面的临时变量问题统一修改

JiabinYang · 2023-04-18T09:33:36Z

paddle/fluid/eager/auto_code_generator/generator/eager_gen.py

@@ -484,7 +488,25 @@ class {} : public egr::GradNodeBase {{
    }}
  }}"""

-CHECK_NAN_AND_INF_TEMPLATE = """  if (FLAGS_check_nan_inf) {{ egr::CheckTensorHasNanOrInf("{}", {}); }}
+CHECK_NAN_AND_INF_TEMPLATE_FORWARD = """
+  std::string forward_trace ="";


是否可以不使用临时变量减少下平凡的构造和析构开销

可以的我下个PR统一修改

JiabinYang

LGTM, fix problem in next pr

…k_nan_inf_level = 0 (PaddlePaddle#52639)

trace back

200406f

AnnaTrainingG added 3 commits April 10, 2023 06:52

功能ok

e1bfd38

update

3b29a43

update

a7ab44c

Xreki reviewed Apr 12, 2023

View reviewed changes

AnnaTrainingG changed the title ~~trace back~~ Print the forward's stack when backward op has nan/inf and FLAGS_check_nan_inf_level = 0 Apr 12, 2023

AnnaTrainingG added 4 commits April 13, 2023 02:37

move traceback to function

f907fd3

update

144d423

update

9d787d3

update

b4eb0c9

AnnaTrainingG force-pushed the trace_back branch from d21bf46 to b4eb0c9 Compare April 14, 2023 09:24

update

78714b7

zhangting2020 approved these changes Apr 18, 2023

View reviewed changes

Xreki reviewed Apr 18, 2023

View reviewed changes

JiabinYang reviewed Apr 18, 2023

View reviewed changes

JiabinYang approved these changes Apr 18, 2023

View reviewed changes

AnnaTrainingG merged commit 660f781 into PaddlePaddle:develop Apr 18, 2023

Xreki mentioned this pull request Apr 19, 2023

Remove a LOG(INFO). #53056

Merged

jjyaoao pushed a commit to jjyaoao/Paddle that referenced this pull request Apr 19, 2023

Print the forward's stack when backward op has nan/inf and FLAGS_chec…

2e7e202

…k_nan_inf_level = 0 (PaddlePaddle#52639)

AnnaTrainingG added a commit to AnnaTrainingG/Paddle that referenced this pull request Apr 23, 2023

Print the forward's stack when backward op has nan/inf and FLAGS_chec…

8b32810

…k_nan_inf_level = 0 (PaddlePaddle#52639)

lijialin03 pushed a commit to lijialin03/Paddle that referenced this pull request Apr 25, 2023

Print the forward's stack when backward op has nan/inf and FLAGS_chec…

8592780

…k_nan_inf_level = 0 (PaddlePaddle#52639)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Print the forward's stack when backward op has nan/inf and FLAGS_check_nan_inf_level = 0 #52639

Print the forward's stack when backward op has nan/inf and FLAGS_check_nan_inf_level = 0 #52639

AnnaTrainingG commented Apr 7, 2023 •

edited

Loading

paddle-bot bot commented Apr 7, 2023

Xreki left a comment

Xreki Apr 12, 2023

AnnaTrainingG Apr 12, 2023

AnnaTrainingG Apr 14, 2023

Xreki Apr 12, 2023

AnnaTrainingG Apr 14, 2023

Xreki Apr 12, 2023

AnnaTrainingG Apr 14, 2023

Xreki Apr 12, 2023

AnnaTrainingG Apr 14, 2023

Xreki Apr 12, 2023

AnnaTrainingG Apr 14, 2023

zhangting2020 left a comment

Xreki Apr 18, 2023

AnnaTrainingG Apr 18, 2023

Xreki Apr 18, 2023 •

edited

Loading

Xreki Apr 18, 2023

AnnaTrainingG Apr 18, 2023

Xreki Apr 18, 2023

AnnaTrainingG Apr 18, 2023

Xreki Apr 18, 2023

AnnaTrainingG Apr 18, 2023

Xreki Apr 18, 2023

AnnaTrainingG Apr 18, 2023

JiabinYang left a comment

JiabinYang Apr 18, 2023

AnnaTrainingG Apr 18, 2023

JiabinYang Apr 18, 2023

AnnaTrainingG Apr 18, 2023

JiabinYang left a comment

		@@ -74,6 +74,12 @@ class Controller {

		void EnableLayoutAutoTune() { tracer_->EnableLayoutAutoTune(); }

		void SetOpPythonStackStr(std::string stack_str) {

Print the forward's stack when backward op has nan/inf and FLAGS_check_nan_inf_level = 0 #52639

Print the forward's stack when backward op has nan/inf and FLAGS_check_nan_inf_level = 0 #52639

Conversation

AnnaTrainingG commented Apr 7, 2023 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented Apr 7, 2023

Xreki left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangting2020 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Xreki Apr 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiabinYang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JiabinYang left a comment

Choose a reason for hiding this comment

AnnaTrainingG commented Apr 7, 2023 •

edited

Loading

Xreki Apr 18, 2023 •

edited

Loading