[Phi] Change the output format of C++ backward api (Part2) #42545
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
你的PR提交成功，感谢你对开源项目的贡献! |
… change_backward_output
… change_backward_output
zyfncg
changed the title
JiabinYang
reviewed
May 12, 2022
| @@ -1321,7 +1321,11 @@ def GenerateNodeDefinition(self, grad_node_creation_str, | |||
| }} | |||
| api_output[i].reserve(returns[i].size()); | |||
| for (size_t j = 0; j < returns[i].size(); ++j) {{ | |||
| api_output[i].push_back(&returns[i][j]); | |||
| if (out_metas[i][j].IsStopGradient()) {{ | |||
There was a problem hiding this comment.
maybe only one for loop can do this all, because we have the same size for out_metas, returns, api_output
paddle/fluid/eager/utils.cc
Outdated
| auto tensor_with_zero = paddle::experimental::full( | ||
| phi::vectorize(tensor_meta.dims), 0.0, tensor_meta.dtype, place); | ||
| grad.set_impl(tensor_with_zero.impl()); | ||
| if (grad_in_meta.HasTensorMeta()) { |
There was a problem hiding this comment.
反向计算经过剪枝后，在高阶导的计算中可能会出现一些不需要的输入Tensor，这里原来的check逻辑会将这些Tensor拦截，导致不能往下计算。
There was a problem hiding this comment.
根据昨天的结论做了调整：
- 非optional的输入梯度维持原来的检查逻辑
- optional类型的输入梯度不进行该项检查，如果没有meta信息就跳过填0的操作。
JiabinYang
reviewed
May 16, 2022
| @@ -218,6 +218,8 @@ void GradNodeBase::SetGradOutMeta(const paddle::experimental::Tensor& fwd_in, | |||
| // Set Stop_gradient | |||
| if (fwd_in_meta) { | |||
| meta.SetStopGradient(fwd_in_meta->StopGradient()); | |||
| } else { | |||
| meta.SetStopGradient(true); | |||
There was a problem hiding this comment.
GradSlotMeta的默认值目前是false
paddle/fluid/eager/utils.cc
Outdated
| VLOG(7) << "Should not set grad node twice, original node is:" | ||
| << autograd_meta->GradNode()->name() | ||
| << "current is: " << grad_node->name(); | ||
| if (autograd_meta) { |
There was a problem hiding this comment.
剪枝后输出的Tensor可能没有AutogradMeta
paddle/fluid/eager/utils.cc
Outdated
| @@ -181,7 +183,7 @@ void EagerUtils::SetOutRankWithSlot(std::vector<AutogradMeta*>* targets, | |||
| } | |||
| } | |||
| void EagerUtils::SetOutRankWithSlot(AutogradMeta* target, size_t slot_id) { | |||
| target->SetSingleOutRankWithSlot(slot_id, 0); | |||
| if (target) target->SetSingleOutRankWithSlot(slot_id, 0); | |||
| for name, (ttype, fwd_position, | ||
| grad_api_position) in backward_grad_inputs_map.items(): | ||
| if name in self.optional_inputs: | ||
| if IsPlainTensorType(ttype): |
There was a problem hiding this comment.
高阶反向在使用paddle.grad时会出现某些反向节点分支不处理的情况，这种情况会给对应下一个节点的grad_in填0，在对应的输入为optional时也需要这样的操作
… change_backward_output
YaoCheng8667
pushed a commit
to WorgenZhang/Paddle
that referenced
this pull request
May 19, 2022
…dle#42545) * change the output format of C++ backward api * fix merge conflict * fix sparse api code auto-gen * fix eager_gen bug * fix bug of output is null * fix bug of conv2d_grad_impl * fix optional grad * fix bug of eager-gen double_grad * fix bug * fix multiply_double_grad bug * fix bug of higher order derivative * fix bug of FillZeroForEmptyGradInput * remove redundant vector in grad_node * fix bug of test_deformable_conv_v1_op * fix bug of test_deformable_conv_v1_op * some refacotr
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR types
Others
PR changes
Others
Describe
新动态图反向计算支持剪枝优化：
前续工作(Part1)：C++ 反向API输出格式调整：返回结果由之前的return调整为作为输入参数。#42677
以
matmul_grad为例，matmul反向一般需要计算输入x和y的梯度x_grad和y_grad，但有些情况下只需要计算x_grad和y_grad其中的一个，而另一个不需要计算。matmul_grad接口会根据传入的x_grad和y_grad是否为空指针来决定是否进行计算，从而可以避免无效的计算。stop_gradient属性来决定传入matmul_grad的输出参数是否为空指针，即只有在stop_gradient为false时才会传入有效的Tensor指针。