Mark wholestageCodeGen as shouldRemove when child nodes are removed #1142
Conversation
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]> Fixes NVIDIA#860 This commit adjusts the accuracy of the Qual tool by targeting the following issues: - child nodes of wholeStageCodeGen would not be assigned to stages if they have no metrics. - there is a corner case when all the childs of wholeStageCodeGen are marked as shouldRemove. In that case, the node would still be considered unsupported and contribute to the speedup. The changes are: - propagate the stageIDs of wholeStageCodeGen to the child nodes - a wholeStageCodeGen node is marked as shouldRemove when all the childs are marked as shouldRemove. - fix unit-test which has 4 different wholeStageCodeGen nodes that contain only `ColumnarToRow` execs
amahussein
added
bug
| // average speedup across the execs in the WholeStageCodegen for now | ||
| val supportedChildren = childNodes.filterNot(_.shouldRemove) | ||
| val avSpeedupFactor = SQLPlanParser.averageSpeedup(supportedChildren.map(_.speedupFactor)) | ||
| // TODO: revisit the logic behind adding the stages of child nodes to the current node. |
There was a problem hiding this comment.
The below allStagesIncludingChildren is there since the early days. I don't quite understand the reason childNodes stages are appended to the wholeStageCodeGen stageIDs.
@tgravescs do you know why we this was needed? If not, I suggest we remove it.
There was a problem hiding this comment.
looking at the comment its saying that the child node might be associated to a stage but the whole stage codegen doesn't match it. So we basically add all the child node ones in. I don't remember when that can happen but would not remove it unless we are positive its not needed.
There was a problem hiding this comment.
so I don't think this comment was addressed before merged? unless you know this doesn't happen I would prefer to see this comment removed or to file an issue to investigate it. This to me could just lead to confusion from someone reading the code.
There was a problem hiding this comment.
My bad! haven't got my cup of coffee. For some reason I misread that I got 2 approvals.
There was a problem hiding this comment.
Posted a followup #1146
|
The bug of not assigning stages to execs inside wholeStageCodeGen also exists in the Profiler. We can either fix it in this PR or file a followup. |
nartal1
changed the title
There was a problem hiding this comment.
Thanks @amahussein ! The logic LGTM for marking wholestageCodeGen as shouldRemove.
amahussein
changed the title
core/src/main/scala/com/nvidia/spark/rapids/tool/planparser/WholeStageExecParser.scala
Show resolved
Hide resolved
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]> Follow-up on NVIDIA#1142 to remove TODO comment line in WholeStageExecParser
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]> Follow-up on #1142 to remove TODO comment line in WholeStageExecParser
Signed-off-by: Ahmed Hussein (amahussein) [email protected]
Fixes #860 , Fixes #793
This commit adjusts the accuracy of the Qual tool by targeting the following issues:
The changes are:
ColumnarToRowexecs