forked from NVIDIA/spark-rapids-tools
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Mark wholestageCodeGen as shouldRemove when childs are removed
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]> Fixes NVIDIA#860 This commit adjusts the accuracy of the Qual tool by targeting the following issues: - child nodes of wholeStageCodeGen would not be assigned to stages if they have no metrics. - there is a corner case when all the childs of wholeStageCodeGen are marked as shouldRemove. In that case, the node would still be considered unsupported and contribute to the speedup. The changes are: - propagate the stageIDs of wholeStageCodeGen to the child nodes - a wholeStageCodeGen node is marked as shouldRemove when all the childs are marked as shouldRemove. - fix unit-test which has 4 different wholeStageCodeGen nodes that contain only `ColumnarToRow` execs
- Loading branch information
1 parent
99e2ffd
commit 2478f99
Showing
3 changed files
with
16 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
core/src/test/resources/QualificationExpectations/write_format_expectation.csv
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,2 @@ | ||
App Name,App ID,Recommendation,Estimated GPU Speedup,Estimated GPU Duration,Estimated GPU Time Saved,SQL DF Duration,SQL Dataframe Task Duration,App Duration,GPU Opportunity,Executor CPU Time Percent,SQL Ids with Failures,Unsupported Read File Formats and Types,Unsupported Write Data Format,Complex Types,Nested Complex Types,Potential Problems,Longest SQL Duration,SQL Stage Durations Sum,NONSQL Task Duration Plus Overhead,Unsupported Task Duration,Supported SQL DF Task Duration,Task Speedup Factor,App Duration Estimated,Unsupported Execs,Unsupported Expressions,Estimated Job Frequency (monthly) | ||
"Spark shell","local-1629442299891","Not Recommended",1.02,19159.68,394.31,1151,920,19554,788,91.72,"","","CSV;JSON","","","",1235,1049,18251,290,630,2.0,false,"Execute InsertIntoHadoopFsRelationCommand csv;Execute InsertIntoHadoopFsRelationCommand json","",30 | ||
"Spark shell","local-1629442299891","Not Recommended",1.02,19080.81,473.18,1151,920,19554,788,91.72,"","","CSV;JSON","","","",1235,1049,18251,290,630,2.5,false,"Execute InsertIntoHadoopFsRelationCommand csv;Execute InsertIntoHadoopFsRelationCommand json","",30 |