-
Notifications
You must be signed in to change notification settings - Fork 37
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor AppBase to use common AppMetaData between Q/P tools #983
Refactor AppBase to use common AppMetaData between Q/P tools #983
Conversation
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]> Contributes to NVIDIA#980 - this code change aims at using common logic to create and update the application info inside Tools.
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @amahussein ! I am half done with the review. Need to go over the callback functions and EventProcessor refactor.
core/src/main/scala/org/apache/spark/sql/rapids/tool/AppBase.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/AppMetaData.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/AppMetaData.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/EventProcessorBase.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/AppBase.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/AppMetaData.scala
Outdated
Show resolved
Hide resolved
|
||
// Returns a boolean true/false. This is used to check whether processing an eventlog was | ||
// successful. | ||
def isAppMetaDefined: Boolean = appMetaData.isDefined |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the appMetaData honestly seems a bit weird to me. I know we had ApplicationCase before, but if you are accessing things from AppBase should those just exist in the AppBase, we have this weird split of things where some are in both but we seem to wrap but we can still access appMetaData directly. I think its fine to leave for now but its confusing and we should better define the encapsulation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @tgravescs
My thought about the AppMetadata that those fields were modified and updated in different places
Do you think an inner class would better? or just getting rid of the AppMetadata by encapsulating operations on fields inside AppBase methods?
Signed-off-by: Ahmed Hussein (amahussein) <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @amahussein !
Signed-off-by: Ahmed Hussein (amahussein) [email protected]
Contributes to #980
Changes
QualificationEventProcessor.doSparkListenerApplicationStart
EventsProcessor.doSparkListenerApplicationStart
EventsProcessor.doSparkListenerApplicationEnd
EventsProcessor.doSparkListenerTaskStart
case class ApplicationCase
case class PlanNodeAccumCase
case QualSQLExecutionInfo
andcase QualApplicationInfo
mlEventLogType
andpysparkLogFlag
to the trait. This goes along having all the predicates in one place such as "isHive, isGpu...etc"appMetaData
fieldappId
becomes a method. thevalue can be retrieved from the AppMetadataappEndTime
because it is redundant as we hold the endTime insideappMetaData
estimateAppEndTime
which is used between Qualification and Profiling to estimate the endTime if it is missing.checkMLOps
to the QualificationAppInfo because it is not being used by the profiling and we need to reconsider how to etract ML anyway.getPlanMetaWithSchema
getPlanInfoWithHiveScan
trimSchema