Refactor AppBase to use common AppMetaData between Q/P tools

[amahussein]

Signed-off-by: Ahmed Hussein (amahussein) [email protected]

Contributes to #980

• this code change aims at using common logic to create and update the application info inside Tools.

Changes

• No changes in unit tests was needed.
• Created a new class AppMetaData that holds the basic information of a spark application: startTime, user, appId, appName, endTime, duration and flag to indicate whether duration is estimated.
• Handling ApplicationStart became common between the EventParser classes: this means that the following functionswere removed:
  • QualificationEventProcessor.doSparkListenerApplicationStart
  • EventsProcessor.doSparkListenerApplicationStart
  • EventsProcessor.doSparkListenerApplicationEnd
  • EventsProcessor.doSparkListenerTaskStart
• Removed unused case classes
  • case class ApplicationCase
  • case class PlanNodeAccumCase
  • case QualSQLExecutionInfo and
  • case QualApplicationInfo
• Moved the ML flags mlEventLogType and pysparkLogFlag to the trait. This goes along having all the predicates in one place such as "isHive, isGpu...etc"
• For AppBase:
  • added appMetaData field
  • appId becomes a method. thevalue can be retrieved from the AppMetadata
  • removed appEndTime because it is redundant as we hold the endTime inside appMetaData
  • Created estimateAppEndTime which is used between Qualification and Profiling to estimate the endTime if it is missing.
  • Temporary, I moved checkMLOps to the QualificationAppInfo because it is not being used by the profiling and we need to reconsider how to etract ML anyway.
  • There is a bunch of methods defined inside AppBase but they do not have anything to do with its fields or methods. I moved those to the object for now until we find a btter class member for them.
    • getPlanMetaWithSchema
    • getPlanInfoWithHiveScan
    • trimSchema

[nartal1]

Thanks @amahussein ! I am half done with the review. Need to go over the callback functions and EventProcessor refactor.

[tgravescs]

the appMetaData honestly seems a bit weird to me. I know we had ApplicationCase before, but if you are accessing things from AppBase should those just exist in the AppBase, we have this weird split of things where some are in both but we seem to wrap but we can still access appMetaData directly. I think its fine to leave for now but its confusing and we should better define the encapsulation

[amahussein]

Thanks @tgravescs

My thought about the AppMetadata that those fields were modified and updated in different places
Do you think an inner class would better? or just getting rid of the AppMetadata by encapsulating operations on fields inside AppBase methods?

[nartal1]

LGTM. Thanks @amahussein !