[FEA] Qualification tool should look at spill metrics

[tgravescs]

Is your feature request related to a problem? Please describe.
spill can be very time consuming, the qualification tool should try to take spill metrics into account.

Note, originally mentioned in #73

Tasks

Beta Give feedback

1. Refactor Stage info code between Q/P tools #971
   bug core_tools +2
   
2. Add heuristics using stage spill metrics to skip apps #1002
   feature request user_tools +2
   

[viadea]

To add some context, I wish Qualification tool can consider both of below factors:

1. "Spilling to disks" "Spilling to memory" "Shuffle read" and "Shuffle write" metrics
2. Cluster shape to figure out the local disk bandwidth per executor

For example, Standard_NC8as_T4_v3 has 240MB/s combined read/write for each local disk per Spark executor;
While a Dataproc node with 2 x T4s and 8 local NVMEs per node, has 1.4GB/s write throughput for local disks per Spark executor.

If the cluster shape information is provided as the input for Qualification tool, it should figure out a way to measure how much negative impact based on above 2 factors.

[tgravescs]

The problem is that just because CPU spills doesn't mean GPU is going to spill and vise versa, cpu might not spill but GPU does. A lot of factors go into this. This is a very hard problem. Maybe we can do some heuristics about if spill on CPU is over certain threshold GPU will likely spill too, or we need to look at other characteristics/metrics in the job with cluster, but that definitely gets complex.

[amahussein]

We discussed offline the initial steps to tackle this issue:

• Make the qualification tool capture spill metrics per stage
• Identify stages that exhibit spills
• Any stage that has spills and they do not have a join agg operator needs to be flagged.