Handle metric names from legacy spark

[amahussein]

Signed-off-by: Ahmed Hussein (amahussein) [email protected]

Fixes #1042

Running the tool on a legacy spark eventlog (2.x) caused problems because the SQL metric names are not the same

Changes

This PR adds a map between <legacy-metric-names, new-metric-names>

• The map is looked-up in the following parts:
  • During the construction of an AccumulableInfo: this guarantees that all metrics created in a TaskEnd/Stage are mapped to the valid names
  • During the contruction of the SQLplanGraph: this guarantees that node metrics are mapped to the most recent labels
• I added an extra match-pattern to avoid this failure from happening in the future. When the mtric-name is missing, there will be a warning message that it is not handled correctly.
• The process to do the fix was painful because I had to look into all spark versions to spot any metric-name change. t is possible that I did not catch them all, but at least we can append more mapping in the future.

I had to change UT because the changes actually fixed a hidden bug in our code.

[nartal1]

LGTM. Thanks @amahussein ! That's a long list of legacy metrics name :)

[parthosa]

Thanks @amahussein for this. In the E2E testing doc, I will add an entry for integration testing on legacy spark event logs.