Handle SparkRapidsBuildInfoEvent in GPU event logs

[cindyyuanjiang]

Fixes #995

Changes

• Handles a new SparkRapidsBuildInfoEvent added in Spark-Rapids
• Write SparkRapidsBuildInfoEvent into file spark_rapids_build_info.json under rapids_4_spark_profile/application_xxxxxx/

Testing

1. Test with GPU event log with SparkRapidsBuildInfoEvent

spark_rapids profiling -v --eventlogs <my_gpu_log> --tools_jar <my_tools_jar>

spark_rapids_build_info.json

[ {
  "sparkRapidsBuildInfo" : {
    "url" : "https://github.com/NVIDIA/spark-rapids.git",
    "branch" : "HEAD",
    "revision" : "f932a7802bbf31b6205358d1abd7c7b49c8bea3c",
    "version" : "24.06.0",
    "date" : "2024-06-13T19:48:28Z",
    "cudf_version" : "24.06.0",
    "user" : "root"
  },
  "sparkRapidsJniBuildInfo" : {
    "url" : "https://github.com/NVIDIA/spark-rapids-jni.git",
    "branch" : "HEAD",
    "gpu_architectures" : "70;75;80;86;90",
    "revision" : "e9c92a5339437ce0cd72bc384084bd7ee45b37f9",
    "version" : "24.06.0",
    "date" : "2024-06-08T01:21:57Z",
    "user" : "root"
  },
  "cudfBuildInfo" : {
    "url" : "https://github.com/rapidsai/cudf.git",
    "branch" : "HEAD",
    "gpu_architectures" : "70;75;80;86;90",
    "revision" : "7c706cc4004d5feaae92544b3b29a00c64f7ed86",
    "version" : "24.06.0",
    "date" : "2024-06-08T01:21:55Z",
    "user" : "root"
  },
  "sparkRapidsPrivateBuildInfo" : {
    "url" : "https://gitlab-master.nvidia.com/nvspark/spark-rapids-private.git",
    "branch" : "HEAD",
    "revision" : "755b4dd03c753cacb7d141f3b3c8ff9f83888b69",
    "version" : "24.06.0",
    "date" : "2024-06-08T11:44:03Z",
    "user" : "root"
  }
} ]

2. Test with event log without SparkRapidsBuildInfoEvent

spark_rapids profiling -v --eventlogs <my_gpu_log> --tools_jar <my_tools_jar>

spark_rapids_build_info.json

[ {
  "sparkRapidsBuildInfo" : { },
  "sparkRapidsJniBuildInfo" : { },
  "cudfBuildInfo" : { },
  "sparkRapidsPrivateBuildInfo" : { }
} ]

Follow up

#1296 to fix the hardcoded SparkRapidsBuildInfoEvent

[tgravescs]

can you update the description to include what the solutions is here, how does user see this information.

[amahussein]

Thanks @cindyyuanjiang
I believe we want to keep the runtimeReport intact.

• The information of the RAPIDS plugin is per-app, because it is exctracted from the eventlog. Therefore, we expect that to be in the Profiler output of the individual application

• We need to maintain backward compatibility. If we profile an older GPU eventlog, we should still build the RAPIDs information. We can use a common object that gets populated either from the eventhandler or from parsing the Spark properties. Eventually the support of the legacy method will be dropped once all customers are using the most recent RAPIDs versions

• The information needs to be passed to the output and to the autotuner as well.

[tgravescs]

overall looks good, it would be nice to have a test to verify that it outputs the file and has valid data when the event is present. Maybe that they are empty when the event isn't present in older ones.

[parthosa]

Thanks @cindyyuanjiang. LGTME. Similar comments as Tom.

[cindyyuanjiang]

overall looks good, it would be nice to have a test to verify that it outputs the file and has valid data when the event is present. Maybe that they are empty when the event isn't present in older ones.

Thanks @tgravescs! I added two test cases to verify spark_rapids_build_info.json output when SparkRapidsBuildInfoEvent is present in the event log and when it is not. PTAL.

[tgravescs]

thanks Cindy!

[parthosa]

Thanks @cindyyuanjiang

[amahussein]

LGTME
Thanks @cindyyuanjiang