Generate CSV data per Spark version for tools [databricks] #10440
Conversation
Signed-off-by: Jason Lowe <[email protected]>
jlowe
requested review from
revans2,
tgravescs,
GaryShen2008,
NvTimLiu and
gerashegalov
as code owners
|
build |
|
build |
There was a problem hiding this comment.
Should we name the dir tools-support to differentiate from the tools repo and to match the Maven module name?
There was a problem hiding this comment.
Sounds good, but we probably want to address this as a followup. I don't want to break downstream pipelines which are expecting to find things under tools/generated_files/, and those would need to be updated to check the new location if the old is missing.
| @@ -264,7 +264,7 @@ function build_single_shim() { | |||
| -Drat.skip="$SKIP_CHECKS" \ | |||
| -Dmaven.scaladoc.skip \ | |||
| -Dmaven.scalastyle.skip="$SKIP_CHECKS" \ | |||
| -pl aggregator -am > "$LOG_FILE" 2>&1 || { | |||
| -pl tools -am > "$LOG_FILE" 2>&1 || { | |||
There was a problem hiding this comment.
Previously the idea was to build the minimum set of artifacts for dist ? Should buildall have a separate option for including tools?
There was a problem hiding this comment.
Should there be an argument to not build tools? I would expect buildall to be used by developers as an easy way to build all the Spark versions, and more often than not they want to build the tools CSV data to know whether or not they need to checkin updated files as part of their PR. The build of these files is quick, so we're saving much by making the default to not build the tools data, and I think building the tools data is a better default. Thoughts?
There was a problem hiding this comment.
It adds less than 4 seconds to the parallel part of the build, so (totalShimsInProfile / numParallelShims) * 4 sec is acceptable. Agree, no pressing need to add a skip option now
|
build |
Fixes #10424. Adds CSV generation per Apache Spark version under tools/generated_files. The existing CSV files for tools data has been preserved to avoid breaking downstream pipelines that use this data. Once those pipelines have been updated to consume the new per-Spark versions of these files, we can remove the old tools CSV files.