Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up build: unnecessary invalidation in the incremental recompile mode [databricks] #9698

Merged
merged 30 commits into from
Nov 17, 2023
Merged

Speed up build: unnecessary invalidation in the incremental recompile mode [databricks] #9698

merged 30 commits into from
Nov 17, 2023

Conversation

gerashegalov
Copy link
Collaborator

@gerashegalov gerashegalov commented Nov 14, 2023 •
edited
Loading

Fixes #9691

This PR aims at reducing unnecessary IncrementalCompiler output invalidations due to jar updates

  • Update build info only when the HEAD revision changes, or on clean
  • Produce the final aggregator jar only if the shade plugin output changed
  • Skip generating META-INF/ShimServiceProvider if already exists
  • It was not breaking anything but it does not make sense to have -Xsource:2.13 in the 2.12 build
  • Replace skip by maven.scaladoc.skip in docs and scripts as a follow up to Bloop updates - require JDK11 in buildall + docs, build bloop for all targets. #9615
  • fix binary-dedupe to not look outside the target/parallel-world dir
  • optional skip for unpack of nightly dependencies, need a general caching solution

Repeated executions

Wall Clock pre-PR Wall Clock post-PR Maven command
01:10 min 13.004 s mvn package -pl tests -am -Dbuildver=330 \
-Dmaven.scaladoc.skip -DskipTests
01:23 min 25.801 s mvn package -pl tests -am -Dbuildver=330 \
-Dmaven.scaladoc.skip -Dsuffixes=rapids.CastOpSuite \
-Dtests='Cast from string to timestamp'
01:30 min 30.00 s mvn package -Dbuildver=330 \
-Ddist.jar.compress=false -DskipTests -Dmaven.scaladoc.skip
48.769 s 19.253 s mvn -T1C package -pl dist -am -Dbuildver=330 \
-Ddist.jar.compress=false -DskipTests \
-Drapids.jni.unpack.skip=true

Signed-off-by: Gera Shegalov [email protected]

@gerashegalov
Consolidate delta-lake poms
7f7d4cb 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
skip build-info if revision unchanged
91e718a
@gerashegalov
scala2.13
afb81b2 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
compile output as dependency
da86cda 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
scala2.13 common
9684214 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
revert to 4.3.0
f6dd0e5 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
scala2.13
3d0fe74 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
downgrade scala-maven-plugin
fd8bbd9 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
incremental compile
f79d88b 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
incremental end-to-end
868e36b 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
may achieve with a single aggregate module
ca299c8 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
remove aggrgator-tmp
2a35697
@gerashegalov
Merge remote-tracking branch 'origin/branch-23.12' into speedupBuildInfo
74da55a
@gerashegalov
delete diff
8ea3f76
@gerashegalov
Revert delta-lake changes
6157a5b
@gerashegalov
scala2.13
baff166 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
revert unneeded changes
56f6ecd 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
change execution id
9cc11f2 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Replace skip with maven.scaladoc.skip
7ea535a 
Follow up to NVIDIA#9615

Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
robuster jar swapping logic
c3d9740 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov gerashegalov added the build Related to CI / CD or cleanly building label Nov 14, 2023
@gerashegalov gerashegalov self-assigned this Nov 14, 2023
@gerashegalov
scala2.13 sync
b4ec26e 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov
use copy instead of move
6af04b5 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov
undo compiler plugin change
db14701 
Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Copy link
Collaborator Author

build

2 similar comments
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov
Copy link
Collaborator Author

build

jlowe
jlowe reviewed Nov 15, 2023
View reviewed changes
pom.xml Outdated Show resolved Hide resolved
@gerashegalov
rapids.jni.unpack.skip
f2051e3 
Signed-off-by: Gera Shegalov <[email protected]>
jlowe
jlowe reviewed Nov 15, 2023
View reviewed changes
tests/README.md Outdated Show resolved Hide resolved
tests/README.md Outdated Show resolved Hide resolved
gerashegalov and others added 2 commits November 15, 2023 14:41
@gerashegalov @jlowe
Apply suggestions from code review
e5aecbc 
Co-authored-by: Jason Lowe <[email protected]>
@gerashegalov
reviews
5efd1c2 
Signed-off-by: Gera Shegalov <[email protected]>
jlowe
jlowe previously approved these changes Nov 15, 2023
View reviewed changes
@gerashegalov
Copy link
Collaborator Author

build

1 similar comment
@gerashegalov
Copy link
Collaborator Author

build

@gerashegalov
Fixes
24ca3c7 
- binary-dedupe should not look outside paralle-world directory
- when unpack is skipped we should restore jni from a staged dir

Signed-off-by: Gera Shegalov <[email protected]>
@gerashegalov
Copy link
Collaborator Author

build

1 similar comment
@gerashegalov
Copy link
Collaborator Author

build

gerashegalov
gerashegalov commented Nov 16, 2023
View reviewed changes
tests/README.md
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@andygrove you have previously asked about different ScalaTest-related commands

@gerashegalov gerashegalov merged commit 30c3df3 into NVIDIA:branch-23.12 Nov 17, 2023
37 checks passed
@gerashegalov gerashegalov deleted the speedupBuildInfo branch November 17, 2023 00:15
@gerashegalov gerashegalov mentioned this pull request Nov 17, 2023
Merged
NVnavkumar pushed a commit to NVnavkumar/spark-rapids that referenced this pull request Nov 18, 2023
@gerashegalov
[BUG] Fix the doc for Maven and Scala 2.13 test example [skip ci] (NV…
4f0e4fa 
…IDIA#9770)

Addendum to NVIDIA#9698 

Add `-f scala2.13` to the ScalaTest README

---------
Signed-off-by: Gera Shegalov <[email protected]>
This was referenced Nov 21, 2023
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jlowe jlowe jlowe approved these changes

@revans2 revans2 Awaiting requested review from revans2 revans2 is a code owner

@tgravescs tgravescs Awaiting requested review from tgravescs tgravescs is a code owner

@GaryShen2008 GaryShen2008 Awaiting requested review from GaryShen2008 GaryShen2008 is a code owner

@NvTimLiu NvTimLiu Awaiting requested review from NvTimLiu NvTimLiu is a code owner

@pxLi pxLi Awaiting requested review from pxLi pxLi is a code owner

@andygrove andygrove Awaiting requested review from andygrove

Assignees

@gerashegalov gerashegalov

Labels
build Related to CI / CD or cleanly building
Projects
None yet
Milestone
No milestone
2 participants
@gerashegalov @jlowe