CI Add unit tests running for Spark 3.0.1 #418
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tgravescs
changed the title
|
build |
revans2
reviewed
Jul 23, 2020
|
yeah I was going to look into running parallel separate. The problem is that unit tests are going to use entire GPU , so we would need multiple GPUs to run in parallel in the same build. We can certainly leave it out of premerge right now if we think it takes to long, I don't think it will be a big deal for nightly. |
revans2
approved these changes
Jul 23, 2020
|
I was thinking multiple jenkins jobs. One to test each platform. We could also drop the initial memory allocated for cudf. We don't run jobs that are that big for the unit tests. We probably could run multiple in parallel that way. |
|
sounds good, I'll talk to Tim and Calvin to see what our ci nodes have for gpus |
|
build |
|
looks like this increased premerge build by about 5 minutes. So its taking around 42 minutes now. |
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Add unit tests running for Spark 3.0.1 * dont' run python integration tests for other versions of Spark Co-authored-by: Thomas Graves <[email protected]>
nartal1
pushed a commit
to nartal1/spark-rapids
that referenced
this pull request
Jun 9, 2021
* Add unit tests running for Spark 3.0.1 * dont' run python integration tests for other versions of Spark Co-authored-by: Thomas Graves <[email protected]>
pxLi
pushed a commit
to pxLi/spark-rapids
that referenced
this pull request
May 12, 2022
tgravescs
pushed a commit
to tgravescs/spark-rapids
that referenced
this pull request
Nov 30, 2023
[auto-merge] bot-auto-merge-branch-22.08 to branch-22.10 [skip ci] [bot]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Nightly and Premerge run unit tests against Spark 3.0.1 as well. 3.1.0 have a few failures we need to fix first.
Let me know thoughts if we think this will take to long for premerge.