CI: add gpu tag test job #478
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mashehu
reviewed
Dec 2, 2024
maxulysse
reviewed
Dec 3, 2024
maxulysse
reviewed
Dec 3, 2024
mashehu
reviewed
Dec 3, 2024
Co-authored-by: Matthias Hörtenhuber <[email protected]>
mashehu
reviewed
Dec 3, 2024
|
mashehu
reviewed
Dec 3, 2024
| nf-test test \ | ||
| --ci \ | ||
| --shard ${{ inputs.shard }}/${{ inputs.total_shards }} \ | ||
| --changed-since HEAD^ \ |
There was a problem hiding this comment.
Suggested change
| --changed-since HEAD^ \ |
we already get the tests to run in the nf-test-shard action (just need to export it there), simpler than running it in two places.
Additionally, this changed since will also include deleted files, no?
There was a problem hiding this comment.
have to check about deleted files
mashehu
reviewed
Dec 3, 2024
| conda-remove-defaults: true | ||
|
|
||
| - name: Clean up Disk space | ||
| uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1 |
There was a problem hiding this comment.
is there a way to make this action not spam the whole log
There was a problem hiding this comment.
I don't see any input for silent: https://github.com/jlumbroso/free-disk-space/blob/54081f138730dfa15788a46383842cd2f914a1be/action.yml#L9
There was a problem hiding this comment.
I ma not convinced that this action is always required. I only include it for specific scenarios such as when 8GB containers have to be downloaded. Will leave it to @sateeshperi
Just an example of what I have done elsewhere: https://github.com/Plant-Food-Research-Open/genepal/blob/ee702d71a9ea4f422c92729533ff0564c231b5e8/.github/workflows/ci.yml#L77
GallVp
reviewed
Dec 3, 2024
nextflow.config
Outdated
| @@ -221,6 +221,17 @@ profiles { | |||
| executor.cpus = 4 | |||
| executor.memory = 8.GB | |||
| } | |||
| docker_self_hosted { | |||
There was a problem hiding this comment.
Should we place it in nf-test.config so as to not cause confusion for the users?
There was a problem hiding this comment.
I am actually curious and think we don't need a special docker profile for GPU runners, because they are not running in rootless docker mode
There was a problem hiding this comment.
I am working on this PR. Will look into it as well. Thank you!
…f related and capped shards with max_shards
GallVp
reviewed
Dec 3, 2024
GallVp
reviewed
Dec 3, 2024
| NFT_VER: ${{ env.NFT_VER }} | ||
| with: | ||
| tags: "cpu" | ||
| max_shards: 5 |
There was a problem hiding this comment.
I have capped the number of shards to 5. This might be a bit too aggressive. @sateeshperi , please feel free to bump it up.
GallVp
reviewed
Dec 3, 2024
| echo "No related tests found." | ||
| else | ||
| # Extract the number of related tests | ||
| number_of_shards=$(echo "$nftest_output" | sed -n 's|.*Executed \([0-9]*\) tests.*|\1|p') |
There was a problem hiding this comment.
Here we use shards = tests strategy. This is good for long running pipeline level tests. For modules, we might want to switch to shards = tests / n as the runtime of a single module is generally comparable to action setup time.
This comment was marked as resolved.
This comment was marked as resolved.
|
Some of the CPU tests are failing due to permission issues. It might be related to |
|
Ah sorry, forgot about the cpu tests. They indeed still run on rootless docker, so need docker_self_hosted but I will try to fix that soon ™ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.