Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

handle running non-GPU jobs on GPU partitions #115

Closed
smoors opened this issue Feb 11, 2024 · 0 comments · Fixed by #116
Closed

handle running non-GPU jobs on GPU partitions #115

smoors opened this issue Feb 11, 2024 · 0 comments · Fixed by #116

Comments

@smoors
Copy link
Collaborator

smoors commented Feb 11, 2024

in some clusters/partitions, when submitting job in a GPU partition, you are required to request at least one GPU.
in our test suite we generate tests for running non-GPU jobs on GPU partitions that have the 'cpu' feature defined.
for those tests a GPU should not be requested unless required by the rules of the cluster.

currently, our tests act as follows:

  • the gromacs and tensorflow tests do not request any GPU for such jobs, which will fail in clusters/partitions that require it
  • the osu test, on the other hand, always requests one or more GPUs for such tests, depending on the number of cores requested.

as this is a cluster/partition-specific policy, ideally this would be a partition-specific setting in the site config file.
for example, we could add a feature must_request_gpus to the partitions that require it,
and add a check in the hooks for the presence of this feature via self.current_partition.features.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant