Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make missing pynvml an error for GPU tests #441

Merged
merged 1 commit into from
Jul 5, 2022

Conversation

bryevdv
Copy link
Contributor

@bryevdv bryevdv commented Jul 5, 2022

This PR makes a small change to the test runner to make pynvml required for stages that access GPU information. Previously, if pynvml was missing, and the GPU stage was run, system.gpus would return an empty list of GPU specs, resulting in a cryptic error suggesting that there were no GPUs present (even if there were):

Traceback (most recent call last):
  File "./test.py", line 30, in <module>
    plan = TestPlan(config, system)
  File "/home/idemeshko/development/cunumeric_clean/tests/_utils/test_plan.py", line 45, in __init__
    self._stages = [
  File "/home/idemeshko/development/cunumeric_clean/tests/_utils/test_plan.py", line 46, in <listcomp>
    STAGES[feature](config, system) for feature in config.features
  File "/home/idemeshko/development/cunumeric_clean/tests/_utils/test_stages/gpu.py", line 43, in __init__
    self.workers = self.compute_workers(config, system)
  File "/home/idemeshko/development/cunumeric_clean/tests/_utils/test_stages/gpu.py", line 66, in compute_workers
    assert len(gpus)
AssertionError

With this change, a GPU stage will instead fail with a clear import error:

dev38 ❯ ./test.py --use=cuda                                  
Traceback (most recent call last):
  File "./test.py", line 30, in <module>
    plan = TestPlan(config, system)
  File "/home/bryan/work/cunumeric/tests/_utils/test_plan.py", line 45, in __init__
    self._stages = [
  File "/home/bryan/work/cunumeric/tests/_utils/test_plan.py", line 46, in <listcomp>
    STAGES[feature](config, system) for feature in config.features
  File "/home/bryan/work/cunumeric/tests/_utils/test_stages/gpu.py", line 43, in __init__
    self.workers = self.compute_workers(config, system)
  File "/home/bryan/work/cunumeric/tests/_utils/test_stages/gpu.py", line 64, in compute_workers
    gpus = system.gpus
  File "/home/bryan/work/cunumeric/tests/_utils/system.py", line 102, in gpus
    import pynvml  # type: ignore[import]
ModuleNotFoundError: No module named 'pynvml'

Non-GPU stages e.g "eager" or "cpus" are unaffected and continue to run with or without pynvml installed.

@bryevdv bryevdv requested a review from ipdemes July 5, 2022 17:33
Copy link
Contributor

@ipdemes ipdemes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bryevdv bryevdv merged commit 13a7dc7 into nv-legate:branch-22.07 Jul 5, 2022
@bryevdv bryevdv deleted the bryanv/missing_pynvml branch July 5, 2022 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants