-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tf32 warnings #6816
Tf32 warnings #6816
Conversation
Signed-off-by: Qingpeng Li <[email protected]>
Signed-off-by: Qingpeng Li <[email protected]>
when a function/class is imported Signed-off-by: Qingpeng Li <[email protected]>
Signed-off-by: Qingpeng Li <[email protected]>
for more information, see https://pre-commit.ci
Signed-off-by: Qingpeng Li <[email protected]>
…into tf32-warnings
Signed-off-by: Qingpeng Li <[email protected]>
/black |
Signed-off-by: monai-bot <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks! please see some minor inline comments
Signed-off-by: Qingpeng Li <[email protected]>
Signed-off-by: Qingpeng Li <[email protected]>
Signed-off-by: monai-bot <[email protected]>
/black |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, it looks good to me, testing with more environments
The failed test is caused by #2161 (also see kornia/kornia#1951). Pytorch only initializes |
Signed-off-by: Qingpeng Li <[email protected]>
…into tf32-warnings
/black |
for more information, see https://pre-commit.ci
Signed-off-by: monai-bot <[email protected]>
ok... in this case how about only calling MONAI/monai/config/deviceconfig.py Line 195 in 65cf5fe
currently the |
/build |
This failed case is kind of interesting🤔 My test env is |
The key of Actually, I have an ugly but safe solution: |
As the disscussion above, we prefer to use alt 2. Feel free to add any comment on alt 2. If alt 2. is okay, I can push an alt 2. commit. Looks like PyNVML is a safe choice. There are some similar usages in PyTorch, such as https://github.com/pytorch/pytorch/blob/v2.0.1/torch/cuda/__init__.py#L771-L794 |
sure, I think we can go for option 2 if it works fine across the test platforms and the import timing doesn't increase too much |
Signed-off-by: Qingpeng Li <[email protected]>
Signed-off-by: Qingpeng Li <[email protected]>
/black |
for more information, see https://pre-commit.ci
Signed-off-by: monai-bot <[email protected]>
/build |
Signed-off-by: Qingpeng Li <[email protected]>
…into tf32-warnings
for more information, see https://pre-commit.ci
Signed-off-by: monai-bot <[email protected]>
/build |
Following #6816 ### Description make `is_tf32_env()` safer. check `cuda` to prevent fallthrough case when `pynvml` is not found ### Types of changes <!--- Put an `x` in all the boxes that apply, and remove the not applicable items --> - [x] Non-breaking change (fix or new feature that would not break existing functionality). - [ ] Breaking change (fix or new feature that would cause existing functionality to change). - [ ] New tests added to cover the changes. - [ ] Integration tests passed locally by running `./runtests.sh -f -u --net --coverage`. - [ ] Quick tests passed locally by running `./runtests.sh --quick --unittests --disttests`. - [ ] In-line docstrings updated. - [ ] Documentation updated, tested `make html` command in the `docs/` folder. Signed-off-by: Qingpeng Li <[email protected]>
about #6754 .
Description
show a warning if any thing may enable tf32 is detected
Types of changes
./runtests.sh -f -u --net --coverage
../runtests.sh --quick --unittests --disttests
.make html
command in thedocs/
folder.