Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_auto3dseg_ensemble torch 1.9.1 #6390

Closed
wyli opened this issue Apr 18, 2023 · 0 comments · Fixed by #6372
Closed

test_auto3dseg_ensemble torch 1.9.1 #6390

wyli opened this issue Apr 18, 2023 · 0 comments · Fixed by #6372
Labels
bug Something isn't working

Comments

@wyli
Copy link
Contributor

wyli commented Apr 18, 2023

Describe the bug

[2023-04-18T16:07:21.134Z] ======================================================================
[2023-04-18T16:07:21.134Z] ERROR: test_get_history (tests.test_auto3dseg_hpo.TestHPO)
[2023-04-18T16:07:21.134Z] ----------------------------------------------------------------------
[2023-04-18T16:07:21.134Z] Traceback (most recent call last):
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_auto3dseg_hpo.py", line 134, in setUp
[2023-04-18T16:07:21.134Z]     bundle_generator.generate(work_dir, num_fold=1)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 569, in generate
[2023-04-18T16:07:21.134Z]     gen_algo.export_to_disk(output_folder, name, fold=f_id)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 160, in export_to_disk
[2023-04-18T16:07:21.134Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 119, in fill_template_config
[2023-04-18T16:07:21.134Z]     mem = get_mem_from_visible_gpus()
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 31, in get_mem_from_visible_gpus
[2023-04-18T16:07:21.134Z]     available_mem_visible_gpus.append(torch.cuda.mem_get_info(device=d)[0])
[2023-04-18T16:07:21.134Z] AttributeError: module 'torch.cuda' has no attribute 'mem_get_info'
[2023-04-18T16:07:21.134Z] 
[2023-04-18T16:07:21.134Z] ======================================================================
[2023-04-18T16:07:21.134Z] ERROR: test_run_algo (tests.test_auto3dseg_hpo.TestHPO)
[2023-04-18T16:07:21.134Z] ----------------------------------------------------------------------
[2023-04-18T16:07:21.134Z] Traceback (most recent call last):
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_auto3dseg_hpo.py", line 134, in setUp
[2023-04-18T16:07:21.134Z]     bundle_generator.generate(work_dir, num_fold=1)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 569, in generate
[2023-04-18T16:07:21.134Z]     gen_algo.export_to_disk(output_folder, name, fold=f_id)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 160, in export_to_disk
[2023-04-18T16:07:21.134Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 119, in fill_template_config
[2023-04-18T16:07:21.134Z]     mem = get_mem_from_visible_gpus()
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 31, in get_mem_from_visible_gpus
[2023-04-18T16:07:21.134Z]     available_mem_visible_gpus.append(torch.cuda.mem_get_info(device=d)[0])
[2023-04-18T16:07:21.134Z] AttributeError: module 'torch.cuda' has no attribute 'mem_get_info'
[2023-04-18T16:07:21.134Z] 
[2023-04-18T16:07:21.134Z] ======================================================================
[2023-04-18T16:07:21.134Z] ERROR: test_run_optuna (tests.test_auto3dseg_hpo.TestHPO)
[2023-04-18T16:07:21.134Z] ----------------------------------------------------------------------
[2023-04-18T16:07:21.134Z] Traceback (most recent call last):
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_auto3dseg_hpo.py", line 134, in setUp
[2023-04-18T16:07:21.134Z]     bundle_generator.generate(work_dir, num_fold=1)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 569, in generate
[2023-04-18T16:07:21.134Z]     gen_algo.export_to_disk(output_folder, name, fold=f_id)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 160, in export_to_disk
[2023-04-18T16:07:21.134Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 119, in fill_template_config
[2023-04-18T16:07:21.134Z]     mem = get_mem_from_visible_gpus()
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 31, in get_mem_from_visible_gpus
[2023-04-18T16:07:21.134Z]     available_mem_visible_gpus.append(torch.cuda.mem_get_info(device=d)[0])
[2023-04-18T16:07:21.134Z] AttributeError: module 'torch.cuda' has no attribute 'mem_get_info'
[2023-04-18T16:07:21.134Z] 
[2023-04-18T16:07:21.134Z] ======================================================================
[2023-04-18T16:07:21.134Z] ERROR: test_ensemble (tests.test_auto3dseg_ensemble.TestEnsembleBuilder)
[2023-04-18T16:07:21.134Z] ----------------------------------------------------------------------
[2023-04-18T16:07:21.134Z] Traceback (most recent call last):
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/tests/test_auto3dseg_ensemble.py", line 140, in test_ensemble
[2023-04-18T16:07:21.134Z]     bundle_generator.generate(work_dir, num_fold=1)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 569, in generate
[2023-04-18T16:07:21.134Z]     gen_algo.export_to_disk(output_folder, name, fold=f_id)
[2023-04-18T16:07:21.134Z]   File "/home/jenkins/agent/workspace/Monai-pytorch-versions/monai/apps/auto3dseg/bundle_gen.py", line 160, in export_to_disk
[2023-04-18T16:07:21.134Z]     self.fill_records = self.fill_template_config(self.data_stats_files, self.output_path, **kwargs)
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 119, in fill_template_config
[2023-04-18T16:07:21.134Z]     mem = get_mem_from_visible_gpus()
[2023-04-18T16:07:21.134Z]   File "/tmp/tmp7sdlj1x3/workdir/algorithm_templates/dints/scripts/algo.py", line 31, in get_mem_from_visible_gpus
[2023-04-18T16:07:21.134Z]     available_mem_visible_gpus.append(torch.cuda.mem_get_info(device=d)[0])
[2023-04-18T16:07:21.134Z] AttributeError: module 'torch.cuda' has no attribute 'mem_get_info'
[2023-04-18T16:07:21.134Z] 
[2023-04-18T16:07:21.134Z] ----------------------------------------------------------------------
@wyli wyli added the bug Something isn't working label Apr 18, 2023
wyli added a commit to wyli/MONAI that referenced this issue Apr 18, 2023
@wyli
fixes Project-MONAI#6390
ca85b25 
Signed-off-by: Wenqi Li <[email protected]>
@wyli wyli mentioned this issue Apr 18, 2023
Merged
7 tasks
wyli added a commit that referenced this issue Apr 18, 2023
@wyli
6371 enhance warning messages when modifying applied operations (#6372)
c30a5b9 
Fixes #6371

raise a warning message early when the applied_operations is modified,
this only becomes a critical issue when we actually use the
`applied_operations` (`InvertD` is one of the use cases).

fixes #6390
fixes #6392 

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [x] New tests added to cover the changes.
- [x] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [x] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [x] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: Wenqi Li <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

6371 enhance warning messages when modifying applied operations wyli/MONAI
1 participant
@wyli