Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add cache option in GridPatchDataset #7180

Merged
merged 32 commits into from
Nov 17, 2023
Merged

Add cache option in GridPatchDataset #7180

merged 32 commits into from
Nov 17, 2023

Conversation

KumoLiu
Copy link
Contributor

@KumoLiu KumoLiu commented Nov 1, 2023 •
edited
Loading

Part of #6904

Description

  • Fix inefficient patching in PatchDataset
  • Add cache option in GridPatchDataset

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
  • Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
  • In-line docstrings updated.
  • Documentation updated, tested make html command in the docs/ folder.

@KumoLiu
Copy link
Contributor Author

KumoLiu commented Nov 2, 2023

Hi @wyli @ericspod, could you please help review the PR, I haven't updated the docstring and unit tests yet.
In this PR, I mainly added:

  • add cache support in GridPatchDataset, if set cache=True, then the patches from data * cache_num will be cached
  • Update PatchDataset to inherit from IterableDataset to avoid duplicated patching
  • GridPatchDataset and PatchDataset can also accept file names instead of only accepting already loaded data.

@KumoLiu KumoLiu requested review from wyli, ericspod and Nic-Ma November 2, 2023 07:46
ericspod
ericspod reviewed Nov 2, 2023
View reviewed changes
monai/data/grid_dataset.py Outdated Show resolved Hide resolved
ericspod
ericspod reviewed Nov 2, 2023
View reviewed changes
monai/data/grid_dataset.py Outdated Show resolved Hide resolved
ericspod
ericspod reviewed Nov 2, 2023
View reviewed changes
monai/data/grid_dataset.py Outdated Show resolved Hide resolved
ericspod
ericspod reviewed Nov 2, 2023
View reviewed changes
monai/data/grid_dataset.py Outdated Show resolved Hide resolved
ericspod
ericspod reviewed Nov 2, 2023
View reviewed changes
monai/data/grid_dataset.py Outdated Show resolved Hide resolved
KumoLiu and others added 9 commits November 3, 2023 13:08
@KumoLiu
Merge branch 'dev' of https://github.com/Project-MONAI/MONAI into dat…
6314848 
…aset
@KumoLiu @ericspod
Update monai/data/grid_dataset.py
7b79b34 
Co-authored-by: Eric Kerfoot <[email protected]>
Signed-off-by: YunLiu <[email protected]>
@KumoLiu
Merge branch 'dataset' of https://github.com/KumoLiu/MONAI into dataset
397907d
@KumoLiu
address comments
f3b3b98 
Signed-off-by: KumoLiu <[email protected]>
@KumoLiu
Merge branch 'dev' of https://github.com/Project-MONAI/MONAI into dat…
0282de4 
…aset
@pre-commit-ci
[pre-commit.ci] auto fixes from pre-commit.com hooks
caec700 
for more information, see https://pre-commit.ci
@KumoLiu
add _generate_patches
e15aeab 
Signed-off-by: KumoLiu <[email protected]>
@KumoLiu
add unittests
acac03d 
Signed-off-by: KumoLiu <[email protected]>
@KumoLiu
Merge remote-tracking branch 'origin/dev' into dataset
28a82a0
@KumoLiu KumoLiu requested a review from ericspod November 8, 2023 14:14
ericspod
ericspod reviewed Nov 9, 2023
View reviewed changes
monai/data/grid_dataset.py Outdated Show resolved Hide resolved
@KumoLiu
remove unused import
1bb1ae7 
Signed-off-by: KumoLiu <[email protected]>
@KumoLiu
update docstring
520e2b3 
Signed-off-by: KumoLiu <[email protected]>
@KumoLiu KumoLiu marked this pull request as ready for review November 10, 2023 10:53
@KumoLiu
fix mypy
32d4412 
Signed-off-by: KumoLiu <[email protected]>
@Nic-Ma
Copy link
Contributor

Nic-Ma commented Nov 15, 2023

/build

@KumoLiu KumoLiu requested a review from ericspod November 16, 2023 05:47
@ericspod
Copy link
Member

Hi @KumoLiu I have a few suggestions that aren't critical but more importantly there's a thread-safety issue we need to fix, otherwise I think we're good. Thanks!

ericspod
ericspod approved these changes Nov 17, 2023
View reviewed changes
Copy link
Member

@ericspod ericspod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me now, thanks!

@ericspod ericspod enabled auto-merge (squash) November 17, 2023 13:18
@KumoLiu
Copy link
Contributor Author

KumoLiu commented Nov 17, 2023

/build

@ericspod ericspod merged commit dfe0b40 into Project-MONAI:dev Nov 17, 2023
28 checks passed
@KumoLiu KumoLiu deleted the dataset branch November 17, 2023 14:15
@KumoLiu KumoLiu mentioned this pull request Nov 20, 2023
Merged
5 tasks
ericspod pushed a commit to Project-MONAI/tutorials that referenced this pull request Nov 20, 2023
@KumoLiu
Fix shuffle error in 2d_slices_from_3d_sampling.ipynb (#1577)
c954aaa 
Fix #1576

The error was introduced by
Project-MONAI/MONAI#7180.

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <[email protected]>
marksgraham pushed a commit to marksgraham/MONAI that referenced this pull request Jan 30, 2024
@KumoLiu @ericspod
@pre-commit-ci @marksgraham
Add cache option in GridPatchDataset (Project-MONAI#7180)
bf166b5 
Part of Project-MONAI#6904

### Description
- Fix inefficient patching in `PatchDataset`
- Add cache option in `GridPatchDataset`

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: KumoLiu <[email protected]>
Signed-off-by: YunLiu <[email protected]>
Co-authored-by: Eric Kerfoot <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Mark Graham <[email protected]>
juampatronics pushed a commit to juampatronics/MONAI that referenced this pull request Mar 25, 2024
@KumoLiu @ericspod
@pre-commit-ci @juampatronics
Add cache option in GridPatchDataset (Project-MONAI#7180)
36beba1 
Part of Project-MONAI#6904

### Description
- Fix inefficient patching in `PatchDataset`
- Add cache option in `GridPatchDataset`

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: KumoLiu <[email protected]>
Signed-off-by: YunLiu <[email protected]>
Co-authored-by: Eric Kerfoot <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Juan Pablo de la Cruz Gutiérrez <[email protected]>
Yu0610 pushed a commit to Yu0610/MONAI that referenced this pull request Apr 11, 2024
@KumoLiu @ericspod
@pre-commit-ci @Yu0610
Add cache option in GridPatchDataset (Project-MONAI#7180)
c2b2fd6 
Part of Project-MONAI#6904

### Description
- Fix inefficient patching in `PatchDataset`
- Add cache option in `GridPatchDataset`

### Types of changes
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [x] Non-breaking change (fix or new feature that would not break
existing functionality).
- [ ] Breaking change (fix or new feature that would cause existing
functionality to change).
- [ ] New tests added to cover the changes.
- [ ] Integration tests passed locally by running `./runtests.sh -f -u
--net --coverage`.
- [ ] Quick tests passed locally by running `./runtests.sh --quick
--unittests --disttests`.
- [ ] In-line docstrings updated.
- [ ] Documentation updated, tested `make html` command in the `docs/`
folder.

---------

Signed-off-by: KumoLiu <[email protected]>
Signed-off-by: YunLiu <[email protected]>
Co-authored-by: Eric Kerfoot <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Yu0610 <[email protected]>
boneseva pushed a commit to boneseva/MONAI-tutorials that referenced this pull request Apr 21, 2024
@KumoLiu
Fix shuffle error in 2d_slices_from_3d_sampling.ipynb (Project-MONA…
a4e80f0 
…I#1577)

Fix Project-MONAI#1576

The error was introduced by
Project-MONAI/MONAI#7180.

### Checks
<!--- Put an `x` in all the boxes that apply, and remove the not
applicable items -->
- [ ] Avoid including large-size files in the PR.
- [ ] Clean up long text outputs from code cells in the notebook.
- [ ] For security purposes, please check the contents and remove any
sensitive info such as user names and private key.
- [ ] Ensure (1) hyperlinks and markdown anchors are working (2) use
relative paths for tutorial repo files (3) put figure and graphs in the
`./figure` folder
- [ ] Notebook runs automatically `./runner.sh -t <path to .ipynb file>`

Signed-off-by: KumoLiu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericspod ericspod ericspod approved these changes

@wyli wyli Awaiting requested review from wyli

@Nic-Ma Nic-Ma Awaiting requested review from Nic-Ma

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@KumoLiu @Nic-Ma @ericspod