Standardising atlas file downloading #351

viktorpm · 2024-08-09T14:21:14Z

Description

This PR introduces a standardized method for downloading atlas files using pooch.

Bug fix
Addition of a new feature
Other

How has this PR been tested?

Scripts were tested locally

Is this a breaking change?

No

Does this PR require an update to the documentation?

Potentially

Checklist:

The code has been tested locally
Tests have been added to cover all new functionality (unit & integration)
The documentation has been updated to reflect any changes
The code has been formatted with pre-commit

…uring packaging

for more information, see https://pre-commit.ci

alessandrofelder

Thanks a lot @viktorpm - this is good progress!
And sorry for letting this sit so long 🙈

I've suggested some minor changes and asked for minor clarifications.

Some high-level comments

there's quite a bit of commented out code that I think should be removed
I'm confused by the ### comments, but if we want to keep them we should add them everywhere.

We should open issues for

removing the current way of parallelising mesh creation everywhere
decide what to do with __version__ and __atlas__
make mesh creation parallel in a more robust way

What do you think?

alessandrofelder · 2024-10-28T14:54:57Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/allen_cord.py

@@ -1,52 +1,58 @@
-__version__ = "1"
-
+### Import


Are these comments useful and why do they start with ###? I may be missing something. If they are useful, should they be added to the admba_3d_dev_mouse script too?

alessandrofelder · 2024-10-28T14:57:29Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/allen_cord.py

+    # ATLAS_NAME = "allen_cord"
+    # SPECIES = "Mus musculus"
+    # ATLAS_LINK = "https://data.mendeley.com/datasets/4rrggzv5d5/1"
+    # CITATION = (
+    #     "Fiederling et al. 2021, https://doi.org/10.1101/2021.05.06.443008"
+    # )
+    # ORIENTATION = "asr"
+    # RESOLUTION = (20, 10, 10)
+    # ROOT_ID = 250
+    # ATLAS_FILE_URL = "https://prod-dcd-datasets-cache-zipfiles.s3.eu-west-1.amazonaws.com/4rrggzv5d5-1.zip"
+    # ATLAS_PACKAGER = "MetaCell LLC, Ltd."


Suggested change

# ATLAS_NAME = "allen_cord"

# SPECIES = "Mus musculus"

# ATLAS_LINK = "https://data.mendeley.com/datasets/4rrggzv5d5/1"

# CITATION = (

# "Fiederling et al. 2021, https://doi.org/10.1101/2021.05.06.443008"

# )

# ORIENTATION = "asr"

# RESOLUTION = (20, 10, 10)

# ROOT_ID = 250

# ATLAS_FILE_URL = "https://prod-dcd-datasets-cache-zipfiles.s3.eu-west-1.amazonaws.com/4rrggzv5d5-1.zip"

# ATLAS_PACKAGER = "MetaCell LLC, Ltd."

I think this can be removed instead of just commenting it out.

alessandrofelder · 2024-10-28T15:02:20Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/azba_zfish.py

-
-    working_dir = working_dir / ATLAS_NAME
+
+    working_dir = working_dir


Suggested change

working_dir = working_dir

I think this line is superfluous?

alessandrofelder · 2024-10-28T15:04:36Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/humanatlas.py

+ORIENTATION = "rpi"
+
+### Settings
+PARALLEL = True  # disable parallel mesh extraction for easier debugging


Does parallelisation work for this atlas? It's probably not good to have it as a default if not... and recently we've been moving away from parallelisation here?

alessandrofelder · 2024-10-28T15:07:36Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/humanatlas.py

@@ -251,6 +255,8 @@ def prune_tree(tree):
            total=tree.size(),
            description="Creating meshes",
        ):
+
+            # root_node = tree.nodes[ROOT_ID]


Suggested change

# root_node = tree.nodes[ROOT_ID]

We should remove unused code.

alessandrofelder · 2024-10-28T15:14:15Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/mpin_zfish.py

@@ -1,9 +1,9 @@
 __version__ = "1"


We should be consistent if we want to move the version away from here (but I'm not sure we do?)

alessandrofelder · 2024-10-28T15:17:08Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/osten_mouse.py

    # Temporary folder for  download:
    download_dir_path = working_dir / "downloads"
    download_dir_path.mkdir(exist_ok=True)
-    atlas_files_dir = download_dir_path / "atlas_files"
+    # atlas_files_dir = download_dir_path / "atlas_files"


Suggested change

# atlas_files_dir = download_dir_path / "atlas_files"

Remove unused code.

alessandrofelder · 2024-10-28T15:20:20Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/perens_lsfm_mouse.py

+ATLAS_FILE_URL = (
+    "https://github.com/Gubra-ApS/LSFM-mouse-brain-atlas/archive/master.tar.gz"
+)
+RESOLUTION = 20  # some resolution, in microns


I don't understand the comment - my best guess is that it's a relic from someone explaining the template script to themselves? If you agree, @viktorpm let's remove (if not, please clarify the comment)

alessandrofelder · 2024-10-28T15:21:02Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/perens_lsfm_mouse.py

    # Temporary folder for  download:
    download_dir_path = working_dir / "downloads"
    download_dir_path.mkdir(exist_ok=True)
-    atlas_files_dir = download_dir_path / "atlas_files"
+    # atlas_files_dir = download_dir_path / "atlas_files"


Suggested change

# atlas_files_dir = download_dir_path / "atlas_files"

Remove unused code

alessandrofelder · 2024-10-28T15:32:39Z

brainglobe_atlasapi/atlas_generation/atlas_scripts/whs_sd_rat.py

@@ -217,7 +221,7 @@ def create_atlas(working_dir):
    ), "No download link provided for atlas in ATLAS_FILE_URL"

    # Generated atlas path:
-    working_dir = working_dir / ATLAS_NAME
+    working_dir = working_dir


Suggested change

working_dir = working_dir

I think this line is superfluous?

viktorpm and others added 8 commits August 9, 2024 14:59

adding known_hash argument to AtlasConfig

6f740cd

defining DEFAULT_WORKDIR in config.py to standardise atlas location d…

f348196

…uring packaging

adding DEFAULT_WORKDIR and __atlas__ to packaging scripts

dfb0db0

implementing pooch

a935fde

implementing pooch

25c844b

complete refactoring of humanatlas.py, implementing pooch

14007f8

Merge branch 'main' into standardising_atlas_file_downloading

a0d6378

[pre-commit.ci] auto fixes from pre-commit.com hooks

9623cdb

for more information, see https://pre-commit.ci

viktorpm marked this pull request as ready for review September 4, 2024 10:50

viktorpm added 2 commits September 6, 2024 10:55

fixing file downloading

14b87e5

Restructuring code: metadata on top

31c5527

viktorpm requested a review from alessandrofelder September 16, 2024 14:30

alessandrofelder requested changes Oct 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Standardising atlas file downloading #351

Standardising atlas file downloading #351

viktorpm commented Aug 9, 2024 •

edited

Loading

alessandrofelder left a comment •

edited

Loading

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024

alessandrofelder Oct 28, 2024


		working_dir = working_dir / ATLAS_NAME

		working_dir = working_dir

Standardising atlas file downloading #351

Are you sure you want to change the base?

Standardising atlas file downloading #351

Conversation

viktorpm commented Aug 9, 2024 • edited Loading

Description

How has this PR been tested?

Is this a breaking change?

Does this PR require an update to the documentation?

Checklist:

alessandrofelder left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

viktorpm commented Aug 9, 2024 •

edited

Loading

alessandrofelder left a comment •

edited

Loading