Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add scripts/managed_repo_symlinks.py #54

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

will-moore
Copy link
Member

@will-moore will-moore commented Apr 20, 2023

For the importing-NGFF-Filesets-without-chunks workflow, we import a metadata-only copy of the image/plate, then need to view an image from each plate before creating symlinks from the Managed Repo to the directory that contains the image/plate.

E.g. see README at https://github.com/IDR/idr0125-way-cellpainting

This PR aims to replace the following manual steps described there...

  • Need to view an image from each plate OR omero render set Image:ID rdef.yml for each plate, to avoid subsequent ResourceError after symlinking. See scripts/render_set_cmd.py

  • Run scripts/get_import_paths.py to create imported_paths.txt with the managed repo path for each imported plate.

  • Run scripts/symlink_cmd.py which consumes imported_paths.txt to output the commands to replace each plate in the managed repo with a symlink to the corresponding plate on a mounted s3 bucket. These can be saved in scripts/symlinks.bash. Need to run sudo chmod +x symlinks.bash then sudo -u omero-server -s and bash symlinks.bash.

...with a single script run:

E.g. For a Screen that contains Plates, where the plates are mounted at /path/to/plates/dir/ we can run:

Default repo is /data/OMERO/ManagedRepository
Script also supports the --dry-run and --report options.

$ sudo -u omero-server -s
$ source /opt/omero/server/venv3/bin/activate
$ python idr-utils/scripts/managed_repo_symlinks Screen:123 /path/to/plates/dir/ --repo /path/to/ManagedRepository --report

@will-moore
Copy link
Member Author

As discussed in IDR meeting yesterday: don't expect to use the omero import workflow for NGFF data since this will be too slow.

@will-moore will-moore closed this May 23, 2023
@will-moore will-moore reopened this May 29, 2023
@will-moore
Copy link
Member Author

Reopened PR for now: Using this branch to update symlinks following upload to BioStudies: see IDR/idr-metadata#659 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant