Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support import from AWS S3 bucket for image collections #87

Open
MyleneSimon opened this issue Sep 24, 2019 · 0 comments
Open

Support import from AWS S3 bucket for image collections #87

MyleneSimon opened this issue Sep 24, 2019 · 0 comments
Labels
feature New feature or request

Comments

@MyleneSimon
Copy link
Collaborator

Description
Sometimes, scientists will store their data on Amazon S3 buckets. Currently, the only option to create an image collection (besides it being the result of a job) is to select files or folders from the user's machine and upload them through the frontend. So in the case of data being in S3 buckets, that would mean downloading the data from S3, and then re-uploading it to WIPP, which is not ideal.
Instead, we could provide an option to import images from S3.

Proposal
From the image collection view, user should have a new option "Import from Amazon S3 bucket", a popup/modal being then displayed to configure the bucket to use, folder to import, etc.
In the backend, WIPP will use the Java AWS SDK to connect to S3, download the files in the temp folder for that collection and then the usual processing of ome.tif conversion and metadata extraction can be applied, just like for uploaded files.
TBD: usage of AWS credentials (with required encryption) vs pre-signed URLs

Additional context
This feature has been requested by @Nicholas-Schaub and others from the Polus team. A workaround for S3 support was made in the form of a WIPP plugin.

@MyleneSimon MyleneSimon added the feature New feature or request label Sep 24, 2019
@MyleneSimon MyleneSimon added this to the 3.0.0-RC1 milestone Sep 24, 2019
@MyleneSimon MyleneSimon removed this from the 3.0.0-beta2 milestone Feb 7, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant