You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Description
Sometimes, scientists will store their data on Amazon S3 buckets. Currently, the only option to create an image collection (besides it being the result of a job) is to select files or folders from the user's machine and upload them through the frontend. So in the case of data being in S3 buckets, that would mean downloading the data from S3, and then re-uploading it to WIPP, which is not ideal.
Instead, we could provide an option to import images from S3.
Proposal
From the image collection view, user should have a new option "Import from Amazon S3 bucket", a popup/modal being then displayed to configure the bucket to use, folder to import, etc.
In the backend, WIPP will use the Java AWS SDK to connect to S3, download the files in the temp folder for that collection and then the usual processing of ome.tif conversion and metadata extraction can be applied, just like for uploaded files.
TBD: usage of AWS credentials (with required encryption) vs pre-signed URLs
Additional context
This feature has been requested by @Nicholas-Schaub and others from the Polus team. A workaround for S3 support was made in the form of a WIPP plugin.
The text was updated successfully, but these errors were encountered:
Description
Sometimes, scientists will store their data on Amazon S3 buckets. Currently, the only option to create an image collection (besides it being the result of a job) is to select files or folders from the user's machine and upload them through the frontend. So in the case of data being in S3 buckets, that would mean downloading the data from S3, and then re-uploading it to WIPP, which is not ideal.
Instead, we could provide an option to import images from S3.
Proposal
From the image collection view, user should have a new option "Import from Amazon S3 bucket", a popup/modal being then displayed to configure the bucket to use, folder to import, etc.
In the backend, WIPP will use the Java AWS SDK to connect to S3, download the files in the
temp
folder for that collection and then the usual processing of ome.tif conversion and metadata extraction can be applied, just like for uploaded files.TBD: usage of AWS credentials (with required encryption) vs pre-signed URLs
Additional context
This feature has been requested by @Nicholas-Schaub and others from the Polus team. A workaround for S3 support was made in the form of a WIPP plugin.
The text was updated successfully, but these errors were encountered: