Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat/s3-for-all-clouds #950

Closed
9876691 opened this issue Jul 19, 2023 · 3 comments
Closed

feat/s3-for-all-clouds #950

9876691 opened this issue Jul 19, 2023 · 3 comments
Assignees
@rbiseck3
Labels
enhancement New feature or request

Comments

@9876691
Copy link

9876691 commented Jul 19, 2023

Is your feature request related to a problem? Please describe.

I can't connect unstructured to s3 services other than those provided by amazon.

The boto library seems to be hardcoded for aws and I couldn't find a way to override that.

We use https://min.io for obect storage on site.

Describe the solution you'd like

Ability to set the URL for where we have object storage.

Possible solution

Use the min.io python library https://min.io/docs/minio/linux/developers/python/minio-py.html

This doesn't tie you into Amazon as many cloud providers provide storage with an s3 compatible protocol.
@9876691 9876691 added the enhancement New feature or request label Jul 19, 2023
@9876691 9876691 changed the title feat/S3 compatible object feat/s3-for-all-clouds Jul 19, 2023
@MthwRobinson
Copy link
Contributor

Thanks @ianpurton ! We'll take a look at this. cc: @ron-unstructured @jackretterer

@rbiseck3 rbiseck3 mentioned this issue Oct 2, 2023
Merged
@rbiseck3 rbiseck3 self-assigned this Oct 2, 2023
@rbiseck3 rbiseck3 added the ingest label Oct 2, 2023
rbiseck3 added a commit that referenced this issue Oct 3, 2023
@rbiseck3 @ryannikolaidis
Roman/s3 minio all cloud support (#1606)
8821689 
### Description
Exposes the endpoint url as an access kwarg when using the s3 filesystem
library via the fsspec abstraction. This allows for any non-aws data
providers that support the s3 protocol to be used with the s3 connector
(i.e. minio)

Closes out #950

---------

Co-authored-by: ryannikolaidis <[email protected]>
Co-authored-by: rbiseck3 <[email protected]>
@SimranJha2408
Copy link

Has this been resolved?

@MthwRobinson
Copy link
Contributor

This was closed by #1606

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rbiseck3 rbiseck3

Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@9876691 @MthwRobinson @rbiseck3 @SimranJha2408