You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When setting up a connection to an S3 source and not using AWS but for example minio, the change in #1691 allows specifying the "endpoint-url" in source.yaml using a line that says "endpoint: https://my.s3.server.com". This setting cannot be made in the UI, as the S3 settings there does not provide a field for specifying the "endpoint-url".
It would neat to be able to make this setting from the UI and maybe directly from the CLI too, without having to manually edit the source.yaml-file.
Alternatives that I considered was to install and use "aws configure" to set the endpoint URL to use, but due to this long standing bug in the aws CLI, it doesn't work: aws/aws-cli#1270
I would like the "Amazon S3" settings to support also minio (and thereby also other compatible S3 sources), and ideally be renamed "S3" and allow configuration of the endpoint-url.
It would be nice if in addition (ie if not specified) the endpoint URL could be picked up from an environment variable. I also think the "minio client" is nicer to configure rather than the aws CLI if one needs to access data from various different S3 sources (other than just AWS).
The text was updated successfully, but these errors were encountered:
Related to #1161. "Object storage" which is not distantly located in the outer clouds provide faster latency due to smaller network lags and minio is a component often used to "bring data closer to computations", recently with support also for "legacy" protocols like ftp/sftp.
When setting up a connection to an S3 source and not using AWS but for example minio, the change in #1691 allows specifying the "endpoint-url" in source.yaml using a line that says "endpoint: https://my.s3.server.com". This setting cannot be made in the UI, as the S3 settings there does not provide a field for specifying the "endpoint-url".
It would neat to be able to make this setting from the UI and maybe directly from the CLI too, without having to manually edit the source.yaml-file.
Alternatives that I considered was to install and use "aws configure" to set the endpoint URL to use, but due to this long standing bug in the aws CLI, it doesn't work: aws/aws-cli#1270
I would like the "Amazon S3" settings to support also minio (and thereby also other compatible S3 sources), and ideally be renamed "S3" and allow configuration of the endpoint-url.
It would be nice if in addition (ie if not specified) the endpoint URL could be picked up from an environment variable. I also think the "minio client" is nicer to configure rather than the aws CLI if one needs to access data from various different S3 sources (other than just AWS).
The text was updated successfully, but these errors were encountered: