S3 Ingestion from non-default endpoints #11798
Conversation
docs/ingestion/native-batch.md
Outdated
| }, | ||
|
|
||
| "properties": { | ||
| "accessKeyId": "KLJ78979SDFdS2", |
There was a problem hiding this comment.
Replace with xxxx or something ?
There was a problem hiding this comment.
Thanks for taking a look. These are fake credentials.
|
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 4 weeks if no further activity occurs. If you think that's incorrect or this pull request should instead be reviewed, please simply write any comment. Even if closed, you can still revive the PR at any time or discuss it on the [email protected] list. Thank you for your contributions. |
| @@ -98,16 +109,49 @@ public S3InputSource( | |||
| @JsonProperty("objects") @Nullable List<CloudObjectLocation> objects, | |||
| @JsonProperty("filter") @Nullable String filter, | |||
| @JsonProperty("properties") @Nullable S3InputSourceConfig s3InputSourceConfig, | |||
| @JsonProperty("proxyConfig") @Nullable AWSProxyConfig awsProxyConfig, | |||
| @JsonProperty("endpointConfig") @Nullable AWSEndpointConfig awsEndpointConfig, | |||
| @JsonProperty("clientConfig") @Nullable AWSClientConfig awsClientConfig, | |||
There was a problem hiding this comment.
Reusing the Config objects here so that we don't duplicate the properties. We could push these under S3InputSourceConfig itself if needed.
|
@maytasm Could you please take a look when you get a chance? Thanks! |
|
@a2l007 Approved. Looks like there is a test failure in intellij inspections. |
| "host='" + host + '\'' + | ||
| ", port=" + port + | ||
| ", username='" + username + '\'' + | ||
| ", password='" + password + '\'' + |
There was a problem hiding this comment.
Minor nit: Should we toString the password here.
There was a problem hiding this comment.
Good catch, removed it.
|
Thanks for the review @cryptoe |
This reverts commit 7504597.
* Add endpoint support for s3inputsource * Changes to tests * Fix docs * Fix config * Fix inspections * Fix spelling * Remove password from toString
Description
S3InputSourcepresently provides an option to override the default S3 credentials which is useful to read from different buckets, but it has to be in the same endpoint. This PR adds functionality to specify a different endpoint and associated client properties as part of theS3InputSourcespec. This can be helpful for operators looking to migrate data from one store to another. This is implemented only for S3 at the moment, but we can support other cloud providers as a followup.Key changed/added classes in this PR
S3InputSourceThis PR has: