[Merged by Bors] - Feature/s3 extension defaults #192
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I'm gonna review this after lunch |
fhennig
suggested changes
Mar 8, 2022
|
I think the "Integration tests added" is a good point here, I see that the current integration test has a Druid cluster definition with the s3 section defined, we should have a second test without the s3 case to test both. I think that test is part of the ticket here and it would need to be created. Edit: At the moment our ingestion test uses a dummy s3 config. We should remove that since it should not be needed anymore with this PR. And i think it would be good to have two ingestion tests:
It would also be good to split out the smoke test again, I think it's good to have that as a seperate test. Sorry for all my change requests! |
|
bors merge |
bors bot
pushed a commit
that referenced
this pull request
Mar 9, 2022
## Description It is not immediately evident from the Apache Druid documentation that only one s3-endpoint can be used per druid cluster: the endpoint cannot be entered in the console UI and must be supplied as a runtime property (only buckets/baseKeys can be entered in the console). The credentials can be passed either as environment variables, as runtime properties, or directly via the UI, so they do not need to exist as runtime properties necessarily. The upshot of this is that the druid-s3-extension requires an s3-endpoint for it to be initialized: making the loading of the s3-extension conditional upon the provision of an `s3:endpoint` in our CR means that s3-functionality will only be activated in the console if it can be used (i.e. that an endpoint is also available). The following have been tested: - quickstart: include s3-extension in list and set a value for `druid.s3.endpoint.url` - ingest sample- and s3-data (adding credentials/bucket in UI) - operator + hdfs-deep-storage + without s3 definition in CR: ingestion of sample data - operator + hdfs-deep-storage + with s3 definition in CR: ingestion of s3-data - operator + s3-deep-storage: ingestion of s3 and non-s3 data - existing integration tests
|
Pull request successfully merged into main. Build succeeded: |
bors
bot
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
It is not immediately evident from the Apache Druid documentation that only one s3-endpoint can be used per druid cluster: the endpoint cannot be entered in the console UI and must be supplied as a runtime property (only buckets/baseKeys can be entered in the console). The credentials can be passed either as environment variables, as runtime properties, or directly via the UI, so they do not need to exist as runtime properties necessarily.
The upshot of this is that the druid-s3-extension requires an s3-endpoint for it to be initialized: making the loading of the s3-extension conditional upon the provision of an
s3:endpointin our CR means that s3-functionality will only be activated in the console if it can be used (i.e. that an endpoint is also available).The following have been tested:
druid.s3.endpoint.urlReview Checklist
Once the review is done, comment
bors r+(orbors merge) to merge. Further information