[Merged by Bors] - Fix OOM error with manual buffer size specification #380
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fhennig
changed the title
5 tasks
|
The PR is ready for review, although operator-rs still needs to be updated |
|
I've started a Test Run |
maltesander
reviewed
Feb 1, 2023
There was a problem hiding this comment.
Finished testing, works fine!
Some points:
- Bump the versions in the test-definition to stackable23.1.0-rc1?
- Error messages could provide better info about what is happening (see other comments) or
- Add checks to reject values that will definitely fail to start up? (e.g. 10Mi as memory)
bors bot
pushed a commit
to stackabletech/operator-rs
that referenced
this pull request
Feb 1, 2023
## Description I have used these functions in my Druid PR already: stackabletech/druid-operator#380 Co-authored-by: Felix Hennig <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
Co-authored-by: Malte Sander <[email protected]>
maltesander
reviewed
Feb 1, 2023
|
Nice, thx! |
|
bors merge |
bors bot
pushed a commit
that referenced
this pull request
Feb 2, 2023
# Description fixes #359 automatic buffer sizing seems to be the problem in #359 This PR requires a new operator-rs version with the operator-rs features from this PR stackabletech/operator-rs#544 This PR is intended to fix this, by calculating a buffer size based on the user provided memory request. Druid guidelines are used to compute all values. We also set the available `MaxDirectAccessMemory` dynamically, instead of hardcoding it. The heap calculation now incorporates this value, so now it shouldn't happen anymore that the JVM tries to allocate more than there is available to the container. The computation has a few other benefits: - maximum buffer size can now be 2GB instead of 1GB (because we're not using druid `auto` anymore) - a fix memory quantity is reserved for the OS, not a scaled one. This saves memory if more memory is available - there we're bugs that didn't surface yet with the MaxDirectAccessMemory of all roles. All of them are fixed now, the MaxDirectAccessMemory + Heap Memory does not exceed the maximum memory available anymore. - The rounding used for the JVM memory is not unit-dependent anymore. Before it would round of .2 of _whatever_ quantity, so there might be 200MB sitting around unused. This is not the case anymore. - If the direct access memory is maxed out, all the remaining memory will be allocated as Heap, to make maxium use of the allocatable memory. This is also for all roles.
|
Pull request successfully merged into main. Build succeeded: |
bors
bot
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
fixes #359
automatic buffer sizing seems to be the problem in #359
This PR requires a new operator-rs version with the operator-rs features from this PR stackabletech/operator-rs#544
This PR is intended to fix this, by calculating a buffer size based on the user provided memory request. Druid guidelines are used to compute all values.
We also set the available
MaxDirectAccessMemorydynamically, instead of hardcoding it. The heap calculation now incorporates this value, so now it shouldn't happen anymore that the JVM tries to allocate more than there is available to the container.The computation has a few other benefits:
autoanymore)Review Checklist
Once the review is done, comment
bors r+(orbors merge) to merge. Further information