-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enhanced bytes limiter with data type param #7414
Conversation
f910fdc
to
6b0040d
Compare
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
Signed-off-by: Justin Jung <[email protected]>
74d2267
to
93136a7
Compare
Signed-off-by: Justin Jung <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Just a small nit on the changelog
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great work Justin!
Signed-off-by: Justin Jung <[email protected]>
Maybe @fpetkovski @MichaHoffmann or @saswatamcode can help take a look? Thanks |
I will merge this change tomorrow if there is no objection or new review comments. Since this change is not user facing at all, we can address in future PRs if you have any other comments |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks, generally LGTM! I think it can be put in changelog as your Store error will now tell you the type of data that exceeded bytes limit?
Also any plans to expose these to user somehow?
Thanks for taking a look @saswatamcode !
I haven't changed the error message returned to users, as they already contained the data type information. The main goal of this change was to enhance the limiter logic with access to the data type, which can be hidden from the users.
Yes! We have been noticing that resource consumption (CPU and memory) for postings, series and chunks are quite different. For example, we were able to fetch up to 1GB/s of chunks until the CPU is exhausted, while we were only able to fetch 258MB/s of series. We should definitely update the limiter logic to assign different weights per data type. |
* Refactor existing stats incrementation for touched and fetched data Signed-off-by: Justin Jung <[email protected]> * Add TypedBytesLimiter Signed-off-by: Justin Jung <[email protected]> * Remove addAndCheck func Signed-off-by: Justin Jung <[email protected]> * Update BytesLimiter interface to accept dataType param Signed-off-by: Justin Jung <[email protected]> * Added tests Signed-off-by: Justin Jung <[email protected]> * Fix build + changelog Signed-off-by: Justin Jung <[email protected]> * Fix wrong data type Signed-off-by: Justin Jung <[email protected]> * Changed storeDataType to be exported Signed-off-by: Justin Jung <[email protected]> * Revert []BytesLimiter to BytesLimtier Signed-off-by: Justin Jung <[email protected]> * Lint Signed-off-by: Justin Jung <[email protected]> * More reverts Signed-off-by: Justin Jung <[email protected]> * More Signed-off-by: Justin Jung <[email protected]> * Rename DefaultBytesLimiterFactory back to NewBytesLimiterFactory Signed-off-by: Justin Jung <[email protected]> * Changed StoreDataType from string to int Signed-off-by: Justin Jung <[email protected]> * Removed nil check for bytesLimiter Signed-off-by: Justin Jung <[email protected]> * nit Signed-off-by: Justin Jung <[email protected]> * Removed changelog Signed-off-by: Justin Jung <[email protected]> --------- Signed-off-by: Justin Jung <[email protected]>
* Refactor existing stats incrementation for touched and fetched data Signed-off-by: Justin Jung <[email protected]> * Add TypedBytesLimiter Signed-off-by: Justin Jung <[email protected]> * Remove addAndCheck func Signed-off-by: Justin Jung <[email protected]> * Update BytesLimiter interface to accept dataType param Signed-off-by: Justin Jung <[email protected]> * Added tests Signed-off-by: Justin Jung <[email protected]> * Fix build + changelog Signed-off-by: Justin Jung <[email protected]> * Fix wrong data type Signed-off-by: Justin Jung <[email protected]> * Changed storeDataType to be exported Signed-off-by: Justin Jung <[email protected]> * Revert []BytesLimiter to BytesLimtier Signed-off-by: Justin Jung <[email protected]> * Lint Signed-off-by: Justin Jung <[email protected]> * More reverts Signed-off-by: Justin Jung <[email protected]> * More Signed-off-by: Justin Jung <[email protected]> * Rename DefaultBytesLimiterFactory back to NewBytesLimiterFactory Signed-off-by: Justin Jung <[email protected]> * Changed StoreDataType from string to int Signed-off-by: Justin Jung <[email protected]> * Removed nil check for bytesLimiter Signed-off-by: Justin Jung <[email protected]> * nit Signed-off-by: Justin Jung <[email protected]> * Removed changelog Signed-off-by: Justin Jung <[email protected]> --------- Signed-off-by: Justin Jung <[email protected]>
Changes
Introducing two main changes to enhance bytes-based limiting:
BytesLimiter.Reserve(num uint64) err
toBytesLimiter.ReserveWithType(num uint64, dataType storeDataType) err
such that the bytes limiter logic to be more flexible. This is useful given that processing of postings, series and chunks have different impact to pod resource usage. For example, user can have a bytes limiter that has different weights for different data types.queryStats.add
func to update query stats info more consistentlyVerification
Added unit test and e2e test.