This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Rollover AD result index less frequently #168
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, we roll over the result index every 30 days or every 300000 docs. Assuming each doc has 1 KB and our result index has five shards, each shard takes about 60 MB, which is too small. Small shards are against ES best practice. This PR increases the rollover threshold to 9000000 docs, which increases the max shard size to roughly 1.8 GB.
wnbts
approved these changes
Jun 22, 2020
Comment on lines
88
to
90
| // Suppose generally per cluster has 200 detectors and all run with 1 minute interval. | ||
| // We will get 288,000 AD result docs. So set it as 300k to avoid multiple roll overs | ||
| // per day. |
There was a problem hiding this comment.
minor. comments are outdated.
ylwu-amzn
approved these changes
Jun 22, 2020
vamshin
reviewed
Jun 22, 2020
| // per day. | ||
| 300 * 1000L, | ||
| 9_000_000L, |
There was a problem hiding this comment.
Is this a problem for 200 detectors? From the comment 300k is taking care of 288k limit to avoid multiple roll overs right?
There was a problem hiding this comment.
it is not a problem for 200 detectors. It is a problem of small shard. Please see my PR descriptions.
There was a problem hiding this comment.
Right i get that. Comment is misleading here. Why not mention the message in PR description as comment? We are talking about 200 detectors and roll overs in the comment. Atleast mention the PR?
One more thing you might want to confirm is does having larger index impact performance when compared to previous smaller index?
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
Currently, we roll over the result index every 30 days or every 300000 docs. Assuming each doc has 1 KB and our result index has five shards, each shard takes about 60 MB, which is too small. Small shards are against ES best practice. This PR increases the rollover threshold to 9000000 docs, which increases the max shard size to roughly 1.8 GB.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.