This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Choose the number of primary shards while creating indices #252
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AD is bottlenecked by the number of primary shards of job, result, and checkpoint index in HC. The number of primary shards in the job index determines how many nodes can run as AD's coordinating nodes. The number of primary shards in the result and checkpoint index determines the extent of index pressure given the same indexing workload. Previously, we used the default setting: in ODFE, the number is 1; in AES, the number is 5. This PR uses the number of hot nodes as the number of primary shards for the checkpoint, result, and job index . The upper limit is 10. Testing done: 1. added unit tests. 2. end-to-end testing
Codecov Report
@@ Coverage Diff @@
## master #252 +/- ##
=========================================
Coverage 73.01% 73.01%
Complexity 1461 1461
=========================================
Files 164 164
Lines 6834 6834
Branches 527 527
=========================================
Hits 4990 4990
Misses 1594 1594
Partials 250 250
Flags with carried forward coverage won't be shown. Click here to find out more.
|
ohltyler
reviewed
Oct 14, 2020
src/main/java/com/amazon/opendistroforelasticsearch/ad/indices/AnomalyDetectionIndices.java
Show resolved
Hide resolved
src/main/java/com/amazon/opendistroforelasticsearch/ad/indices/AnomalyDetectionIndices.java
Show resolved
Hide resolved
src/main/java/com/amazon/opendistroforelasticsearch/ad/indices/AnomalyDetectionIndices.java
Show resolved
Hide resolved
src/main/java/com/amazon/opendistroforelasticsearch/ad/indices/AnomalyDetectionIndices.java
Show resolved
Hide resolved
weicongs-amazon
approved these changes
Oct 14, 2020
ohltyler
approved these changes
Oct 14, 2020
kaituo
added a commit
to kaituo/anomaly-detection
that referenced
this pull request
Oct 16, 2020
This PR is a conglomerate of the following PRs. opendistro-for-elasticsearch#247 opendistro-for-elasticsearch#249 opendistro-for-elasticsearch#250 opendistro-for-elasticsearch#252 opendistro-for-elasticsearch#253 opendistro-for-elasticsearch#256 opendistro-for-elasticsearch#257 opendistro-for-elasticsearch#258 opendistro-for-elasticsearch#259 opendistro-for-elasticsearch#260 opendistro-for-elasticsearch#261 opendistro-for-elasticsearch#262 opendistro-for-elasticsearch#263 opendistro-for-elasticsearch#264 opendistro-for-elasticsearch#265 opendistro-for-elasticsearch#266 opendistro-for-elasticsearch#267 opendistro-for-elasticsearch#268 opendistro-for-elasticsearch#269 This spreadsheet contains the mappings from files to PR number: https://quip-amazon.com/DiHkAmz9oSLu/HC-PR Testing done: 1. Add unit tests except four classes (excluded in build.gradle). Will add them in the later PR. 2. Manual testing passes.
kaituo
added a commit
that referenced
this pull request
Oct 16, 2020
* Add support filtering the data by one categorical variable This PR is a conglomerate of the following PRs. #247 #249 #250 #252 #253 #256 #257 #258 #259 #260 #261 #262 #263 #264 #265 #266 #267 #268 #269 This spreadsheet contains the mappings from files to PR number: https://quip-amazon.com/DiHkAmz9oSLu/HC-PR Testing done: 1. Add unit tests except four classes (excluded in build.gradle). Will add them in the later PR. 2. Manual testing passes.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note: since there are a lot of dependencies, I only list the main class and test code to save reviewers' time. The build will fail due to missing dependencies. I will use that PR just for review. will not merge it. Will have a big one in the end and merge once after all review PRs get approved.
Issue #, if available:
Description of changes:
AD is bottle-necked by the number of primary shards of job, result, and checkpoint index in HC. The number of primary shards in the job index determines how many nodes can run as AD's coordinating nodes. The number of primary shards in the result and checkpoint index determines the extent of index pressure given the same indexing workload.
Previously, we used the default setting: in ODFE, the number is 1; in AES, the number is 5. This PR uses the number of hot nodes as the number of primary shards for the checkpoint, result, and job index . The upper limit is 10.
Testing done:
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.