This repository has been archived by the owner on Aug 2, 2022. It is now read-only.
Switches UpdateManagedIndexMetaData to batch tasks using custom executor #209
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
...management/transport/action/updateindexmetadata/TransportUpdateManagedIndexMetaDataAction.kt
Outdated
Show resolved
Hide resolved
...management/transport/action/updateindexmetadata/TransportUpdateManagedIndexMetaDataAction.kt
Show resolved
Hide resolved
qreshi
approved these changes
Apr 29, 2020
jinsoor-amzn
approved these changes
Apr 29, 2020
zengyan-amazon
approved these changes
Apr 29, 2020
dbbaughe
added a commit
to dbbaughe/index-management
that referenced
this pull request
Apr 30, 2020
…tor (opendistro-for-elasticsearch#209) * Switches UpdateManagedIndexMetaData to batch tasks using custom executor * Removes unneeded type
dbbaughe
added a commit
to dbbaughe/index-management
that referenced
this pull request
Apr 30, 2020
…tor (opendistro-for-elasticsearch#209) * Switches UpdateManagedIndexMetaData to batch tasks using custom executor * Removes unneeded type
dbbaughe
added a commit
to dbbaughe/index-management
that referenced
this pull request
Apr 30, 2020
…tor (opendistro-for-elasticsearch#209) * Switches UpdateManagedIndexMetaData to batch tasks using custom executor * Removes unneeded type
dbbaughe
added a commit
to dbbaughe/index-management
that referenced
this pull request
Apr 30, 2020
…tor (opendistro-for-elasticsearch#209) * Switches UpdateManagedIndexMetaData to batch tasks using custom executor * Removes unneeded type
dbbaughe
added a commit
to dbbaughe/index-management
that referenced
this pull request
Apr 30, 2020
…tor (opendistro-for-elasticsearch#209) * Switches UpdateManagedIndexMetaData to batch tasks using custom executor * Removes unneeded type
dbbaughe
added a commit
to dbbaughe/index-management
that referenced
this pull request
Apr 30, 2020
…tor (opendistro-for-elasticsearch#209) * Switches UpdateManagedIndexMetaData to batch tasks using custom executor * Removes unneeded type
dbbaughe
added a commit
to dbbaughe/index-management
that referenced
this pull request
Apr 30, 2020
…tor (opendistro-for-elasticsearch#209) * Switches UpdateManagedIndexMetaData to batch tasks using custom executor * Removes unneeded type
This was referenced Apr 30, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Issue #, if available:
Description of changes:
The task updating the cluster state for ISM were being executed and applied individually. This moves us to our own executor that will batch the tasks (of ISM type) together so they can be processed faster.
From testing this has made significant improvements in the timeouts experienced on clusters with large # of managed indices.
Tested on two domains with 10 data nodes (m4.xlarge), 3 dedicated masters (m4.large) with 5,000 Managed Indices each.
The current ISM would not be able to initialize all managed indices and would take hours to get through the queue while eventually timing out 3.5k of them until it was able to manage 1.5k without timeouts.
ISM with these changes was able to process all 5k managed indices right away and was able bring down pending_tasks queue to comparably low # with no timeouts. Was able to stress test further by reducing job_interval to 1 minute (5k jobs running per minute, ~10k cluster state updates per minute) and this has been running with no failures/timeouts overnight. Not recommending to reduce to 1 minute though until other improvements, but it's good to see impact.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.