Wait for pending tasks in TransportSearchFailuresIT #44543
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The change in elastic#44433 introduces a state in which the cluster has no relocating shards but still has a pending reroute task which might start a shard relocation. `TransportSearchFailuresIT` failed on a PR build seemingly because it did not wait for this pending task to complete too, reporting more active shards than expected: 2> java.lang.AssertionError: Expected: <9> but: was <10> at __randomizedtesting.SeedInfo.seed([4057CA4301FE95FA:207EC88573747235]:0) at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) at org.junit.Assert.assertThat(Assert.java:956) at org.junit.Assert.assertThat(Assert.java:923) at org.elasticsearch.search.basic.TransportSearchFailuresIT.testFailedSearchWithWrongQuery(TransportSearchFailuresIT.java:97) This commit addresses this failure by waiting until there are neither pending tasks nor shard relocations in progress.
DaveCTurner
added
>test
|
Pinging @elastic/es-distributed |
DaveCTurner
added a commit
that referenced
this pull request
Jul 18, 2019
Today we reroute the cluster as part of the process of starting a shard, which runs at `URGENT` priority. In large clusters, rerouting may take some time to complete, and this means that a mere trickle of shard-started events can cause starvation for other, lower-priority, tasks that are pending on the master. However, it isn't really necessary to perform a reroute when starting a shard, as long as one occurs eventually. This commit removes the inline reroute from the process of starting a shard and replaces it with a deferred one that runs at `NORMAL` priority, avoiding starvation of higher-priority tasks. Backport of #44433 and #44543.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The change in #44433 introduces a state in which the cluster has no relocating
shards but still has a pending reroute task which might start a shard
relocation.
TransportSearchFailuresITfailed on a PR build seemingly becauseit did not wait for this pending task to complete too, reporting more active shards
than expected:
This commit addresses this failure by waiting until there are neither pending
tasks nor shard relocations in progress.