[CI] org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT test {p0=indices.stats/13_fields/*} randomly fails on CI

[colings86]

Example Build URL: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+java9-periodic/1112/console

Reproduce command (I cannot reproduce this locally):

gradle :qa:mixed-cluster:v5.6.4-SNAPSHOT#mixedClusterTestRunner \
  -Dtests.seed=9094F5306A9A5286 \
  -Dtests.class=org.elasticsearch.backwards.MixedClusterClientYamlTestSuiteIT \
  -Dtests.method="test {p0=indices.stats/13_fields/Fields - blank}" \
  -Dtests.security.manager=true \
  -Dtests.jvm.argline="--add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.nio.file=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED --add-opens=java.base/java.util.regex=ALL-UNNAMED" \
  -Dtests.locale=ki \
  -Dtests.timezone=Etc/GMT+8

Failure stack trace:

<span class="timestamp"><b>12:35:47</b> </span>FAILURE 2.48s | MixedClusterClientYamlTestSuiteIT.test {p0=indices.stats/13_fields/Fields - blank} &lt;&lt;&lt; FAILURES!
<span class="timestamp"><b>12:35:47</b> </span>   &gt; Throwable #1: java.lang.AssertionError: Failure at [indices.stats/13_fields:72]: field [_all.total.fielddata.memory_size_in_bytes] is not greater than [0]
<span class="timestamp"><b>12:35:47</b> </span>   &gt; Expected: a value greater than &lt;0&gt;
<span class="timestamp"><b>12:35:47</b> </span>   &gt;      but: &lt;0&gt; was equal to &lt;0&gt;
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at __randomizedtesting.SeedInfo.seed([9094F5306A9A5286:18C0CAEAC4663F7E]:0)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:349)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.test(ESClientYamlSuiteTestCase.java:329)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at java.base/java.lang.Thread.run(Thread.java:844)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; Caused by: java.lang.AssertionError: field [_all.total.fielddata.memory_size_in_bytes] is not greater than [0]
<span class="timestamp"><b>12:35:47</b> </span>   &gt; Expected: a value greater than &lt;0&gt;
<span class="timestamp"><b>12:35:47</b> </span>   &gt;      but: &lt;0&gt; was equal to &lt;0&gt;
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at org.elasticsearch.test.rest.yaml.section.GreaterThanAssertion.doAssert(GreaterThanAssertion.java:65)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at org.elasticsearch.test.rest.yaml.section.Assertion.execute(Assertion.java:76)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	at org.elasticsearch.test.rest.yaml.ESClientYamlSuiteTestCase.executeSection(ESClientYamlSuiteTestCase.java:345)
<span class="timestamp"><b>12:35:47</b> </span>   &gt; 	... 37 more

[colings86]

@jpountz any ideas what could be going on here? Maybe this is Java 9 related as I have only seen this fail with Java 9 on CI?

[tlrx]

I pushed 6256c33 so that the test now waits for all the shards to be active with no ongoing relocations, because I suspect that the search queries from the set up sections could hit some shards that are later relocated to another node, losing the field data or completion stats.