[CI] TimeSeriesLifecycleActionsIT.testAllocateOnlyAllocation failures

[dimitris-athanasiou]

Build scan: https://gradle-enterprise.elastic.co/s/fbnlw7ql4crpo

Repro line:

./gradlew ':x-pack:plugin:ilm:qa:multi-node:javaRestTest' --tests "org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testAllocateOnlyAllocation" -Dtests.seed=BF58DBD092F76F42 -Dtests.security.manager=true -Dtests.locale=ca-ES -Dtests.timezone=America/Fortaleza -Druntime.java=8

Reproduces locally?: No

Applicable branches: master/7.x

Failure history: link

Failure excerpt:

Expected: <{"phase":"warm","action":"complete","name":"complete"}>
         but: was <{"phase":"warm","action":"allocate","name":"check-allocation"}>
        at __randomizedtesting.SeedInfo.seed([BF58DBD092F76F42:F9E6BEF28E1DC14E]:0)
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.lambda$testAllocateOnlyAllocation$10(TimeSeriesLifecycleActionsIT.java:265)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1010)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:983)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testAllocateOnlyAllocation(TimeSeriesLifecycleActionsIT.java:264)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[matriv]

There is also a timeout in the logs in this occurence: https://gradle-enterprise.elastic.co/s/kqgngupsrct2i

org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT > testFullPolicy FAILED
09:14:09     java.net.SocketTimeoutException: 60.000 milliseconds timeout on connection http-outgoing-137 [ACTIVE]
09:14:09         at __randomizedtesting.SeedInfo.seed([D62BF52A4214A06:E96978692D79D19D]:0)
09:14:09         at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:865)
09:14:09         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:283)
09:14:09         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:286)
09:14:09         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:286)
09:14:09         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:286)
09:14:09         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
09:14:09         at org.elasticsearch.test.rest.ESRestTestCase.ensureHealth(ESRestTestCase.java:1155)
09:14:09         at org.elasticsearch.test.rest.ESRestTestCase.ensureHealth(ESRestTestCase.java:1148)
09:14:09         at org.elasticsearch.test.rest.ESRestTestCase.ensureGreen(ESRestTestCase.java:1135)
09:14:09         at org.elasticsearch.xpack.TimeSeriesRestDriver.createIndexWithSettings(TimeSeriesRestDriver.java:283)
09:14:09         at org.elasticsearch.xpack.TimeSeriesRestDriver.createIndexWithSettings(TimeSeriesRestDriver.java:268)
09:14:09         at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testFullPolicy(TimeSeriesLifecycleActionsIT.java:102)
09:14:09 
09:14:09         Caused by:
09:14:09         java.net.SocketTimeoutException: 60.000 milliseconds timeout on connection http-outgoing-137 [ACTIVE]
09:14:09             at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387)
09:14:09             at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
09:14:09             at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
09:14:09             at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
09:14:09             at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:261)
09:14:09             at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:502)
09:14:09             at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:211)
09:14:09             at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
09:14:09             at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
09:14:09             at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
09:14:09             at java.lang.Thread.run(Thread.java:748)