[CI] SnapshotLifecycleRestIT testStartStopStatus ESRestTestCase cleanup problem

[hendrikmuhs]

Failed build: https://gradle-enterprise.elastic.co/s/nqhxngzzxxooq

Repro:

./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testStartStopStatus" -Dtests.seed=B2A6A505B6E252BD -Dtests.security.manager=true -Dtests.locale=es-HN -Dtests.timezone=Europe/Mariehamn -Dcompiler.java=13 -Druntime.java=8
10:49:05   2> java.lang.RuntimeException: failed to delete policy: slm-history-ilm-policy

Relevant log:

 2> REPRODUCE WITH: ./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.slm.SnapshotLifecycleRestIT.testStartStopStatus" -Dtests.seed=B2A6A505B6E252BD -Dtests.security.manager=true -Dtests.locale=es-HN -Dtests.timezone=Europe/Mariehamn -Dcompiler.java=13 -Druntime.java=8
10:49:05   2> java.lang.RuntimeException: failed to delete policy: slm-history-ilm-policy
10:49:05         at __randomizedtesting.SeedInfo.seed([B2A6A505B6E252BD:A283BCFED5EA623B]:0)
10:49:05         at org.elasticsearch.test.rest.ESRestTestCase.lambda$deleteAllILMPolicies$8(ESRestTestCase.java:741)
10:49:05         at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
10:49:05         at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
10:49:05         at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1556)
10:49:05         at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
10:49:05         at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
10:49:05         at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
10:49:05         at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
10:49:05         at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
10:49:05         at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
10:49:05         at org.elasticsearch.test.rest.ESRestTestCase.deleteAllILMPolicies(ESRestTestCase.java:737)
10:49:05         at org.elasticsearch.test.rest.ESRestTestCase.wipeCluster(ESRestTestCase.java:578)
10:49:05         at org.elasticsearch.test.rest.ESRestTestCase.cleanUpCluster(ESRestTestCase.java:318)
10:49:05         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
10:49:05         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
10:49:05         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
10:49:05         at java.lang.reflect.Method.invoke(Method.java:498)
10:49:05         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
10:49:05         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
10:49:05         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
10:49:05         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
10:49:05         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
10:49:05         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
10:49:05         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
10:49:05         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
10:49:05         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
10:49:05         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
10:49:05         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
10:49:05         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
10:49:05         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
10:49:05         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
10:49:05         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
10:49:05         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
10:49:05         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
10:49:05         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
10:49:05         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
10:49:05         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
10:49:05         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
10:49:05         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
10:49:05         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
10:49:05         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
10:49:05         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
10:49:05         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
10:49:05         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
10:49:05         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
10:49:05         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
10:49:05         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
10:49:05         at java.lang.Thread.run(Thread.java:748)
10:49:05 
10:49:05         Caused by:
10:49:05         org.elasticsearch.client.ResponseException: method [DELETE], host [http://[::1]:43808], URI [/_ilm/policy/slm-history-ilm-policy], status line [HTTP/1.1 400 Bad Request]
10:49:05         {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Cannot delete policy [slm-history-ilm-policy]. It is in use by one or more indices: [.slm-history-1-000001]"}],"type":"illegal_argument_exception","reason":"Cannot delete policy [slm-history-ilm-policy]. It is in use by one or more indices: [.slm-history-1-000001]"},"status":400}
10:49:05             at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
10:49:05             at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
10:49:05             at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
10:49:05             at org.elasticsearch.test.rest.ESRestTestCase.lambda$deleteAllILMPolicies$8(ESRestTestCase.java:739)
10:49:05             ... 47 more

Looking at the stack trace, this failure happens in the @After phase of the test and seems to be a timing issue: I guess an ILM policy is not deleted quick enough (async?).

Does not reproduce locally for me.

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)