[CI] TransformRobustnessIT.testTaskRemovalAfterInternalIndexGotDeleted fails rarely

[henningandersen]

Looks like this test was introduced on Jan 17th and has since failed in two 7.6 builds and a number of PR builds.

I investigated the build failure here (which is a PR build), but did not come to a conclusion on why this failed.

It happens during @After waitForDataFrame, where it calls wipeTransforms that searches .transform-internal-004 to check that it is empty, but gets an "all shards failed" error:

18:13:33   2> org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:34165], URI [.transform-internal-004/_search], status line [HTTP/1.1 503 Service Unavailable]
18:13:33     {"error":{"root_cause":[],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[]},"status":503}
18:13:33         at __randomizedtesting.SeedInfo.seed([7FED7A091BC346F9:E0CF7958A02C5009]:0)
18:13:33         at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283)
18:13:33         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261)
18:13:33         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:267)
18:13:33         at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235)
18:13:33         at org.elasticsearch.xpack.transform.integration.TransformRestTestCase.wipeTransforms(TransformRestTestCase.java:407)
18:13:33         at org.elasticsearch.xpack.transform.integration.TransformRestTestCase.waitForDataFrame(TransformRestTestCase.java:367)

The reproduction line did not reproduce locally:

./gradlew ':x-pack:plugin:transform:qa:single-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.transform.integration.TransformRobustnessIT.testTaskRemovalAfterInternalIndexGotDeleted" -Dtests.seed=7FED7A091BC346F9 -Dtests.security.manager=true -Dtests.locale=ar-SA -Dtests.timezone=Etc/GMT+6 -Dcompiler.java=13

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[hendrikmuhs]

My best guess at the moment:

1. The test removes the internal index (this is what the test is about)
2. the index gets re-created automatically via a cluster state listener
3. At the time we fire the search the index is in the process of re-creation, meaning shards are not initialized yet,
4. we get a search phase execution failure with shard failures

Solution: call ensureNoInitializingShards() after step 1 or after the test.

[hendrikmuhs]

Still an issue, #51406 did not fix it: https://gradle-enterprise.elastic.co/s/jwniyzbhw4nje

[dnhatn]

Failed again on master: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob+fast+part2/3674/consoleText

Build scans: https://gradle-enterprise.elastic.co/s/wnekcstytq36g

[hendrikmuhs]

I investigated this issue. It's not the same, it's different. This time it fails in cleanup of TransformRestTestCase: it expects an empty index but finds a state doc. This state doc is probably written by the task after the index got deleted.