[Transform] improve TransformRestTestCase robustness

[hendrikmuhs]

handles/retries temporary SearchPhaseExecutionErrors

fixes #54810

[elasticmachine]

Pinging @elastic/ml-core (:ml/Transform)

[droberts195]

I wonder if the root cause of the failure is that

elasticsearch/x-pack/plugin/transform/qa/single-node-tests/src/test/java/org/elasticsearch/xpack/transform/integration/TransformRestTestCase.java

Line 383 in df44a82

ensureNoInitializingShards();

doesn't wait for hidden indices? The cluster health check in

elasticsearch/test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java

Line 1026 in 87ce139

Request request = new Request("GET", "/_cluster/health");

doesn't specify any indices options. This might explain why the failures started when hidden indices were introduced.

[hendrikmuhs]

I wonder if the root cause of the failure is that

elasticsearch/x-pack/plugin/transform/qa/single-node-tests/src/test/java/org/elasticsearch/xpack/transform/integration/TransformRestTestCase.java

Line 383 in df44a82

ensureNoInitializingShards();

doesn't wait for hidden indices? The cluster health check in

I found this:

https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/action/admin/cluster/health/ClusterHealthRequest.java#L42

Looks like hidden indices are part of the default.
@gwbrown can you confirm?

Regarding ensureNoInitializingShards: I moved this in the last attempt to fix this bug:

df44a82

[hendrikmuhs]

The test started failing march 26th, this is 6 days after the notification index was made hidden.

(The audit logging was added february 18th.)

The logs unfortunately do not show a full picture, because we lack when ensureNoInitializingShards has been called. We could add logging.

The cluster health reported in the logs, turns green after the search phase execution exception:

     [2020-04-24T09:29:35,602][INFO ][o.e.c.r.a.AllocationService] [integTest-0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.transform-notifications-000002][0]]])." previous.health="YELLOW" reason="shards started [[.transform-notifications-000002][0]]"

This indicates that the cluster health indeed might have been green at the time ensureNoInitializingShards has been called.

LBNL, the failure only happens for testDeleteConfigurationLeftOver, but there are many more tests using TransformRestTestCase. The test is special in the sense that it does not interact much with the transform API but simulates a corner case situation: It indexes a document and checks whether transform DELETE is able to delete the doc. I think there is only 1 audit message in total, send late by the delete call. The audit message is indexed async, so maybe it is indeed a timing issue of parallel threads on a slow host (I was never able to reproduce this issue on my laptop).

Instead of the assertBusy loop I added in the log forwarder, I could add a busy loop to check, that the notification index exists, which is a valid assumption for TransformRestTestCase.

[droberts195]

Looks like hidden indices are part of the default

In that case I am happy to work around this in the test code, so this PR LGTM