Try to stabilise tests by checking for cluster health early

[pebrc]

Attempt to fix #7172

[pebrc]

buildkite test this -f p=kind,DEPLOYER_KIND_NODE_IMAGE=kindest/node:v1.28.0@sha256:b7a4cad12c197af3ba43202d3efe03246b3f0793f162afb40a33c923952d5b31,t=TestSystemIntegrationRecipe

[barkbay]

LGTM

[barkbay]

nit: we already have a dedicated step, no real advantage compared to your proposal except that we would have a dedicated entry in the logs:

=== RUN   TestAgentConfigRef/Elasticsearch_cluster_should_be_created
Retries (5m0s timeout): .
    --- PASS: TestAgentConfigRef/Elasticsearch_cluster_should_be_created (0.57s)
=== RUN   TestAgentConfigRef/ES_cluster_health_should_eventually_be_green
Retries (5m0s timeout): ......
    --- PASS: TestAgentConfigRef/ES_cluster_health_should_eventually_be_green (41.40s)
=== RUN   TestAgentConfigRef/Creating_an_Agent_should_succeed
    --- PASS: TestAgentConfigRef/Creating_an_Agent_should_succeed (1.48s)

Suggested change

	// check cluster health here already to make sure any dependent builders can rely on a ready ES
	// this is special casing Elasticsearch a little bit in the interest of test stability see https://github.com/elastic/cloud-on-k8s/issues/7172
	return clusterHealthIs(b, k, esv1.ElasticsearchGreenHealth)
	return nil
	}),
	},
	// check cluster health here already to make sure any dependent builders can rely on a ready ES
	// this is special casing Elasticsearch a little bit in the interest of test stability see https://github.com/elastic/cloud-on-k8s/issues/7172
	CheckClusterHealth(b, k),

[pebrc]

I think my motivation was to have this check in the create steps. While the existing check runs in the check steps. By having it in the create steps I want to introduce a delay so that the Agent creation (and other resource creations) does not run before ES is ready. I could remove the dedicated step from the check steps to avoid the duplication.

[pebrc]

Ah I misunderstood your proposal! I think that makes sense actually.

[pebrc]

buildkite test this -f E2E_TAGS=es,t=*

[pebrc]

buildkite test this -f E2E_TAGS=es,p=gke

[thbkrkr]

We have e2e tests where ES does not start voluntarily (see TestForceUpgrade*). Perhaps a boolean SkipInitialCheckClusterHealth in the ES builder to conditionally add the check is enough to cover this.

[pebrc]

buildkite test this -f t=TestForceUpgrade,p=gke

[pebrc]

buildkite test this -f t=TestRemoteCluster,p=gke

[pebrc]

buildkite test this -f t=TestRemoteCluster,p=gke