Cluster health should distinguish RED from "no master"

[DaveCTurner]

A RED cluster health means that there is either no master or else at least one primary is unassigned. However, in 7.0 there seems to be no simple way for clients to distinguish these two cases, and it would be useful to do so. For example:

• a per-node health check should verify the node-level property of cluster membership, but may not be interested in the cluster-wide property of whether all the primaries are assigned.

• an orchestrator may wish to wait for a freshly-started node to join a cluster before performing some followup actions, again without caring about whether all the primaries are assigned. Today this often happens by waiting for the HTTP port to be open, but this is unreliable: if no master is discovered within discovery.initial_state_timeout (default 30s) then we open the HTTP port anyway.

In the 6.x series a node responds to GET / with 503 Service Unavailable if it believes there to be no master (i.e. either NO_MASTER_BLOCK_* or STATE_NOT_RECOVERED_BLOCK is present), which allows these cases to be distinguished. However #29045 changes this in 7.0 so that we will always respond 200 OK to GET / (as is right and proper) so another mechanism is needed.

I think we should expose the presence or absence of these blocks in the output to GET _cluster/health and add the ability to wait for their absence, e.g.:

GET _cluster/health?local&wait_for_discovered_master

[elasticmachine]

Pinging @elastic/es-core-infra

[DaveCTurner]

We discussed this and broadly agreed to do it. Note that we are not intending to change the meaning of RED health - it already means no master OR unassigned primaries, and we would like it to keep this meaning.

We contemplated identifying the discovered master in the cluster health response. I think I'd rather just have a boolean "discovered_master": true, on the grounds that you can find out which node is the discovered master via other APIs (GET _nodes/_master for instance) and the identity of the master isn't important for the cluster's health.

[getsaurabh02]

I am currently working on it.

[DaveCTurner]

We discussed this again recently and determined that GET _cluster/health?timeout=0s returns 503 Service Unavailable when there is no master, and 200 OK otherwise, so it looks like we don't need any further action here.

[getsaurabh02]

Hi @DaveCTurner , thanks for responding on this. I tried running this and verified form code as well that GET _cluster/health?timeout=0s returns a valid response when used along with the flag local=true when no master is present. It does not return 503 Service Unavailable as expected otherwise.

This makes the API behavior inconsistent with the presence of local=true field when master is not discovered on a node. Also, using the health API request always without the local field will not be an efficient choice for large clusters, when triggered frequently.

Introducing the "discovered_master": true, as thought earlier, allows the no-master cases to be distinguished consistently, with and without the presence of the local field. Thoughts?

[DaveCTurner]

This makes the API behavior inconsistent

Asking a node to compute the cluster health locally is always possible, so 200 seems like the right response there. 503 here doesn't mean simply "unhealthy", it means "so unhealthy that I can't even answer your question".

Also, using the health API request always without the local field will not be an efficient choice for large clusters, when triggered frequently.

This deserves some further investigation. Health requests should be pretty cheap, and there are benefits to getting an answer from the master. Are you really seeing the master struggling with the load here? Can you provide more details?

[getsaurabh02]

Hi @DaveCTurner replying back and requesting opening this thread again to get some more thoughts. We have seen instances where master node gets overwhelmed with too many (periodic) health requests being delegated to it from data nodes (without local=true), typically in a large cluster, resulting into steep JVM spikes and causing master duress. This is more apparent when the master node is already under stress due to increased number of pending tasks, or busy managing cluster states, such as during full cluster restart.

Even for non dedicated master setup, this can get aggravated with even with a small cluster sizes (less than 10 nodes). This could typically happens in scenarios when cluster has a font end service, or a load balancer or maybe some monitoring script which intends to look for node health periodically, to send or restrict traffic to node, based upon nodes ability to process the request. Without the presence of field local=true in the health API call it is not possible to uncover scenarios where a node is not connected to the active master. Also cluster/health?timeout=0s returns a valid response when used along with the flag local=true when no master is present.

As there could be multiple reason which could lead a node to this situation, such as leader check failures due to network partitioning etc. Please let me know if it still makes sense to have a dedicated "discovered_master": true for better clarity. This will leverage the local cluster state, without having to delegate each request to master to discover its presence.

[DaveCTurner]

Repeating my previous message, health requests should be pretty cheap, can you analyse this in more detail so we can understand better how and why it's struggling?