Should GET / return 503 in case of discovery.zen.no_master_block: write ?

[Mpdreamz]

Given we have two nodes one (A) with:

node.master: false
discovery.zen.no_master_block: write

and (B) being a vanilla master node.

When we stop node (B), (A) is still allowed to service read requests.
However when calling

GET http://(A):9200/ HTTP/1.1

It currently returns:

HTTP/1.1 503 Service Unavailable

but is the service really unavailable in this case? Since we now explicitly allow you to configure for this state IMO it should return 200 OK with a possible boolean in the response signalling its in readonly mode.

A call to _search in this state also results in a 200 and not 503.

[martijnvg]

@Mpdreamz Agreed, the main rest endpoint should return 200 in case there is no elected master. It doesn't report on cluster state related things, just configured cluster_name and a couple of node related stats.

[javanna]

Agreed, the main rest endpoint should return 200 in case there is no elected master.

should it always return 200 or only if we block writes? If we block all operations we should still return 503 maybe?

[martijnvg]

true, we should return 503 when all operations are blocked, but in the case
of when just block writes we should return 200, because the node is still
partially operational.

On 16 January 2015 at 12:23, Luca Cavanna [email protected] wrote:

Agreed, the main rest endpoint should return 200 in case there is no
elected master.

should it always return 200 or only if we block writes? If we block all
operations we should still return 503 maybe?

—
Reply to this email directly or view it on GitHub
#8902 (comment)
.

Met vriendelijke groet,

Martijn van Groningen

[bleskes]

+1 . During master lost we go into a new master election which takes 3s (by default). During those 3s the node has a configured block - if it allows read we should indeed return 200. This is likely a transient state which will be solved before we start rejecting indexing requests (remember they wait up to 1m for the situation to be resolved).

[clintongormley]

I'm not sure that this is the right thing to do. Imagine you're using sniffing. You try to perform a write and get back a 503 so you sniff and get back a 200, then you try the write again, get back a 503, etc etc.

That said, the above would work for reads. I know the python client aborts after 3 attempts, while the Perl client keeps going until it gets back a 503 on sniffing. Perhaps we should only sniff once before giving up.

[bleskes]

@clintongormley how are other transient errors that are not reflected by / handled? for example - if the queues are full we return a 429 code. Does that have special handling? Another example - the circuit breaker throws a 503 too. That one is not reflected by /. Should it?

Another aspect to consider here - on master loss (since 1.4), all data nodes will have a master block for 3s. If you hit / at that moment, no node will be happy and that's I believe this ticket is about. I'm not saying that this is what we should do but I think this should be taken into account in the solution.

[clintongormley]

We have said that a 503 response code should mean "retry on another node".

if the queues are full we return a 429 code. Does that have special handling?

Not in the Perl client, but not sure about the others. I think 429 should probably not retry but instead backoff.

the circuit breaker throws a 503 too

That means retry on another node.... This one is debatable. If you've sent the request that has triggered the circuit breaker, you could then replicate that bad behaviour across all nodes in the cluster by retrying.

[jasontedor]

A 503 is completely broken behavior here. A REST status is a response for the given request. A 503 means "I am overloaded right now, I can not handle your request." That is completely out of alignment with a discovery.zen.no_master_block block. If the server can respond to the / request, it is not overloaded.

[jasontedor]

I opened #29045.