chash: retry on another endpoint if the first one fails

[pushrax]

When using the upstream-hash-by feature, if an endpoint fails, nginx will keep retrying on the same endpoint. This is a reasonable default configuration. However, if the application can tolerate it, better availability can be achieved by retrying on another endpoint (e.g. the next endpoint in the hash ring).

In particular this is useful during deploys, as nginx will try to send requests to a terminating pod for a couple seconds until the backends are synchronized. In our application, we are working around this by sleeping in a preStop hook for a few seconds, but the time for nginx to sync is not guaranteed, and it also slows down pod shutdown.

This is similar to a feature that was implemented for the session affinity balancer: session-cookie-change-on-failure in #4048

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Gacko]

/assign