Robust hash ring failure retry mechanism

[yashh]

So once a host failed and we have a server_retry_timeout of 30 secs nutcracker retries the failed host on production traffic. I think nutcracker needs to perform a background hearbeat request like fetch a simple key and assure that host is up.

[shapirus]

I too thought about this. Implementing a background heartbeat check, however, may actually be too difficult and not worth the effort (I'm not sure that twmemproxy's design supports anything background at all; I may be wrong though). What could be done instead is something along these lines (pseudocode):

...
if (time_passed > retry_timeout) {
    if (try_connection(ejected_server)) {
        inject_server_back(ejected_server);
    } else {
        leave_server_ejected(ejected_server);
    }
    do_the_job(key, value);
}
...

It will allow this given request not to fail and be dispatched to another host if the original server is still down after the retry_timeout has expired. The cost of such approach will be one client request delayed for timeout ms once in every server_retry_timeout ms while the failed server stays down, which seems acceptable to me (it is better than the client having to redo its request anyway).

[mezzatto]

+1 for this request, but being done in the background...

[manjuraj]

@shapirus would you be interested to submit a patch for this? :)

[shapirus]

Probably, but I will need to understand the current code and algorithms. Where do the server failure handling routines live and where do they get called from?

[manjuraj]

server failure handing routing is in nc_server.c:server_failure(). This is triggered whenever a server is closed - nc.sever.c:server_close() which gets invoked from conn->close() in core_close()

The server struct in nc_server.h maintains two fields to keep track of failures:

int64_t            next_retry;    /* next retry time in usec */
uint32_t           failure_count; /* # consecutive failures */

The server_pool struct in nc_server.h keep track of how many failures to allow before letting dead/live servers to be ejected and/or put-back

uint32_t           ncontinuum;           /* # continuum points */
uint32_t           nserver_continuum;    /* # servers - live and dead on continuum (const) */
struct continuum   *continuum;           /* continuum */
uint32_t           nlive_server;         /* # live server */
int64_t            next_rebuild;         /* next distribution rebuild time in usec */

int                timeout;              /* timeout in msec */
int64_t            server_retry_timeout; /* server retry timeout in usec */
unsigned           auto_eject_hosts:1;   /* auto_eject_hosts? */

Whenever a server is ejected or put back, we have to rebuild the hash ring. See server_pool_run() and server_pool_update(). We call server_pool_update() when:

server is ejected - server_failure()
server needs to be added back - server_pool_conn()

One idea in terms of implementing what @yashh is asking for is to use the "normal traffic" to eject a server on a failure. But once a server is ejected we use a timer to do background "heartbeat traffic" on the bad server and add the server back only if these hearbeat checks succeed. Timers and timer expiry is implemented in twemproxy use event loop. See core_timeout()

@shapirus let me know if you have questions or thoughts

[charsyam]

@manjuraj Hi, Could you review my code for this issue. I don't send patch for this yet.
I think I have to get code review from you.
charsyam@4748ed4

briefly speacking,
1] it is using core_timeout.

[charsyam]

@manjuraj, I add a function which check servers' status with 'get command'

There are 2 steps.

1] check connection
-> until now, I still use failures' array in retry_connection to connection test.

2] check command
-> if connection is established, at that time send 'get command'
If this operation is success, and then update the hash_ring.

But there is a limitation. it needs timeout setting

charsyam@f2bf742

I think it is better that implementing cron module like redis's serverCron and regularly check all servers' status.

[diwayou]

"What will your client do when a server is unavailable or provides an invalid response?

In the dark days of memcached, the default was to always "failover", by trying the next server in the list. That way if a server crashes, its keys will get reassigned to other instances and everything moves on happily.

However there're many ways to kill a machine. Sometimes they don't even like to stay dead. Given the scenario:

Sysadmin Bob walks by Server B and knocks the ethernet cable out of its port.
Server B's keys get "rerouted" to other instances.
Sysadmin Bob is an attentive (if portly) fellow and dutifully restores the ethernet cable from its parted port.
Server B's keys get "rerouted" back to itself.
Now everything goes scary. Any updates you've made to your cache in the time it took Bob to realize his mistake have been lost, and old data is presented to the user. This gets even worse if:

Server B's ethernet clip was broken by Bob's folly and later falls out of its port unattended.
Now your data has flipped back to yet another set. Annoying.

Another erroneous client feature would actually amend the server list when a server goes out of commission, which ends up remapping far more keys than it should.

Modern life encourages the use of "Failure", when possible. That is, if the server you intend to fetch or store a cache entry to is unavailable, simply proceed as though it was a cache miss. You might still flap between old and new data if you have a Server B situation, but the effects are reduced."

I copied this from https://code.google.com/p/memcached/wiki/NewConfiguringClient - Failure or Failover,
twemproxy use failover, but if it's a network break problem as above, we can send a "flush_all" command to the back(ok) server to solve this problem. is it the right way to do this?