Enhance SENTINEL FAILOVER to use the FAILOVER command to avoid data loss

[enjoy-binbin]

Currently, SENTINEL FAILOVER selects a replica to send REPLICAOF NO ONE,
then waits for PROMOTION, and then sends REPLICAOF new_ip new_port to the
old primary and other replicas to complete the failover. The problem here
is that if the old primary has written to it before the role change during
the failover, these writes will be lost.

We can use the FAILOVER command to avoid this data loss. FAILOVER was added
in 0d18a1e, it can coordinates the failover
between the primary and the replica.

Before the original step REPLICAOF NO ONE, we try to send FAILOVER TO ip port
to the primary to go through the FAILOVER process. If the primary does not
support FAILOVER, for example, returns an error, it will fallback to the old
logic, that means fallback to REPLICAOF NO ONE.

[enjoy-binbin]

This is a PR that was written very quickly (I wrote it at night and i want to push it as soon as possible). There may be problems with the details or it can be optimized.

Also see the redis/redis#6062 for more data loss details. The test case also proves data loss locally.

[zuiderkwast]

Yes, this looks good.

I'm not enough familiar with Sentinel. @hwware you are a sentinel expert. Do you want to take a look?

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 40 lines in your changes missing coverage. Please review.

Project coverage is 70.64%. Comparing base (c21f1dc) to head (21a6f4c).
Report is 7 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/sentinel.c	0.00%	40 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1238      +/-   ##
============================================
- Coverage     70.72%   70.64%   -0.08%     
============================================
  Files           114      114              
  Lines         63150    63195      +45     
============================================
- Hits          44660    44642      -18     
- Misses        18490    18553      +63     

Files with missing lines	Coverage Δ
src/sentinel.c	0.00% <0.00%> (ø)

... and 12 files with indirect coverage changes

[hwware]

In real production, this step: call rename-command explicitly is must?

[enjoy-binbin]

No, it is just a test step that try to cover the fallback logic.

[zuiderkwast]

Let's explain in the comment

# We simulate a server that doesn't have the FAILOVER command.

[hwware]

Overall, LTGM, just one concern about the test case.