ZOOKEEPER-3758: Leader reachability check fails with single address #1288

symat · 2020-03-19T09:21:14Z

Since ZooKeeper 3.6.0 we can specify multiple addresses for each ZooKeeper server instance
(this can increase availability when multiple physical network interfaces can be used parallel
in the cluster). ZooKeeper will perform ICMP ECHO requests or try to establish a TCP connection
on port 7 (Echo) of the destination host in order to find the reachable addresses. This should
happen only if the user provide multiple addresses in the configuration, in a single address is
used then ZooKeeper shouldn’t send any ICMP requests.

This works as we expected for the leader election connections, but in this Jira issue we found
a bug when the reachability check was performed even with a single address when the Follower
tries to connect to the newly elected Leader.

The fix follows the same approach we discussed for the election protocol before (see
ZOOKEEPER-3698). We avoid the reachability check for single addresses. Also when we have
multiple addresses and none of them can be reached, we still start to connect to all addresses
in parallel.

…er Quorum address Since ZooKeeper 3.6.0 we can specify multiple addresses for each ZooKeeper server instance (this can increase availability when multiple physical network interfaces can be used parallel in the cluster). ZooKeeper will perform ICMP ECHO requests or try to establish a TCP connection on port 7 (Echo) of the destination host in order to find the reachable addresses. This should happen only if the user provide multiple addresses in the configuration, in a single addess is used then ZooKeeper shouldn’t send any ICMP requests. This works as we expected for the leader election connections, but in this Jira issue we found a bug when the reachability check was performed even with a single address when the Follower tries to connect to the newly elected Leader. The fix follows the same approach we discussed for the election protocol before (see ZOOKEEPER-3698). We avoid the reachability check for single addresses. Also when we have multiple addresses and none of them can be reached, we still start to connect to all addresses in parallel.

eolivelli · 2020-03-19T10:43:22Z

zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/Learner.java

@@ -303,7 +303,9 @@ protected void connectToLeader(MultipleAddresses multiAddr, String hostname) thr
        this.leaderAddr = multiAddr;
        Set<InetSocketAddress> addresses;
        if (self.isMultiAddressReachabilityCheckEnabled()) {


Isn't this disabled by default?

no, unfortunately not. There are two separate parameters:

multiAddress.enabled : this is for enabling/disabling the whole multiaddress feature, this is set to false to default

multiAddress.reachabilityCheckEnabled : this one can be used to disable the ICMP messages when multiple addresses are used. But this is enabled by default, as disabling it actually makes the multiAddress recovery unreliable (as ZK will not really be able to select among the addresses)

There is already a logic in the MultiAddress.getReachableOrOne() method which will skip the ICMP check if there is only a single address is provided:

zookeeper/zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/MultipleAddresses.java

Lines 151 to 158 in a5a4743

public InetSocketAddress getReachableOrOne() {

InetSocketAddress address;

// if there is only a single address provided then we don't do any reachability check

if (addresses.size() == 1) {

return getOne();

}

basically the same logic was missing from the MultipleAddresses.getAllReachableAddresses method.

I should have thinking of it, but forget when I fixed ZOOKEEPER-3698.

eolivelli · 2020-03-19T11:41:24Z

@nkalmar this is a good candidate for 3.6.1

please take a look

nkalmar

LGTM

eolivelli

LGTM

@nkalmar can you please merge ?

nkalmar · 2020-03-19T12:19:05Z

I messed up my python env, and the script is not running, I'll try to fix it. I can merge after.

symat · 2020-03-19T14:35:51Z

@eolivelli @nkalmar thanks for checking this so quickly!

Since ZooKeeper 3.6.0 we can specify multiple addresses for each ZooKeeper server instance (this can increase availability when multiple physical network interfaces can be used parallel in the cluster). ZooKeeper will perform ICMP ECHO requests or try to establish a TCP connection on port 7 (Echo) of the destination host in order to find the reachable addresses. This should happen only if the user provide multiple addresses in the configuration, in a single address is used then ZooKeeper shouldn’t send any ICMP requests. This works as we expected for the leader election connections, but in this Jira issue we found a bug when the reachability check was performed even with a single address when the Follower tries to connect to the newly elected Leader. The fix follows the same approach we discussed for the election protocol before (see ZOOKEEPER-3698). We avoid the reachability check for single addresses. Also when we have multiple addresses and none of them can be reached, we still start to connect to all addresses in parallel. Author: Mate Szalay-Beko <[email protected]> Reviewers: Enrico Olivelli <[email protected]>, Norbert Kalmar <[email protected]> Closes #1288 from symat/ZOOKEEPER-3758 (cherry picked from commit b1105cc) Signed-off-by: Norbert Kalmar <[email protected]>

nkalmar · 2020-03-19T16:11:11Z

pushed to master and branch-3.6, thanks @symat

Since ZooKeeper 3.6.0 we can specify multiple addresses for each ZooKeeper server instance (this can increase availability when multiple physical network interfaces can be used parallel in the cluster). ZooKeeper will perform ICMP ECHO requests or try to establish a TCP connection on port 7 (Echo) of the destination host in order to find the reachable addresses. This should happen only if the user provide multiple addresses in the configuration, in a single address is used then ZooKeeper shouldn’t send any ICMP requests. This works as we expected for the leader election connections, but in this Jira issue we found a bug when the reachability check was performed even with a single address when the Follower tries to connect to the newly elected Leader. The fix follows the same approach we discussed for the election protocol before (see ZOOKEEPER-3698). We avoid the reachability check for single addresses. Also when we have multiple addresses and none of them can be reached, we still start to connect to all addresses in parallel. Author: Mate Szalay-Beko <[email protected]> Reviewers: Enrico Olivelli <[email protected]>, Norbert Kalmar <[email protected]> Closes apache#1288 from symat/ZOOKEEPER-3758

eolivelli reviewed Mar 19, 2020

View reviewed changes

nkalmar approved these changes Mar 19, 2020

View reviewed changes

eolivelli approved these changes Mar 19, 2020

View reviewed changes

asfgit closed this in b1105cc Mar 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZOOKEEPER-3758: Leader reachability check fails with single address #1288

ZOOKEEPER-3758: Leader reachability check fails with single address #1288

symat commented Mar 19, 2020 •

edited

Loading

eolivelli Mar 19, 2020

symat Mar 19, 2020 •

edited

Loading

eolivelli commented Mar 19, 2020

nkalmar left a comment

eolivelli left a comment

nkalmar commented Mar 19, 2020

symat commented Mar 19, 2020

nkalmar commented Mar 19, 2020

	public InetSocketAddress getReachableOrOne() {
	InetSocketAddress address;

	// if there is only a single address provided then we don't do any reachability check
	if (addresses.size() == 1) {
	return getOne();
	}

ZOOKEEPER-3758: Leader reachability check fails with single address #1288

ZOOKEEPER-3758: Leader reachability check fails with single address #1288

Conversation

symat commented Mar 19, 2020 • edited Loading

eolivelli Mar 19, 2020

Choose a reason for hiding this comment

symat Mar 19, 2020 • edited Loading

Choose a reason for hiding this comment

eolivelli commented Mar 19, 2020

nkalmar left a comment

Choose a reason for hiding this comment

eolivelli left a comment

Choose a reason for hiding this comment

nkalmar commented Mar 19, 2020

symat commented Mar 19, 2020

nkalmar commented Mar 19, 2020

symat commented Mar 19, 2020 •

edited

Loading

symat Mar 19, 2020 •

edited

Loading