From a59a972413cef886eb7b2f048aa8dc08a61bf1a2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Niklas=20Hamb=C3=BCchen?= Date: Thu, 18 Jun 2020 02:08:17 +0200 Subject: [PATCH] consul.passthru.tests: Fix failure on current consul. Fixes #90613. Done by setting `autopilot.min_quorum = 3`. Techncially, this would have been required to keep the test correct since Consul's "autopilot" "Dead Server Cleanup" was enabled by default (I believe that was in Consul 0.8). Practically, the issue only occurred with our NixOS test with releases >= `1.7.0-beta2` (see #90613). The setting itself is available since Consul 1.6.2. However, this setting was not documented clearly enough for anybody to notice, and only the upstream issue https://github.com/hashicorp/consul/issues/8118 I filed brought that to light. As explained there, the test could also have been made pass by applying the more correct rolling reboot procedure -m.wait_until_succeeds("[ $(consul members | grep -o alive | wc -l) == 5 ]") +m.wait_until_succeeds( + "[ $(consul operator raft list-peers | grep true | wc -l) == 3 ]" +) but we also intend to test that Consul can regain consensus even if the quorum gets temporarily broken. --- nixos/tests/consul.nix | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/nixos/tests/consul.nix b/nixos/tests/consul.nix index eb7dd45923fc4..ffbbd835885e5 100644 --- a/nixos/tests/consul.nix +++ b/nixos/tests/consul.nix @@ -73,6 +73,10 @@ let extraConfig = defaultExtraConfig // { server = true; bootstrap_expect = numConsensusServers; + # Tell Consul that we never intend to drop below this many servers. + # Ensures to not permanently lose consensus after temporary loss. + # See https://github.com/hashicorp/consul/issues/8118#issuecomment-645330040 + autopilot.min_quorum = numConsensusServers; retry_join = # If there's only 1 node in the network, we allow self-join; # otherwise, the node must not try to join itself, and join only the other servers.