etcdv3: Add two new member failed.

[RayQiQi]

Add new members procedure:

1. The initial etcd cluster with 3 nodes is healthy.
2. Re-edit etcd.conf on all the 5 nodes:
   ETCD_INITIAL_CLUSTER="node0=http://192.168.3.30:2380,node1=http://192.168.3.31:2380,node2=http://192.168.3.40:2380,node3=http://192.168.3.44:2380,node4=http://192.168.3.45:2380"
3. Edit etcd.conf on the two new added nodes:
   ETCD_INITIAL_CLUSTER_STATE="existing"
4. Add new member on the forth node, success.
   [root@bvnf-demo-app-4 ~]# etcdctl --endpoints http://192.168.3.30:2379 member add bvnf-demo-app-4 http://192.168.3.44:2380
   Added member named bvnf-demo-app-4 with ID 50af961773cd8148 to cluster

ETCD_NAME="bvnf-demo-app-4"
ETCD_INITIAL_CLUSTER="bvnf-demo-app-4=http://192.168.3.44:2380,bvnf-demo-oam-1=http://192.168.3.31:2380,bvnf-demo-oam-0=http://192.168.3.30:2380,bvnf-demo-app-0=http://192.168.3.40:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
4. Add another new member on the fifth node, failed.
[root@bvnf-demo-app-5 ~]# etcdctl --endpoints http://192.168.3.30:2379 member add bvnf-demo-app-5 http://192.168.3.45:2380
client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://192.168.3.30:2379 has no leader
; error #1: client: etcd member http://192.168.3.31:2379 has no leader
; error #2: client: etcd member http://192.168.3.40:2379 has no leader

[gyuho]

Your ETCD_INITIAL_CLUSTER doesn't match with bvnf-demo-app-5. And after adding the first member, did you start the new etcd member? And did it start fine?

[RayQiQi]

I can not start the new etcd member. the first one always can be added, but the health check show it is unreachable and can not start service, the second one can not be added.

ETCD_INITIAL_CLUSTER="node0=http://192.168.3.30:2380,node1=http://192.168.3.31:2380,node2=http://192.168.3.40:2380,bvnf-demo-app-4=http://192.168.3.44:2380,node4=http://192.168.3.45:2380"

[root@bvnf-demo-app-4 ~]# etcdctl --endpoints http://192.168.3.30:2379 member add bvnf-demo-app-4 http://192.168.3.44:2380
Added member named bvnf-demo-app-4 with ID 2394907e7c82432f to cluster

ETCD_NAME="bvnf-demo-app-4"
ETCD_INITIAL_CLUSTER="bvnf-demo-app-4=http://192.168.3.44:2380,bvnf-demo-oam-1=http://192.168.3.31:2380,bvnf-demo-oam-0=http://192.168.3.30:2380,bvnf-demo-app-0=http://192.168.3.40:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"
[root@bvnf-demo-app-4 ~]# etcdctl --endpoints http://192.168.3.30:2379 cluster-health
member 2394907e7c82432f is unreachable: no available published client urls
member 563500a8c6ef2736 is healthy: got healthy result from http://192.168.3.31:2379
member b0f2f891d631812f is healthy: got healthy result from http://192.168.3.30:2379
member d1e96dbc913a4e5f is healthy: got healthy result from http://192.168.3.40:2379
cluster is healthy
[root@bvnf-demo-app-4 ~]# systemctl start etcd
Job for etcd.service failed because the control process exited with error code. See "systemctl status etcd.service" and "journalctl -xe" for details.
[root@bvnf-demo-app-4 ~]# journalctl -f -t etcd
-- Logs begin at 三 2017-11-08 15:03:51 CST. --
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: recognized environment variable ETCD_LISTEN_CLIENT_URLS, but unused: shadowed by corresponding flag 
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: etcd Version: 3.2.5
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: Git SHA: d0d1a87
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: Go Version: go1.8.3
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: Go OS/Arch: linux/amd64
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: setting maximum number of CPUs to 2, total number of available CPUs is 2
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: the server is already initialized as member before, starting as etcd member...
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: listening for peers on http://192.168.3.44:2380
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: listening for client requests on 192.168.3.44:2379
11月 10 10:20:23 bvnf-demo-app-4 etcd[21652]: error validating peerURLs {ClusterID:1c25537163017fac Members:[&{ID:563500a8c6ef2736 RaftAttributes:{PeerURLs:[http://192.168.3.31:2380]} Attributes:{Name:bvnf-demo-oam-1 ClientURLs:[http://192.168.3.31:2379]}} &{ID:b0f2f891d631812f RaftAttributes:{PeerURLs:[http://192.168.3.30:2380]} Attributes:{Name:bvnf-demo-oam-0 ClientURLs:[http://192.168.3.30:2379]}} &{ID:d1e96dbc913a4e5f RaftAttributes:{PeerURLs:[http://192.168.3.40:2380]} Attributes:{Name:bvnf-demo-app-0 ClientURLs:[http://192.168.3.40:2379]}} &{ID:2394907e7c82432f RaftAttributes:{PeerURLs:[http://192.168.3.44:2380]} Attributes:{Name: ClientURLs:[]}}] RemovedMemberIDs:[]}: member count is unequal
-----------------------------------------------------
[root@bvnf-demo-app-5 ~]# cat /etc/etcd/etcd.conf
#[member]
ETCD_NAME="bvnf-demo-app-5"
ETCD_LISTEN_PEER_URLS="http://192.168.3.45:2380"
ETCD_DATA_DIR="/appdata/etcd"
ETCD_LISTEN_CLIENT_URLS="http://192.168.3.45:2379"
#[cluster]
ETCD_INITIAL_CLUSTER="node0=http://192.168.3.30:2380,node1=http://192.168.3.31:2380,node2=http://192.168.3.40:2380,node3=http://192.168.3.44:2380,bvnf-demo-app-5=http://192.168.3.45:2380"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.3.45:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.3.45:2379"
ETCD_INITIAL_CLUSTER_STATE="existing"
ETCD_INITIAL_CLUSTER_TOKEN="lcm_test"
#[proxy]
ETCD_PROXY="off"

[root@bvnf-demo-app-5 ~]# etcdctl --endpoints http://192.168.3.30:2379 member add bvnf-demo-app-5 http://192.168.3.45:2380
client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://192.168.3.30:2379 has no leader
; error #1: client: etcd member http://192.168.3.31:2379 has no leader
; error #2: client: etcd member http://192.168.3.40:2379 has no leader

[gyuho]

Added member named bvnf-demo-app-4 with ID 2394907e7c82432f to cluster

When you start the new member after this, initial cluster count should be 4, not 5.
You are starting with initial cluster flag of 5, when you only added 1.

[RayQiQi]

Looks like that if I edit the initial cluster count to 5 on the new added two members and then add member and start service one by one will fail.
First edit the initial cluster count to 4, add member and start service, success.
Second edit the initial cluster count to 5 on the other node, add member and start service, success.
Thanks very much.

[NStanleyZhang]

I have a question here. Why ETCD report below error if we run “etcdctl member add” to increase two ETCD server to a 3-member cluster? BTW, The cluster is healthy without starting fourth node.
##################
etcdctl --endpoints http://192.168.3.30:2379 member add bvnf-demo-app-5 http://192.168.3.45:2380
client: etcd cluster is unavailable or misconfigured; error #0: client: etcd member http://192.168.3.30:2379 has no leader
; error #1: client: etcd member http://192.168.3.31:2379 has no leader
; error #2: client: etcd member http://192.168.3.40:2379 has no leader