Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cluster not working after reboot #262

Closed
jzeweniuk opened this issue Jun 3, 2020 · 10 comments
Closed

Cluster not working after reboot #262

jzeweniuk opened this issue Jun 3, 2020 · 10 comments
Assignees
@iwilltry42
Labels
bug Something isn't working help wanted Extra attention is needed k3s This is likely an issue with k3s not k3d itself
Milestone
v3.0 - FROM scratch

Comments

@jzeweniuk
Copy link

What did you do?
Problem restarting the cluster after a reboot

  • How was the cluster created?
    k3d create cluster dev --masters 3 --workers 3

  • What did you do afterwards?
    reboot

What did you expect to happen?
Kubectl to work after K3d start cluster dev

Concise description of what you expected to happen after doing what you described above.
Running k3d in Wsl2 Debian
Creating the cluster works and I can initially issue kubectl commands to it. If I reboot the cluster is down on reboot. Issuing a "k3d start cluster dev" appears to work and "docker ps" shows the running containers but kubectl returns Error from server (ServiceUnavailable): the server is currently unable to handle the request.

If I delete the cluster and recreate then I can get it to work but cannot seem to get the existing cluster working after a restart

.

Screenshots or terminal output
k3dSettings

Which OS & Architecture?
Windows 10 WSL2 Debian

Which version of k3d?
v3.0.0-beta.1

  • output of k3d --version
    k3d version v3.0.0-beta.1

Which version of docker?
19.03.11

  • output of docker version

Version: 19.03.11
API version: 1.40
Go version: go1.13.10
Git commit: 42e35e61f3
Built: Mon Jun 1 09:12:41 2020
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:29:16 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683
@jzeweniuk jzeweniuk added the bug Something isn't working label Jun 3, 2020
@jzeweniuk jzeweniuk changed the title [BUG] Cluster not working after reboot Jun 3, 2020
@jzeweniuk
Copy link
Author

Update: If I Create a cluster with k3d create, verify kubectl cluster-info works then do k3d stop cluster then do k3d start cluster I get the same behavior. I am no longer able to issue cluster-info etc.

@yexiang841
Copy link

it happens to me similarly. but something wrong occurred before restart

What did you do?
install dashboard after creating a 2 master 2 worker cluster in macOS.

How was the cluster created?
k3d create cluster q3d-k3s-local --api-port 6550 -p 8000:80@loadbalancer --masters 2 --workers 2 --volume /opt/deploy/rancher/registries.yaml:/etc/rancher/k3s/registries.yaml

What did you do afterwards?

kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.1/aio/deploy/recommended.yaml；

(and create admin-user / admin-user-role following https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/)

k3d stop cluster q3d-k3s-local
k3d start cluster q3d-k3s-local
export KUBECONFIG=$(k3d get kubeconfig q3d-k3s-local)
echo $KUBECONFIG
/Users/myaccount/.kube/config
kubectl config view
apiVersion: v1
clusters:

  • cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://0.0.0.0:6550
    name: k3d-q3d-k3s-local
    contexts:
  • context:
    cluster: k3d-q3d-k3s-local
    user: admin@k3d-q3d-k3s-local
    name: k3d-q3d-k3s-local
    current-context: k3d-q3d-k3s-local
    kind: Config
    preferences: {}
    users:
  • name: admin@k3d-q3d-k3s-local
    user:
    password: eaced5521e2cf490caae19da3260ad90
    username: admin

kubectl cluster-info

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Error from server (ServiceUnavailable): the server is currently unable to handle the request

kubectl get pods
Error from server (ServiceUnavailable): the server is currently unable to handle the request

What did you expect to happen?
Kubectl to work after K3d start cluster q3d-k3s-local

OS:
macOS Catalina
10.15.4 (19E287)

k3d version:
k3d version v3.0.0-beta.1

docker version:
Client: Docker Engine - Community
Version: 19.03.8
API version: 1.40
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:21:11 2020
OS/Arch: darwin/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: afacb8b
Built: Wed Mar 11 01:29:16 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: v1.2.13
GitCommit: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc:
Version: 1.0.0-rc10
GitCommit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
docker-init:
Version: 0.18.0
GitCommit: fec3683

for more information,i execute:

kubectl --v=8 cluster-info

and got:

I0604 07:07:02.237225 54688 loader.go:375] Config loaded from file: /Users/myaccout/.kube/config
I0604 07:07:02.238571 54688 round_trippers.go:420] GET https://0.0.0.0:6550/api?timeout=32s
I0604 07:07:02.238583 54688 round_trippers.go:427] Request Headers:
I0604 07:07:02.238587 54688 round_trippers.go:431] Accept: application/json, /
I0604 07:07:02.238591 54688 round_trippers.go:431] User-Agent: kubectl/v1.16.6 (darwin/amd64) kubernetes/e7f962b
I0604 07:07:02.238595 54688 round_trippers.go:431] Authorization: Basic
I0604 07:07:02.249776 54688 round_trippers.go:446] Response Status: 503 Service Unavailable in 11 milliseconds
I0604 07:07:02.249789 54688 round_trippers.go:449] Response Headers:
I0604 07:07:02.249793 54688 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I0604 07:07:02.249796 54688 round_trippers.go:452] X-Content-Type-Options: nosniff
I0604 07:07:02.249798 54688 round_trippers.go:452] Date: Wed, 03 Jun 2020 23:07:02 GMT
I0604 07:07:02.249803 54688 round_trippers.go:452] Content-Length: 9
I0604 07:07:02.261888 54688 request.go:968] Response Body: starting
I0604 07:07:02.276065 54688 request.go:1171] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }
I0604 07:07:02.276671 54688 cached_discovery.go:121] skipped caching discovery info due to the server is currently unable to handle the request
I0604 07:07:02.277205 54688 round_trippers.go:420] GET https://0.0.0.0:6550/api?timeout=32s
I0604 07:07:02.277214 54688 round_trippers.go:427] Request Headers:
I0604 07:07:02.277221 54688 round_trippers.go:431] Accept: application/json, /
I0604 07:07:02.277226 54688 round_trippers.go:431] User-Agent: kubectl/v1.16.6 (darwin/amd64) kubernetes/e7f962b
I0604 07:07:02.277233 54688 round_trippers.go:431] Authorization: Basic
I0604 07:07:02.280761 54688 round_trippers.go:446] Response Status: 503 Service Unavailable in 3 milliseconds
I0604 07:07:02.280775 54688 round_trippers.go:449] Response Headers:
I0604 07:07:02.280779 54688 round_trippers.go:452] Date: Wed, 03 Jun 2020 23:07:02 GMT
I0604 07:07:02.280782 54688 round_trippers.go:452] Content-Length: 9
I0604 07:07:02.280785 54688 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I0604 07:07:02.280787 54688 round_trippers.go:452] X-Content-Type-Options: nosniff
I0604 07:07:02.290376 54688 request.go:968] Response Body: starting
I0604 07:07:02.302608 54688 request.go:1171] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }
I0604 07:07:02.302624 54688 cached_discovery.go:121] skipped caching discovery info due to the server is currently unable to handle the request
I0604 07:07:02.302969 54688 shortcut.go:89] Error loading discovery information: the server is currently unable to handle the request
I0604 07:07:02.303097 54688 round_trippers.go:420] GET https://0.0.0.0:6550/api?timeout=32s
I0604 07:07:02.303104 54688 round_trippers.go:427] Request Headers:
I0604 07:07:02.303113 54688 round_trippers.go:431] Authorization: Basic
I0604 07:07:02.303117 54688 round_trippers.go:431] Accept: application/json, /
I0604 07:07:02.303120 54688 round_trippers.go:431] User-Agent: kubectl/v1.16.6 (darwin/amd64) kubernetes/e7f962b
I0604 07:07:02.305476 54688 round_trippers.go:446] Response Status: 503 Service Unavailable in 2 milliseconds
I0604 07:07:02.305495 54688 round_trippers.go:449] Response Headers:
I0604 07:07:02.305499 54688 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I0604 07:07:02.305502 54688 round_trippers.go:452] X-Content-Type-Options: nosniff
I0604 07:07:02.305504 54688 round_trippers.go:452] Date: Wed, 03 Jun 2020 23:07:02 GMT
I0604 07:07:02.305507 54688 round_trippers.go:452] Content-Length: 9
I0604 07:07:02.315822 54688 request.go:968] Response Body: starting
I0604 07:07:02.329930 54688 request.go:1171] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }
I0604 07:07:02.329984 54688 cached_discovery.go:121] skipped caching discovery info due to the server is currently unable to handle the request
I0604 07:07:02.330284 54688 round_trippers.go:420] GET https://0.0.0.0:6550/api?timeout=32s
I0604 07:07:02.330298 54688 round_trippers.go:427] Request Headers:
I0604 07:07:02.330308 54688 round_trippers.go:431] Authorization: Basic
I0604 07:07:02.330314 54688 round_trippers.go:431] Accept: application/json, /
I0604 07:07:02.330318 54688 round_trippers.go:431] User-Agent: kubectl/v1.16.6 (darwin/amd64) kubernetes/e7f962b
I0604 07:07:02.335584 54688 round_trippers.go:446] Response Status: 503 Service Unavailable in 5 milliseconds
I0604 07:07:02.335628 54688 round_trippers.go:449] Response Headers:
I0604 07:07:02.335634 54688 round_trippers.go:452] Content-Length: 9
I0604 07:07:02.335638 54688 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I0604 07:07:02.335640 54688 round_trippers.go:452] X-Content-Type-Options: nosniff
I0604 07:07:02.335643 54688 round_trippers.go:452] Date: Wed, 03 Jun 2020 23:07:02 GMT
I0604 07:07:02.346945 54688 request.go:968] Response Body: starting
I0604 07:07:02.360346 54688 request.go:1171] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }
I0604 07:07:02.360370 54688 cached_discovery.go:121] skipped caching discovery info due to the server is currently unable to handle the request
I0604 07:07:02.360443 54688 round_trippers.go:420] GET https://0.0.0.0:6550/api?timeout=32s
I0604 07:07:02.360448 54688 round_trippers.go:427] Request Headers:
I0604 07:07:02.360452 54688 round_trippers.go:431] User-Agent: kubectl/v1.16.6 (darwin/amd64) kubernetes/e7f962b
I0604 07:07:02.360456 54688 round_trippers.go:431] Authorization: Basic
I0604 07:07:02.360460 54688 round_trippers.go:431] Accept: application/json, /
I0604 07:07:02.361990 54688 round_trippers.go:446] Response Status: 503 Service Unavailable in 1 milliseconds
I0604 07:07:02.362000 54688 round_trippers.go:449] Response Headers:
I0604 07:07:02.362003 54688 round_trippers.go:452] Content-Type: text/plain; charset=utf-8
I0604 07:07:02.362006 54688 round_trippers.go:452] X-Content-Type-Options: nosniff
I0604 07:07:02.362008 54688 round_trippers.go:452] Date: Wed, 03 Jun 2020 23:07:02 GMT
I0604 07:07:02.362011 54688 round_trippers.go:452] Content-Length: 9
I0604 07:07:02.371577 54688 request.go:968] Response Body: starting
I0604 07:07:02.384753 54688 request.go:1171] body was not decodable (unable to check for Status): couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:"apiVersion,omitempty""; Kind string "json:"kind,omitempty"" }
I0604 07:07:02.384798 54688 cached_discovery.go:121] skipped caching discovery info due to the server is currently unable to handle the request

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
I0604 07:07:02.385419 54688 helpers.go:199] server response object: [{
"metadata": {},
"status": "Failure",
"message": "the server is currently unable to handle the request",
"reason": "ServiceUnavailable",
"details": {
"causes": [
{
"reason": "UnexpectedServerResponse",
"message": "starting"
}
]
},
"code": 503
}]
F0604 07:07:02.386073 54688 helpers.go:114] Error from server (ServiceUnavailable): the server is currently unable to handle the request

@iwilltry42 iwilltry42 self-assigned this Jun 4, 2020
@iwilltry42 iwilltry42 added this to the v3.0 - FROM scratch milestone Jun 4, 2020
@iwilltry42 iwilltry42 added the k3s This is likely an issue with k3s not k3d itself label Jun 4, 2020
@iwilltry42
Copy link
Member

iwilltry42 commented Jun 4, 2020 •
edited
Loading

Hi there, thanks for opening this issue 👍
I can confirm that this is an issue, but unfortunately, we cannot solve it inside k3d, because it's a known issue (...no known leader...) with dqlite in k3s, where the initializing master node (master-0) must not go down.
k3s is on track to switch dqlite with embedded etcd, so this will hopefully be fixed soon :)
UPDATE: I also included this in the FAQ now: https://k3d.io/faq/faq/#restarting-a-multi-master-cluster-or-the-initializing-master-node-fails

@http403 http403 mentioned this issue Jun 17, 2020
Closed
@bukowa
Copy link

bukowa commented Mar 12, 2021

@iwilltry42 what's the current state on this (i still got an error but PR is merged)

@iwilltry42
Copy link
Member

@bukowa what are your errors?
K3d uses embedded etcd for Multi-Server clusters for quite some time now and k3d did some adjustments to account for that.

@bukowa
Copy link

bukowa commented Mar 12, 2021

cluster created with 3 servers --no-lb after restart i had an error that lead me to this issue. In general i could not connect to kubernetes API again. I recreated the cluster with one server, i am sure i will try later with more servers again and report back more informations if ill encounter an error.

@doc4child
Copy link

still doesn't work

@bukowa
Copy link

bukowa commented Mar 23, 2021

@iwilltry42 I just tried again and still after restart i cannot connect.

$ k3d version
k3d version v4.3.0
k3s version v1.20.4-k3s1 (default)

$ k3d cluster create test1 --servers=3
←[36mINFO←[0m[0000] Prep: Network
←[36mINFO←[0m[0000] Created network 'k3d-test1'
←[36mINFO←[0m[0000] Created volume 'k3d-test1-images'
←[36mINFO←[0m[0000] Creating initializing server node
←[36mINFO←[0m[0000] Creating node 'k3d-test1-server-0'
←[36mINFO←[0m[0001] Creating node 'k3d-test1-server-1'
←[36mINFO←[0m[0002] Creating node 'k3d-test1-server-2'
←[36mINFO←[0m[0002] Creating LoadBalancer 'k3d-test1-serverlb'
←[36mINFO←[0m[0002] Starting cluster 'test1'
←[36mINFO←[0m[0002] Starting the initializing server...
←[36mINFO←[0m[0002] Starting Node 'k3d-test1-server-0'
←[36mINFO←[0m[0003] Starting servers...
←[36mINFO←[0m[0003] Starting Node 'k3d-test1-server-1'
←[36mINFO←[0m[0022] Starting Node 'k3d-test1-server-2'
←[36mINFO←[0m[0037] Starting agents...
←[36mINFO←[0m[0037] Starting helpers...
←[36mINFO←[0m[0037] Starting Node 'k3d-test1-serverlb'
←[36mINFO←[0m[0037] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
←[36mINFO←[0m[0039] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
←[36mINFO←[0m[0039] Cluster 'test1' created successfully!
←[36mINFO←[0m[0039] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
←[36mINFO←[0m[0039] You can now use it like this:
kubectl config use-context k3d-test1
kubectl cluster-info

Then i closed docker && wsl --shutdown
I am on windows docker with WSL. How can i provide more informations about the error?
Server 0 docker log:

W0323 15:48:39.970658       7 sysinfo.go:203] Nodes topology is not available, providing CPU topology
W0323 15:48:39.971286       7 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
E0323 15:48:39.978078       7 info.go:114] Failed to get system UUID: open /etc/machine-id: no such file or directory
W0323 15:48:39.979061       7 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
{"level":"warn","ts":"2021-03-23T15:48:40.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:40.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:45.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:45.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:50.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:50.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:55.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:55.185Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:00.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:00.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:05.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:05.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:10.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:10.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:15.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:15.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:20.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:20.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:25.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:25.186Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:30.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:30.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:35.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:35.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:40.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:40.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:45.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:45.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:50.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:50.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:55.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:55.187Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:00.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:00.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:05.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:05.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:10.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:10.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:15.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:15.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:20.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:20.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:25.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:25.188Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:30.189Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:30.189Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:35.189Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:35.189Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}

server 1 docker logs:

{"level":"info","ts":"2021-03-23T15:43:33.394Z","caller":"raft/raft.go:811","msg":"d4d6044be856ff55 [logterm: 2, index: 1671] sent MsgVote request to 283ae73b048d6089 at term 3"}
{"level":"info","ts":"2021-03-23T15:43:33.394Z","caller":"raft/raft.go:811","msg":"d4d6044be856ff55 [logterm: 2, index: 1671] sent MsgVote request to d7380397c3ec4b90 at term 3"}
{"level":"info","ts":"2021-03-23T15:43:33.400Z","caller":"raft/raft.go:824","msg":"d4d6044be856ff55 received MsgVoteResp from d7380397c3ec4b90 at term 3"}
{"level":"info","ts":"2021-03-23T15:43:33.400Z","caller":"raft/raft.go:1302","msg":"d4d6044be856ff55 has received 2 MsgVoteResp votes and 0 vote rejections"}
{"level":"info","ts":"2021-03-23T15:43:33.400Z","caller":"raft/raft.go:765","msg":"d4d6044be856ff55 became leader at term 3"}
{"level":"info","ts":"2021-03-23T15:43:33.400Z","caller":"raft/node.go:325","msg":"raft.node: d4d6044be856ff55 elected leader d4d6044be856ff55 at term 3"}
{"level":"warn","ts":"2021-03-23T15:43:33.408Z","caller":"etcdserver/util.go:121","msg":"failed to apply request","took":"34.6µs","request":"header:<ID:5444984302074413895 > lease_revoke:<id:7f55785fbce2ac99>","response":"size:28","error":"lease not found"}
{"level":"info","ts":"2021-03-23T15:43:33.413Z","caller":"etcdserver/server.go:2039","msg":"published local member to cluster through raft","local-member-id":"d4d6044be856ff55","local-member-attributes":"{Name:k3d-test1-server-1-5d1ce7ce ClientURLs:[https://172.19.0.5:2379]}","request-path":"/0/members/d4d6044be856ff55/attributes","cluster-id":"17d3753d35ddbc03","publish-timeout":"15s"}
{"level":"info","ts":"2021-03-23T15:43:33.414Z","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"172.19.0.5:2379"}
{"level":"info","ts":"2021-03-23T15:43:33.414Z","caller":"embed/serve.go:191","msg":"serving client traffic securely","address":"127.0.0.1:2379"}
time="2021-03-23T15:43:33.418119800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:43:38.421346100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:43:39.819364400Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:43:39 +0000 UTC"
time="2021-03-23T15:43:43.425095200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:43:44.213348500Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:43:44 +0000 UTC"
time="2021-03-23T15:43:48.427975700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:43:53.432687300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:43:58.435202100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:03.438253700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:08.441198100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:13.443852300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:18.446558500Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:23.449241100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:28.451935600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:33.454763800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:38.457674500Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:43.460396200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:48.463051600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:53.465791500Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:44:57.302663200Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:44:57 +0000 UTC"
time="2021-03-23T15:44:57.302942700Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:44:57 +0000 UTC"
time="2021-03-23T15:44:57.686394300Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:44:57 +0000 UTC"
time="2021-03-23T15:44:57.686749000Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:44:57 +0000 UTC"
time="2021-03-23T15:44:58.468826900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:03.471800600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:07.327119900Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:45:07 +0000 UTC"
time="2021-03-23T15:45:08.474670700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:13.477496200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:18.480350200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:23.483115700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:28.485904400Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:28.729338200Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:45:28 +0000 UTC"
time="2021-03-23T15:45:33.488745800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:38.491537100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:43.494512000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:48.497403900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:53.500537300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:45:58.503268300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:03.506061100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:08.508698900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:13.511463000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:18.514174200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:23.517129000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:28.519912700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:33.522673900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:38.525353100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:43.528253400Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:48.531074800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:53.533771100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:46:58.536532600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:03.539484400Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:08.542318900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:13.544982200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:16.810461500Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:47:16 +0000 UTC"
time="2021-03-23T15:47:18.547862300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:20.306393100Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:47:20 +0000 UTC"
time="2021-03-23T15:47:20.306910000Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:47:20 +0000 UTC"
time="2021-03-23T15:47:23.550895000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:28.553805800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:33.556320200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:38.559202900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:43.561751700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:48.564606900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:53.567347300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:47:58.569949800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:03.572483100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:07.686279000Z" level=info msg="certificate CN=k3s,O=k3s signed by CN=k3s-server-ca@1616513906: notBefore=2021-03-23 15:38:26 +0000 UTC notAfter=2022-03-23 15:48:07 +0000 UTC"
time="2021-03-23T15:48:08.575572000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:13.578340000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:18.581163500Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:23.583968600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:28.586747800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:33.589583000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:38.592204200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:43.595123000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:48.597859500Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:53.600642700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:48:58.603275400Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:03.606060800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:08.608817300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:13.611693700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:18.614701700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:23.617404000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:28.620152500Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:33.622989600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:38.625766700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:43.628369800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:48.631000500Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:53.633964900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:49:58.636516800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:03.639203600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:08.641828300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:13.644625100Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:18.647493700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:23.650261900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:28.653084900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:33.655691300Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:38.658535200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:43.661305400Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:48.663809000Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:53.666649700Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:50:58.670761900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:51:03.673444200Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:51:08.676100800Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:51:13.678751600Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"
time="2021-03-23T15:51:18.681177900Z" level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5"

server 2 docker logs:

{"level":"warn","ts":"2021-03-23T15:48:35.427Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
W0323 15:48:39.969998       8 sysinfo.go:203] Nodes topology is not available, providing CPU topology
W0323 15:48:39.970579       8 sysfs.go:348] unable to read /sys/devices/system/cpu/cpu0/online: open /sys/devices/system/cpu/cpu0/online: no such file or directory
E0323 15:48:39.976719       8 info.go:114] Failed to get system UUID: open /etc/machine-id: no such file or directory
W0323 15:48:39.977703       8 info.go:53] Couldn't collect info from any of the files in "/etc/machine-id,/var/lib/dbus/machine-id"
{"level":"warn","ts":"2021-03-23T15:48:40.427Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:40.427Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:45.427Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:45.427Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:50.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:50.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:55.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:48:55.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:00.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:00.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:05.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:05.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:10.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:10.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:15.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:15.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:20.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:20.428Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:25.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:25.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:30.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:30.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:35.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:35.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:40.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:40.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:45.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:45.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:50.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:50.429Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:55.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:49:55.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:00.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:00.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:05.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:05.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:10.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:10.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:15.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:15.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:20.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:20.430Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:25.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:25.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:30.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:30.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:35.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:35.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:40.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:40.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:45.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:45.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:50.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:50.431Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:55.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:50:55.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:00.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:00.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:05.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:05.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:10.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:10.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:15.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:15.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:20.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:20.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:25.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:25.432Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:30.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:30.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:35.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:35.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:40.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:40.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:45.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:45.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:50.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:50.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:55.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:51:55.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:52:00.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}
{"level":"warn","ts":"2021-03-23T15:52:00.433Z","caller":"rafthttp/probing_status.go:70","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"d4d6044be856ff55","rtt":"0s","error":"dial tcp 172.19.0.3:2380: connect: connection refused"}

@iwilltry42 iwilltry42 added the help wanted Extra attention is needed label Mar 30, 2021
@iwilltry42
Copy link
Member

iwilltry42 commented Mar 30, 2021 •
edited
Loading

Thanks for the input @bukowa 👍
At first sight this looks like etcd fails to rebuild the cluster.
The log lines of the format level=info msg="Failed to test data store connection: this server is a not a member of the etcd cluster. Found [k3d-test1-server-2-96a67933=https://172.19.0.4:2380 k3d-test1-server-1-5d1ce7ce=https://172.19.0.3:2380 k3d-test1-server-0-09ac1081=https://172.19.0.2:2380], expect: k3d-test1-server-1-5d1ce7ce=172.19.0.5" make it look like docker assigned the wrong IP to the node k3d-test1-server-1 which is not equal to the one saved in the etcd config 🤔

UPDATE: Following up in new issue: #550

@iwilltry42 iwilltry42 mentioned this issue Apr 7, 2021
Closed
@iwilltry42
Copy link
Member

@bukowa see #550 (comment)

@jkleinlercher jkleinlercher mentioned this issue Sep 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iwilltry42 iwilltry42

Labels
bug Something isn't working help wanted Extra attention is needed k3s This is likely an issue with k3s not k3d itself
Projects
None yet
Milestone
v3.0 - FROM scratch
Development

No branches or pull requests
5 participants
@yexiang841 @iwilltry42 @doc4child @bukowa @jzeweniuk