We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Our cluster has an unhealthy node which keeps printing the following in the logs. Let me know if you need more information.
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,639 - Calling paginated ec2:describe_instances with {'Filters': [{'Values': ['etcd-cluster-etcd'], 'Name': 'tag:aws:cloudformation:stack-name'}]} Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,650 - Starting new HTTPS connection (1): ec2.eu-central-1.amazonaws.com Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,738 - Starting new HTTP connection (1): 172.31.131.127 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,754 - Starting new HTTP connection (1): 172.31.131.127 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,757 - Starting new HTTP connection (1): 172.31.131.127 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,759 - My clientURLs list is not empty: ['http://172.31.131.225:2379'] Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,759 - My data directory exists=True Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:29,761 - Started new /bin/etcd process with pid: 444 and args: ['-name', 'i-05982c02c444ea551', '--data-dir', 'data', '-listen-peer-urls', 'http://0.0.0.0:2380', '-initial-advertise-peer-urls', 'http://172.31.131.225:2380', '-listen-client-urls', 'http://0.0.0.0:2379', '-advertise-client-urls', 'http://172.31.131.225:2379', '-initial-cluster', 'i-05982c02c444ea551=http://172.31.131.225:2380,i-06cda1ba0307bcd19=http://172.31.140.202:2380,i-06ed4eb078b00f623=http://172.31.153.93:2380,i-077dfb38fc0039342=http://172.31.159.46:2380,i-0a665d73f1d8c7cfa=http://172.31.131.127:2380', '-initial-cluster-token', 'etcd-cluster-etcd', '-initial-cluster-state', 'existing'] Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791434 W | flags: unrecognized environment variable ETCDVERSION=3.0.15 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791768 I | etcdmain: etcd Version: 3.0.15 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791939 I | etcdmain: Git SHA: fc00305 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792099 I | etcdmain: Go Version: go1.6.3 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792262 I | etcdmain: Go OS/Arch: linux/amd64 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792433 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792618 N | etcdmain: the server is already initialized as member before, starting as etcd member... Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792891 I | etcdmain: listening for peers on http://0.0.0.0:2380 Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.793147 I | etcdmain: listening for client requests on 0.0.0.0:2379 Jun 28 11:58:35 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:58:35,505 - Starting new HTTP connection (1): 172.31.131.225 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.301983 I | etcdserver: recovered store from snapshot at index 222437884 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302392 I | etcdserver: name = i-05982c02c444ea551 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302576 I | etcdserver: data dir = data Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302734 I | etcdserver: member dir = data/member Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302895 I | etcdserver: heartbeat = 100ms Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303048 I | etcdserver: election = 1000ms Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303180 I | etcdserver: snapshot count = 10000 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303341 I | etcdserver: advertise client URLs = http://172.31.131.225:2379 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: ERROR 2017-06-28 11:58:38,616 - Exception in HouseKeeper main loop Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last): Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 377, in _make_request Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: httplib_response = conn.getresponse(buffering=True) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: TypeError: getresponse() got an unexpected keyword argument 'buffering' Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last): Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 379, in _make_request Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: httplib_response = conn.getresponse() Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: response.begin() Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3.5/http/client.py", line 297, in begin Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: version, status, reason = self._read_status() Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3.5/http/client.py", line 258, in _read_status Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3.5/socket.py", line 575, in readinto Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: return self._sock.recv_into(b) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: socket.timeout: timed out Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last): Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: timeout=timeout Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: _stacktrace=sys.exc_info()[2]) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 247, in increment Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: raise six.reraise(type(error), error, _stacktrace) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: raise value Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: body=body, headers=headers) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: self._raise_timeout(err=e, url=url, timeout_value=read_timeout) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 309, in _raise_timeout Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='172.31.131.225', port=2379): Read timed out. (read timeout=3.1) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last): Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/bin/etcd.py", line 586, in run Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: if self.manager.etcd_pid != 0 and self.is_leader(): Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/bin/etcd.py", line 497, in is_leader Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: return self.manager.me.is_leader() Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/bin/etcd.py", line 194, in is_leader Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: return not self.api_get('stats/leader') is None Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/bin/etcd.py", line 163, in api_get Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: response = requests.get(url, timeout=self.API_TIMEOUT) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: return request('get', url, params=params, **kwargs) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: return session.request(method=method, url=url, **kwargs) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: resp = self.send(prep, **send_kwargs) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: r = adapter.send(request, **kwargs) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: raise ReadTimeout(e, request=request) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: requests.exceptions.ReadTimeout: HTTPConnectionPool(host='172.31.131.225', port=2379): Read timed out. (read timeout=3.1) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.899646 I | etcdserver: restarting member 53733ad8236d11ce in cluster 5cee1e413d02 at commit index 222455528 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.900135 C | raft: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776] Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: panic: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776] Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: goroutine 1 [running]: Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: panic(0xd450a0, 0xc835481250) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/usr/local/go/src/runtime/panic.go:481 +0x3e6 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820190620, 0x12375a0, 0x2b, 0xc832070f40, 0x4, 0x4) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).loadState(0xc83639f450, 0x1b334, 0x88b67ac3e24bbdb2, 0xd4266e8, 0x0, 0x0, 0x0) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:942 +0x2a2 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.newRaft(0xc8201078d8, 0x0) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:225 +0x8ff Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.RestartNode(0xc8201078d8, 0x0, 0x0) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:213 +0x45 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.restartNode(0xc82006fb80, 0xc820066000, 0x29, 0xc820107d78, 0x0, 0x0, 0x0, 0x0) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:369 +0x7c7 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc82006fb80, 0x0, 0x0, 0x0) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:348 +0x430e Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc8201a8000, 0x0, 0x0, 0x0) Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:374 +0x245f Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2() Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:116 +0x2101 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main() Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:36 +0x21e Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: main.main() Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/main.go:28 +0x14 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:58:38,933 - Process 444 finished with exit code 2 Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:58:38,933 - Sleeping 30 seconds before next try... Jun 28 11:59:08 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:08,944 - Calling paginated ec2:describe_instances with {'Filters': [{'Values': ['etcd-cluster-etcd'], 'Name': 'tag:aws:cloudformation:stack-name'}]} Jun 28 11:59:08 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:08,959 - Starting new HTTPS connection (1): ec2.eu-central-1.amazonaws.com Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:09,067 - Starting new HTTP connection (1): 172.31.131.127 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:09,071 - Starting new HTTP connection (1): 172.31.131.127 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:09,074 - Starting new HTTP connection (1): 172.31.131.127 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:09,076 - My clientURLs list is not empty: ['http://172.31.131.225:2379'] Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:09,076 - My data directory exists=True Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO 2017-06-28 11:59:09,077 - Started new /bin/etcd process with pid: 450 and args: ['-name', 'i-05982c02c444ea551', '--data-dir', 'data', '-listen-peer-urls', 'http://0.0.0.0:2380', '-initial-advertise-peer-urls', 'http://172.31.131.225:2380', '-listen-client-urls', 'http://0.0.0.0:2379', '-advertise-client-urls', 'http://172.31.131.225:2379', '-initial-cluster', 'i-05982c02c444ea551=http://172.31.131.225:2380,i-06cda1ba0307bcd19=http://172.31.140.202:2380,i-06ed4eb078b00f623=http://172.31.153.93:2380,i-077dfb38fc0039342=http://172.31.159.46:2380,i-0a665d73f1d8c7cfa=http://172.31.131.127:2380', '-initial-cluster-token', 'etcd-cluster-etcd', '-initial-cluster-state', 'existing'] Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108064 W | flags: unrecognized environment variable ETCDVERSION=3.0.15 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108416 I | etcdmain: etcd Version: 3.0.15 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108593 I | etcdmain: Git SHA: fc00305 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108771 I | etcdmain: Go Version: go1.6.3 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108929 I | etcdmain: Go OS/Arch: linux/amd64 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109088 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109278 N | etcdmain: the server is already initialized as member before, starting as etcd member... Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109530 I | etcdmain: listening for peers on http://0.0.0.0:2380 Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109792 I | etcdmain: listening for client requests on 0.0.0.0:2379 Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.859842 I | etcdserver: recovered store from snapshot at index 222437884 Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860265 I | etcdserver: name = i-05982c02c444ea551 Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860450 I | etcdserver: data dir = data Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860609 I | etcdserver: member dir = data/member Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860753 I | etcdserver: heartbeat = 100ms Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860895 I | etcdserver: election = 1000ms Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.861047 I | etcdserver: snapshot count = 10000 Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.861212 I | etcdserver: advertise client URLs = http://172.31.131.225:2379 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:18.446568 I | etcdserver: restarting member 53733ad8236d11ce in cluster 5cee1e413d02 at commit index 222455528 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:18.447078 C | raft: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776] Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: panic: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776] Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: goroutine 1 [running]: Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: panic(0xd450a0, 0xc8355fb560) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/usr/local/go/src/runtime/panic.go:481 +0x3e6 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820190620, 0x12375a0, 0x2b, 0xc833422f40, 0x4, 0x4) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).loadState(0xc836564270, 0x1b334, 0x88b67ac3e24bbdb2, 0xd4266e8, 0x0, 0x0, 0x0) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:942 +0x2a2 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.newRaft(0xc8201078d8, 0x0) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:225 +0x8ff Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.RestartNode(0xc8201078d8, 0x0, 0x0) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:213 +0x45 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.restartNode(0xc82006fb80, 0xc820066000, 0x29, 0xc820107d78, 0x0, 0x0, 0x0, 0x0) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:369 +0x7c7 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc82006fb80, 0x0, 0x0, 0x0) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:348 +0x430e Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc8201a8000, 0x0, 0x0, 0x0) Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:374 +0x245f Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2() Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:116 +0x2101 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main() Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:36 +0x21e Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: main.main() Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/main.go:28 +0x14 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:59:18,481 - Process 450 finished with exit code 2 Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:59:18,481 - Sleeping 30 seconds before next try...
The text was updated successfully, but these errors were encountered:
No branches or pull requests
Our cluster has an unhealthy node which keeps printing the following in the logs. Let me know if you need more information.
The text was updated successfully, but these errors were encountered: