Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Member fails to recover #43

Open
linki opened this issue Jun 28, 2017 · 0 comments
Open

Member fails to recover #43

linki opened this issue Jun 28, 2017 · 0 comments

Comments

@linki
Copy link
Member

linki commented Jun 28, 2017

Our cluster has an unhealthy node which keeps printing the following in the logs. Let me know if you need more information.

Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,639 - Calling paginated ec2:describe_instances with {'Filters': [{'Values': ['etcd-cluster-etcd'], 'Name': 'tag:aws:cloudformation:stack-name'}]}
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,650 - Starting new HTTPS connection (1): ec2.eu-central-1.amazonaws.com
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,738 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,754 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,757 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,759 - My clientURLs list is not empty: ['http://172.31.131.225:2379']
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,759 - My data directory exists=True
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:29,761 - Started new /bin/etcd process with pid: 444 and args: ['-name', 'i-05982c02c444ea551', '--data-dir', 'data', '-listen-peer-urls', 'http://0.0.0.0:2380', '-initial-advertise-peer-urls', 'http://172.31.131.225:2380', '-listen-client-urls', 'http://0.0.0.0:2379', '-advertise-client-urls', 'http://172.31.131.225:2379', '-initial-cluster', 'i-05982c02c444ea551=http://172.31.131.225:2380,i-06cda1ba0307bcd19=http://172.31.140.202:2380,i-06ed4eb078b00f623=http://172.31.153.93:2380,i-077dfb38fc0039342=http://172.31.159.46:2380,i-0a665d73f1d8c7cfa=http://172.31.131.127:2380', '-initial-cluster-token', 'etcd-cluster-etcd', '-initial-cluster-state', 'existing']
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791434 W | flags: unrecognized environment variable ETCDVERSION=3.0.15
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791768 I | etcdmain: etcd Version: 3.0.15
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.791939 I | etcdmain: Git SHA: fc00305
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792099 I | etcdmain: Go Version: go1.6.3
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792262 I | etcdmain: Go OS/Arch: linux/amd64
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792433 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792618 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.792891 I | etcdmain: listening for peers on http://0.0.0.0:2380
Jun 28 11:58:29 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:29.793147 I | etcdmain: listening for client requests on 0.0.0.0:2379
Jun 28 11:58:35 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:58:35,505 - Starting new HTTP connection (1): 172.31.131.225
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.301983 I | etcdserver: recovered store from snapshot at index 222437884
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302392 I | etcdserver: name = i-05982c02c444ea551
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302576 I | etcdserver: data dir = data
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302734 I | etcdserver: member dir = data/member
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.302895 I | etcdserver: heartbeat = 100ms
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303048 I | etcdserver: election = 1000ms
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303180 I | etcdserver: snapshot count = 10000
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.303341 I | etcdserver: advertise client URLs = http://172.31.131.225:2379
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: ERROR  2017-06-28 11:58:38,616 - Exception in HouseKeeper main loop
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 377, in _make_request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     httplib_response = conn.getresponse(buffering=True)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: TypeError: getresponse() got an unexpected keyword argument 'buffering'
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 379, in _make_request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     httplib_response = conn.getresponse()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     response.begin()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/http/client.py", line 297, in begin
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     version, status, reason = self._read_status()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3.5/socket.py", line 575, in readinto
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return self._sock.recv_into(b)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: socket.timeout: timed out
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     timeout=timeout
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     _stacktrace=sys.exc_info()[2])
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 247, in increment
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise six.reraise(type(error), error, _stacktrace)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise value
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     body=body, headers=headers)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='172.31.131.225', port=2379): Read timed out. (read timeout=3.1)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: During handling of the above exception, another exception occurred:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: Traceback (most recent call last):
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 586, in run
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     if self.manager.etcd_pid != 0 and self.is_leader():
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 497, in is_leader
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return self.manager.me.is_leader()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 194, in is_leader
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return not self.api_get('stats/leader') is None
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/bin/etcd.py", line 163, in api_get
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     response = requests.get(url, timeout=self.API_TIMEOUT)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return request('get', url, params=params, **kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     return session.request(method=method, url=url, **kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     resp = self.send(prep, **send_kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     r = adapter.send(request, **kwargs)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:   File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:     raise ReadTimeout(e, request=request)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: requests.exceptions.ReadTimeout: HTTPConnectionPool(host='172.31.131.225', port=2379): Read timed out. (read timeout=3.1)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.899646 I | etcdserver: restarting member 53733ad8236d11ce in cluster 5cee1e413d02 at commit index 222455528
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:58:38.900135 C | raft: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: panic: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: goroutine 1 [running]:
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: panic(0xd450a0, 0xc835481250)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/usr/local/go/src/runtime/panic.go:481 +0x3e6
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820190620, 0x12375a0, 0x2b, 0xc832070f40, 0x4, 0x4)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).loadState(0xc83639f450, 0x1b334, 0x88b67ac3e24bbdb2, 0xd4266e8, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:942 +0x2a2
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.newRaft(0xc8201078d8, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:225 +0x8ff
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.RestartNode(0xc8201078d8, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:213 +0x45
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.restartNode(0xc82006fb80, 0xc820066000, 0x29, 0xc820107d78, 0x0, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:369 +0x7c7
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc82006fb80, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:348 +0x430e
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc8201a8000, 0x0, 0x0, 0x0)
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:374 +0x245f
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:116 +0x2101
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:36 +0x21e
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: main.main()
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/main.go:28 +0x14
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:58:38,933 - Process 444 finished with exit code 2
Jun 28 11:58:38 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:58:38,933 - Sleeping 30 seconds before next try...
Jun 28 11:59:08 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:08,944 - Calling paginated ec2:describe_instances with {'Filters': [{'Values': ['etcd-cluster-etcd'], 'Name': 'tag:aws:cloudformation:stack-name'}]}
Jun 28 11:59:08 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:08,959 - Starting new HTTPS connection (1): ec2.eu-central-1.amazonaws.com
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,067 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,071 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,074 - Starting new HTTP connection (1): 172.31.131.127
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,076 - My clientURLs list is not empty: ['http://172.31.131.225:2379']
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,076 - My data directory exists=True
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: INFO   2017-06-28 11:59:09,077 - Started new /bin/etcd process with pid: 450 and args: ['-name', 'i-05982c02c444ea551', '--data-dir', 'data', '-listen-peer-urls', 'http://0.0.0.0:2380', '-initial-advertise-peer-urls', 'http://172.31.131.225:2380', '-listen-client-urls', 'http://0.0.0.0:2379', '-advertise-client-urls', 'http://172.31.131.225:2379', '-initial-cluster', 'i-05982c02c444ea551=http://172.31.131.225:2380,i-06cda1ba0307bcd19=http://172.31.140.202:2380,i-06ed4eb078b00f623=http://172.31.153.93:2380,i-077dfb38fc0039342=http://172.31.159.46:2380,i-0a665d73f1d8c7cfa=http://172.31.131.127:2380', '-initial-cluster-token', 'etcd-cluster-etcd', '-initial-cluster-state', 'existing']
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108064 W | flags: unrecognized environment variable ETCDVERSION=3.0.15
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108416 I | etcdmain: etcd Version: 3.0.15
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108593 I | etcdmain: Git SHA: fc00305
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108771 I | etcdmain: Go Version: go1.6.3
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.108929 I | etcdmain: Go OS/Arch: linux/amd64
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109088 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109278 N | etcdmain: the server is already initialized as member before, starting as etcd member...
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109530 I | etcdmain: listening for peers on http://0.0.0.0:2380
Jun 28 11:59:09 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:09.109792 I | etcdmain: listening for client requests on 0.0.0.0:2379
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.859842 I | etcdserver: recovered store from snapshot at index 222437884
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860265 I | etcdserver: name = i-05982c02c444ea551
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860450 I | etcdserver: data dir = data
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860609 I | etcdserver: member dir = data/member
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860753 I | etcdserver: heartbeat = 100ms
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.860895 I | etcdserver: election = 1000ms
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.861047 I | etcdserver: snapshot count = 10000
Jun 28 11:59:17 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:17.861212 I | etcdserver: advertise client URLs = http://172.31.131.225:2379
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:18.446568 I | etcdserver: restarting member 53733ad8236d11ce in cluster 5cee1e413d02 at commit index 222455528
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: 2017-06-28 11:59:18.447078 C | raft: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: panic: 53733ad8236d11ce state.commit 222455528 is out of range [222437884, 222438776]
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]:
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: goroutine 1 [running]:
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: panic(0xd450a0, 0xc8355fb560)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/usr/local/go/src/runtime/panic.go:481 +0x3e6
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog.(*PackageLogger).Panicf(0xc820190620, 0x12375a0, 0x2b, 0xc833422f40, 0x4, 0x4)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/pkg/capnslog/pkg_logger.go:75 +0x191
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.(*raft).loadState(0xc836564270, 0x1b334, 0x88b67ac3e24bbdb2, 0xd4266e8, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:942 +0x2a2
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.newRaft(0xc8201078d8, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/raft.go:225 +0x8ff
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft.RestartNode(0xc8201078d8, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/raft/node.go:213 +0x45
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.restartNode(0xc82006fb80, 0xc820066000, 0x29, 0xc820107d78, 0x0, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/raft.go:369 +0x7c7
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.NewServer(0xc82006fb80, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:348 +0x430e
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcd(0xc8201a8000, 0x0, 0x0, 0x0)
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:374 +0x245f
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.startEtcdOrProxyV2()
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/etcd.go:116 +0x2101
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain.Main()
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdmain/main.go:36 +0x21e
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: main.main()
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: #011/home/gyuho/go/src/github.com/coreos/etcd/cmd/main.go:28 +0x14
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:59:18,481 - Process 450 finished with exit code 2
Jun 28 11:59:18 ip-172-31-131-225 docker/27d00dc9d174[820]: WARNING 2017-06-28 11:59:18,481 - Sleeping 30 seconds before next try...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant