Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: unexpected error from lease detach #6133

Closed
imkira opened this issue Aug 8, 2016 · 10 comments
Closed

panic: unexpected error from lease detach #6133

imkira opened this issue Aug 8, 2016 · 10 comments

Comments

@imkira
Copy link

imkira commented Aug 8, 2016 •
edited
Loading

I am using etcd v3.0.4 (release) with a 3 node cluster using the following config:

{
  "name": "etcd2",
  "data-dir": "/var/lib/etcd",
  "heartbeat-interval": 100,
  "election-timeout": 1000,
  "strict-reconfig-check": true,
  "initial-cluster": "etcd1=http://10.0.0.1:2380,etcd2=http://10.0.0.2:2380,etcd3=http://10.0.0.3:2380",
  "initial-cluster-token": "hidden",
  "initial-cluster-state": "new",
  "initial-advertise-peer-urls": "http://10.0.0.2:2380",
  "listen-peer-urls": "http://10.0.0.2:2380",
  "listen-client-urls": "http://10.0.0.2:2379,http://127.0.0.1:2379",
  "advertise-client-urls": "http://10.0.0.2:2379"
}

For another reason the cluster crashed and now when I try to boot up node 2, I get this:

2016-08-08 22:25:48.634421 I | etcdmain: Loading server configuration from "/etc/etcd/etcd.conf"
2016-08-08 22:25:48.635040 I | etcdmain: etcd Version: 3.0.4
2016-08-08 22:25:48.635054 I | etcdmain: Git SHA: d53923c
2016-08-08 22:25:48.635060 I | etcdmain: Go Version: go1.6.3
2016-08-08 22:25:48.635067 I | etcdmain: Go OS/Arch: linux/amd64
2016-08-08 22:25:48.635075 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
2016-08-08 22:25:48.635131 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2016-08-08 22:25:48.635246 I | etcdmain: listening for peers on http://10.0.0.2:2380
2016-08-08 22:25:48.635315 I | etcdmain: listening for client requests on 10.0.0.2:2379
2016-08-08 22:25:48.635367 I | etcdmain: listening for client requests on 127.0.0.1:2379
2016-08-08 22:25:48.637656 I | etcdserver: recovered store from snapshot at index 1250447
2016-08-08 22:25:48.637673 I | etcdserver: name = etcd2
2016-08-08 22:25:48.637679 I | etcdserver: data dir = /var/lib/etcd
2016-08-08 22:25:48.637685 I | etcdserver: member dir = /var/lib/etcd/member
2016-08-08 22:25:48.637691 I | etcdserver: heartbeat = 100ms
2016-08-08 22:25:48.637696 I | etcdserver: election = 1000ms
2016-08-08 22:25:48.637708 I | etcdserver: snapshot count = 10000
2016-08-08 22:25:48.637719 I | etcdserver: advertise client URLs = http://10.0.0.2:2379
2016-08-08 22:25:48.995781 I | etcdserver: restarting member 9028d3f66740926e in cluster cce5c99013f6a765 at commit index 1251175
2016-08-08 22:25:48.996074 I | raft: 9028d3f66740926e became follower at term 134034
2016-08-08 22:25:48.996120 I | raft: newRaft 9028d3f66740926e [peers: [9028d3f66740926e,c34a856543a09c53,f5c9c300eab4cfb9], term: 134034, commit: 1251175, applied: 1250447, lastindex: 1251192, lastterm: 1317]
2016-08-08 22:25:48.996419 I | api: enabled capabilities for version 3.0
2016-08-08 22:25:48.996451 I | membership: added member 9028d3f66740926e [http://10.0.0.2:2380] to cluster cce5c99013f6a765 from store
2016-08-08 22:25:48.996466 I | membership: added member c34a856543a09c53 [http://10.0.0.1:2380] to cluster cce5c99013f6a765 from store
2016-08-08 22:25:48.996480 I | membership: added member f5c9c300eab4cfb9 [http://10.0.0.3:2380] to cluster cce5c99013f6a765 from store
2016-08-08 22:25:48.996499 I | membership: set the cluster version to 3.0 from store
2016-08-08 22:25:49.015628 I | rafthttp: starting peer c34a856543a09c53...
2016-08-08 22:25:49.015718 I | rafthttp: started HTTP pipelining with peer c34a856543a09c53
2016-08-08 22:25:49.021564 I | rafthttp: started peer c34a856543a09c53
2016-08-08 22:25:49.021638 I | rafthttp: added peer c34a856543a09c53
2016-08-08 22:25:49.021666 I | rafthttp: starting peer f5c9c300eab4cfb9...
2016-08-08 22:25:49.021689 I | rafthttp: started HTTP pipelining with peer f5c9c300eab4cfb9
2016-08-08 22:25:49.033823 I | rafthttp: started peer f5c9c300eab4cfb9
2016-08-08 22:25:49.033902 I | rafthttp: added peer f5c9c300eab4cfb9
2016-08-08 22:25:49.033961 I | etcdserver: starting server... [version: 3.0.4, cluster version: 3.0]
2016-08-08 22:25:49.034365 I | rafthttp: started streaming with peer f5c9c300eab4cfb9 (writer)
2016-08-08 22:25:49.034402 I | rafthttp: started streaming with peer c34a856543a09c53 (writer)
2016-08-08 22:25:49.034429 I | rafthttp: started streaming with peer c34a856543a09c53 (writer)
2016-08-08 22:25:49.034456 I | rafthttp: started streaming with peer c34a856543a09c53 (stream MsgApp v2 reader)
2016-08-08 22:25:49.034756 I | rafthttp: started streaming with peer c34a856543a09c53 (stream Message reader)
2016-08-08 22:25:49.034906 I | rafthttp: started streaming with peer f5c9c300eab4cfb9 (writer)
2016-08-08 22:25:49.034952 I | rafthttp: started streaming with peer f5c9c300eab4cfb9 (stream MsgApp v2 reader)
2016-08-08 22:25:49.035758 I | rafthttp: started streaming with peer f5c9c300eab4cfb9 (stream Message reader)
2016-08-08 22:25:49.037199 I | raft: 9028d3f66740926e [logterm: 1317, index: 1251192, vote: 0] ignored vote from c34a856543a09c53 [logterm: 134034, index: 1634905] at term 134034: lease is not expired (remaining ticks: 1)
panic: unexpected error from lease detach

goroutine 113 [running]:
panic(0xd3d2a0, 0xc82026cb40)
    /usr/local/go/src/runtime/panic.go:481 +0x3e6
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc.(*store).put(0xc8200ec540, 0xc82023eb40, 0x52, 0x60, 0xc8202db050, 0x24, 0x30, 0x1c535657224e72a6)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/kvstore.go:553 +0xe2d
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc.(*store).Put(0xc8200ec540, 0xc82023eb40, 0x52, 0x60, 0xc8202db050, 0x24, 0x30, 0x1c535657224e72a6, 0xc820440400)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/kvstore.go:144 +0x89
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc.(*watchableStore).Put(0xc82000d290, 0xc82023eb40, 0x52, 0x60, 0xc8202db050, 0x24, 0x30, 0x1c535657224e72a6, 0x0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/mvcc/watchable_store.go:93 +0xfd
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Put(0xc820020440, 0xffffffffffffffff, 0xc8202570c0, 0x1, 0x0, 0x0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:174 +0x2fb
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Put(0xc82026ae80, 0xffffffffffffffff, 0xc8202570c0, 0xc82023eb40, 0x0, 0x0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:630 +0xbd
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Put(0xc82048f940, 0xffffffffffffffff, 0xc8202570c0, 0x7fd5bd754ed8, 0x0, 0x0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply_auth.go:59 +0xfb
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*applierV3backend).Apply(0xc820020440, 0xc820289340, 0x10)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply.go:104 +0x21f
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*quotaApplierV3).Apply(0xc82026ae80, 0xc820289340, 0xc82027dc28)
    <autogenerated>:167 +0x59
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*authApplierV3).Apply(0xc82048f940, 0xc820289340, 0x0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/apply_auth.go:50 +0x1e8
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntryNormal(0xc8200b0000, 0xc82007f710)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1086 +0x4b6
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).apply(0xc8200b0000, 0xc82029c048, 0x2d8, 0x31b, 0xc820263b40, 0x0, 0x0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:1027 +0xc2
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyEntries(0xc8200b0000, 0xc820263b40, 0xc8200966e0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:751 +0x2bd
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).applyAll(0xc8200b0000, 0xc820263b40, 0xc8200966e0)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:614 +0xb4
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver.(*EtcdServer).run.func2(0x7fd5bd7532e8, 0xc820263b00)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/etcdserver/server.go:593 +0x32
github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.(*fifo).run(0xc820169260)
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:160 +0x323
created by github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule.NewFIFOScheduler
    /home/gyuho/go/src/github.com/coreos/etcd/cmd/vendor/github.com/coreos/etcd/pkg/schedule/schedule.go:71 +0x27d

Is this supposed to be fixed by #6098 ?
@xiang90
Copy link
Contributor

xiang90 commented Aug 8, 2016

@imkira Is it a fresh cluster from 3.0.4? Or it was an old cluster upgraded to 3.0.4? Are you able to reproduce the issue?

@xiang90
Copy link
Contributor

xiang90 commented Aug 8, 2016

@imkira Can you send the data dirs of the 3 nodes to me at [email protected]?

@imkira
Copy link
Author

imkira commented Aug 8, 2016 •
edited
Loading

@xiang90 it is a fresh new 3.0.4 cluster.
Yes, I just need to restart etcd. I have just sent you the data.
Thanks in advance.

@xiang90
Copy link
Contributor

xiang90 commented Aug 8, 2016

@imkira Do you have data dirs for other members? Do other member also panic?

@imkira
Copy link
Author

imkira commented Aug 8, 2016

@xiang90 Yes I do have the data dirs. No, they don't panic. I will send them right away.

@imkira
Copy link
Author

imkira commented Aug 9, 2016

@xiang90 just sent you the data dirs now. Thanks in advance

@xiang90
Copy link
Contributor

xiang90 commented Aug 9, 2016

Do you still have the etcd server log before crashing?

@imkira
Copy link
Author

imkira commented Aug 9, 2016

@xiang90 Yes I do. I have just sent you. Please take a look. Thanks

@xiang90
Copy link
Contributor

xiang90 commented Aug 9, 2016

I looked into the data dirs, and I can confirm you that your issue is fixed by #6098.

For the huge memory usage issue, can you please open another issue?

@xiang90 xiang90 closed this as completed Aug 9, 2016
@imkira
Copy link
Author

imkira commented Aug 9, 2016

Thanks a lot!

I have just filed issue #6134 to address the memory usage problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@imkira @xiang90