etcd raft should take database error in count

[armstrongli]

I have one 4 members' cluster. And one of the DB size is extremely larger than the other ones. And the larger one has used up all the spaces of the machine. The the whole cluster is unoperationable.

/ # ectl  put a 1
Error:  etcdserver: mvcc: database space exceeded

https://tess-master-cbtkd-8128.dev.*.*:4001, 87e6a178ab5a0f63, 3.0.15, 11 GB, true, 2868, 59480228
https://tess-master-temb2-9053.dev.*.*:4001, b3f0b5ddc047ed37, 3.0.15, 57 MB, false, 2868, 59480355
https://tess-master-13mny-8094.dev.*.*:4001, cc83177fcd721b4f, 3.0.15, 57 MB, false, 2868, 59480510
https://salt-master-7505.dev.*.*:4001, ea015efc8847afdf, 3.0.15, 57 MB, false, 2868, 59480579

There should be some techs to resolve such kind of issue.

[vimalk78]

can you use the --write-out=fileds option, it prints more details

./etcdctl endpoint status --endpoints=${ALL_ENDPOINTS} --write-out=fields

for my simple 3 node cluster it outputs like this

"ClusterID" : 17237436991929493444
"MemberID" : 9372538179322589801
"Revision" : 601
"RaftTerm" : 17
"Version" : "3.1.0-rc.1+git"
"DBSize" :" 114688
"Leader" :" 9372538179322589801
"RaftIndex" :" 620
"RaftTerm" :" 17
"Endpoint" : "127.0.0.1:2379"

[armstrongli]

@vimalk78 The version of the cluster is etcd 3.0.15.

[vimalk78]

etcd raft should take database error in count

@armstrongli i did not understand what you mean by this

from the other issue, the db size has become normal after defragmentation #7116

[gyuho]

one of the DB size is extremely larger than the other ones.

Did you add other members later? How did this happen?

Also see https://github.com/coreos/etcd/blob/master/Documentation/op-guide/maintenance.md#space-quota.

[armstrongli]

Did you add other members later? How did this happen?

No. Those members are added months ago. And they have been running for weeks. I didn't notice that until exceeded database space.

The space size decreased to 3.7 MB from 11 GB after defragment. Then catch up with other members to 57MB after minutes.
cc @xiang90

[xiang90]

I am closing this. raft should know nothing about db. The original issue is also resolved and closed.