Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcdv3 client warn message: lease keepalive response queue is full; dropping response send #2249

Closed
mingjunyang opened this issue Oct 28, 2020 · 2 comments

Comments

@mingjunyang
Copy link

env

etcd server: 3.3.25

ct_server: 1.1.1
log_signer: 1.3.11
log_server: 1.3.11

server info : Linux ip-xx.xx.xx.xx.ap-mmmm-1.compute.internal 4.14.198-152.320.amzn2.x86_64 #1 SMP Wed Sep 23 23:57:28 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

when server start running a few moment, the console print warning message.

log_signer_1_d621790f498d | {"level":"warn","ts":"2020-10-28T02:52:39.049Z","caller":"clientv3/lease.go:524","msg":"lease keepalive response queue is full; dropping response send","queue-size":16,"queue-capacity":16}
log_server_1_891d40c2df74 | {"level":"warn","ts":"2020-10-28T02:52:44.006Z","caller":"clientv3/lease.go:524","msg":"lease keepalive response queue is full; dropping response send","queue-size":16,"queue-capacity":16}

I don't know this warning should or not affect long-term operation.

I think it might be related to this issues etcd-io/etcd#9911 . the man report server memory highly load due to too many connections.

@AlCutter
Copy link
Member

Hi,

This is not an error we've come across before, but it sounds like it might be an issue with etcd/etcd client internals.

I found this blog post which seems to be written by someone who had a similar issue and looked into what was causing it, perhaps it may give some insight:
https://blog.csdn.net/puss0/article/details/96481856

mhutchinson added a commit that referenced this issue Jan 19, 2022
This PR can let trillian proactively Listen "LeaseKeepAliveResponse" channel returned by KeepAlive in ETCD client. When automatic renewal interruption is detected, Exit the program by canceling the context.

Fixes #2654,#2249

Co-authored-by: Simba Peng <[email protected]>
Co-authored-by: Martin Hutchinson <[email protected]>
@px3303
Copy link
Contributor

px3303 commented Jan 19, 2022

@AlCutter

Fixed by #2655

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants