Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make heartbeats wait for the connection queue to empty, with a timeout #1551

Closed
teor2345 opened this issue Jan 5, 2021 · 0 comments · Fixed by #2009
Closed

Make heartbeats wait for the connection queue to empty, with a timeout #1551

teor2345 opened this issue Jan 5, 2021 · 0 comments · Fixed by #2009
Labels
A-rust Area: Updates to Rust code C-enhancement Category: This is an improvement
Milestone
2021 Sprint 7

Comments

@teor2345
Copy link
Contributor

teor2345 commented Jan 5, 2021

Is your feature request related to a problem? Please describe.

In #1531, we fixed a bug where a heartbeat would wait forever for the request sink to be ready:
https://github.com/ZcashFoundation/zebra/pull/1531/files#diff-558ef42996967fa2dcffe0d47a6959571254ec41db7d9b0d26d7c3e240feadacR491

Instead, we made heartbeats close the connection if the queue is full. But this results in more connection churn than we need.

Describe the solution you'd like

Wait for the the request sink to be ready, or a timeout to elapse. If the timeout elapses, close the connection.

Describe alternatives you've considered

Do nothing. We churn connections more than needed, particularly under heavy load.

Revert to the old behaviour before #1531. Connections can hang forever.

Additional context

We should prioritise this work based on the performance impact of the extra connection churn.
@teor2345 teor2345 added A-rust Area: Updates to Rust code C-enhancement Category: This is an improvement S-needs-triage Status: A bug report needs triage labels Jan 5, 2021
@teor2345 teor2345 mentioned this issue Jan 5, 2021
Merged
1 task
@mpguerra mpguerra removed the S-needs-triage Status: A bug report needs triage label Feb 2, 2021
@teor2345 teor2345 changed the title Make heartbeats timeout if the connection request queue stays full Make heartbeats wait for the connection queue to empty, with a timeout Apr 13, 2021
teor2345 added a commit to teor2345/zebra that referenced this issue Apr 13, 2021
@teor2345
Make heartbeats wait for the connection queue to empty, with a timeout (
80ec455 
ZcashFoundation#1551)
teor2345 added a commit to teor2345/zebra that referenced this issue Apr 14, 2021
@teor2345
Make heartbeats wait for the connection queue to empty, with a timeout (
44422fd 
ZcashFoundation#1551)

Also cleanup the heartbeat code, so each heartbeat request/response runs
in a future with a single timeout.
@teor2345 teor2345 mentioned this issue Apr 14, 2021
Merged
2 tasks
@mpguerra mpguerra added this to the 2021 Sprint 7 milestone Apr 14, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-rust Area: Updates to Rust code C-enhancement Category: This is an improvement
Projects
None yet
Milestone
2021 Sprint 7
Development

Successfully merging a pull request may close this issue.

Make sure each peer heartbeat has a timeout teor2345/zebra
2 participants
@mpguerra @teor2345