Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adjust zebrad Buffer sizes, and document Buffer constraints #1685

Closed
5 tasks
teor2345 opened this issue Feb 3, 2021 · 2 comments
Closed
5 tasks

Adjust zebrad Buffer sizes, and document Buffer constraints #1685

teor2345 opened this issue Feb 3, 2021 · 2 comments
Labels
A-docs Area: Documentation A-rust Area: Updates to Rust code C-cleanup Category: This is a cleanup C-security Category: Security issues I-heavy Problems with excessive memory, disk, or CPU usage I-invalid-data Zebra relies on invalid or untrusted data, or sends invalid data

Comments

@teor2345
Copy link
Contributor

teor2345 commented Feb 3, 2021

Is your feature request related to a problem? Please describe.

zebrad uses a lot of Buffered services, but we haven't really checked that their bounds are high enough.

The underlying issues are:

  • If lots of tasks with data dependencies use a Buffer or Batch with a small bound, Zebra can hang.
  • Buffers or Batches with small bounds can also limit concurrency, particularly if the underlying service does a lot of work in its call.

Describe the solution you'd like

  • Review Buffer bounds across zebrad
  • Consider memory denial of service risks, particularly for the inbound buffer, inbound downloads buffer, and sync downloads buffer

Document the Buffer constraints in:

  • commands::start
  • components::sync::ChainSync
  • components::sync::Inbound

See #1593 for details - ideally we should point to an RFC here.

Describe alternatives you've considered

Do nothing: the code might hang if future changes break the poll_ready/call invariants.

Additional context

This issue was discovered during the review in #1593. It is a routine cleanup.

@teor2345 teor2345 added A-docs Area: Documentation A-rust Area: Updates to Rust code C-cleanup Category: This is a cleanup S-needs-triage Status: A bug report needs triage P-Medium labels Feb 3, 2021
@teor2345 teor2345 changed the title Use ServiceExt::oneshot in ChainSync Use ServiceExt::oneshot in zebrad::components::sync Feb 3, 2021
@teor2345 teor2345 changed the title Use ServiceExt::oneshot in zebrad::components::sync Document the poll_ready/call hang risk in zebrad Feb 15, 2021
@teor2345
Copy link
Contributor Author

This task has been revised based on the fixes to the underlying Buffer issue, see #1593 for details.

@teor2345 teor2345 changed the title Document the poll_ready/call hang risk in zebrad Document the Buffer/Batch hang risk in zebrad Feb 15, 2021
@teor2345 teor2345 changed the title Document the Buffer/Batch hang risk in zebrad Adjust zebrad Buffer sizes, and document Buffer constraints Feb 15, 2021
@mpguerra mpguerra removed the S-needs-triage Status: A bug report needs triage label Feb 18, 2021
@teor2345 teor2345 added C-security Category: Security issues I-invalid-data Zebra relies on invalid or untrusted data, or sends invalid data I-heavy Problems with excessive memory, disk, or CPU usage labels May 24, 2021
@mpguerra
Copy link
Contributor

we will create other issues as needed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-docs Area: Documentation A-rust Area: Updates to Rust code C-cleanup Category: This is a cleanup C-security Category: Security issues I-heavy Problems with excessive memory, disk, or CPU usage I-invalid-data Zebra relies on invalid or untrusted data, or sends invalid data
Projects
None yet
Development

No branches or pull requests

2 participants