Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Occasional UDP packet drop on validator nodes #1224

Closed
pgarg66 opened this issue Sep 14, 2018 · 0 comments
Closed

Occasional UDP packet drop on validator nodes #1224

pgarg66 opened this issue Sep 14, 2018 · 0 comments

Comments

@pgarg66
Copy link
Contributor

pgarg66 commented Sep 14, 2018

Some packet drops are noticed on validator nodes due to rcvbuf_error. Most UDP traffic to validators consists of Blobs (broadcasted/forwarded/repaired). The first two category of data is received on replicate port.

The packet drop is not as significant as on the leader node. But, any dropped blob will trigger repair request and would impact the overall performance.

screen shot 2018-09-14 at 12 21 00 pm

Proposed solutions:

  1. Use multiple sockets to receive data (similar to receiving transactions on the leader node)
  2. Use recvmmsg() API to fetch blobs from the socket.
@pgarg66 pgarg66 changed the title Use multiple sockets for replicate port on validator nodes Occasional UDP packet drop on validator nodes Sep 14, 2018
pgarg66 referenced this issue in pgarg66/solana Sep 14, 2018
@pgarg66
Use multiple sockets for receiving blobs on validators
0578889 
- The blobs that are broadcasted by leader or retransmitted by peer
  validators are received on replicate_port
- Using reuse_addr/reuse_port, multiple sockets can be opened for
  the same port
- This allows the kernel to queue data to user space app on multiple
  socket queues, preventing over-running one queue
- This helps with reducing packets dropped due to queue over-runs

Fixes anza-xyz#1224
@pgarg66 pgarg66 mentioned this issue Sep 14, 2018
Merged
pgarg66 referenced this issue in pgarg66/solana Sep 14, 2018
@pgarg66
Use multiple sockets for receiving blobs on validators
fefcda8 
- The blobs that are broadcasted by leader or retransmitted by peer
  validators are received on replicate_port
- Using reuse_addr/reuse_port, multiple sockets can be opened for
  the same port
- This allows the kernel to queue data to user space app on multiple
  socket queues, preventing over-running one queue
- This helps with reducing packets dropped due to queue over-runs

Fixes anza-xyz#1224
pgarg66 added a commit that referenced this issue Sep 14, 2018
@pgarg66
Use multiple sockets for receiving blobs on validators (#1228)
e142aaf 
* Use multiple sockets for receiving blobs on validators

- The blobs that are broadcasted by leader or retransmitted by peer
  validators are received on replicate_port
- Using reuse_addr/reuse_port, multiple sockets can be opened for
  the same port
- This allows the kernel to queue data to user space app on multiple
  socket queues, preventing over-running one queue
- This helps with reducing packets dropped due to queue over-runs

Fixes #1224

* Fixed failing tests
This was referenced Jan 17, 2022
Open
Closed
@behzadnouri behzadnouri mentioned this issue Apr 23, 2024
Open
palinko91 pushed a commit to palinko91/solana that referenced this issue May 8, 2024
@jeffwashington @brooksprumo
make ancient appending default to packing (solana-labs#1224)
6725fe3 
* make ancient appending default to packing

* appending ancient packing doesn't re-write packed ancient append vecs from previous runs

* Update accounts-db/src/accounts_db.rs

Co-authored-by: Brooks <[email protected]>

* use default

---------

Co-authored-by: Brooks <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pgarg66