Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distinguish unwritable channel from blocked transport thread #77838

Open
DaveCTurner opened this issue Sep 16, 2021 · 1 comment
Open

Distinguish unwritable channel from blocked transport thread #77838

DaveCTurner opened this issue Sep 16, 2021 · 1 comment
Labels
:Distributed Coordination/Network Http and internode communication implementations >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.

Comments

@DaveCTurner
Copy link
Contributor

Today we log slow processing of outbound messages like this:

sending transport message [Request{indices:data/write/bulk[s][r]}{123456}{false}{false}{false}] of size [1234] on [Netty4TcpChannel{localAddress=/99.99.99.99:9999, remoteAddress=99.99.99.99/99.99.99.99:9999, profile=default}] took [12345ms] which is above the warn threshold of [5000ms] with success [true]

The took time records the time until the message is completely written to the channel, which includes time spent waiting for the channel to become writeable as well as time spent waiting for the transport thread to get around to the transmission. We care about the latter (it indicates a bug) much more than the former (it indicates network saturation). It would be good to distinguish these two causes.
@DaveCTurner DaveCTurner added >enhancement :Distributed Coordination/Network Http and internode communication implementations labels Sep 16, 2021
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Sep 16, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@DaveCTurner DaveCTurner mentioned this issue Sep 16, 2021
Closed
@DaveCTurner DaveCTurner mentioned this issue Nov 10, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Coordination/Network Http and internode communication implementations >enhancement Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@DaveCTurner @elasticmachine