-
Notifications
You must be signed in to change notification settings - Fork 28.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-24029][core] Set SO_REUSEADDR on listen sockets. #21110
Conversation
This allows sockets to be bound even if there are still sockets from a previous application that are still pending closure. It avoids bind issues when, for example, re-starting the SHS. Don't enable the option on Windows though. The following page explains some odd behavior that this option can have there: https://msdn.microsoft.com/en-us/library/windows/desktop/ms740621%28v=vs.85%29.aspx I intentionally ignored server sockets that always bind to ephemeral ports, since those don't benefit from this options.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -98,7 +99,8 @@ private void init(String hostToBind, int portToBind) { | |||
.group(bossGroup, workerGroup) | |||
.channel(NettyUtils.getServerChannelClass(ioMode)) | |||
.option(ChannelOption.ALLOCATOR, allocator) | |||
.childOption(ChannelOption.ALLOCATOR, allocator); | |||
.childOption(ChannelOption.ALLOCATOR, allocator) | |||
.childOption(ChannelOption.SO_REUSEADDR, !SystemUtils.IS_OS_WINDOWS); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It could be even nicer if we leave a link and few comments here why we don't enable on Windows specifically though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where would you put it? Every place where this is set? Not a fan of that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I usually put some comments when it takes longer than usual to read tho. It's sometimes hard to debug in particular when it's OS specific. It bothered me to leave a comment saying you could leave it optionally if you feel in the same way. am fine as is too.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM.
Test build #89602 has finished for PR 21110 at commit
|
retest this please |
From a very quick look, the flakiness looks global. |
Test build #89611 has finished for PR 21110 at commit
|
retest this please |
1 similar comment
retest this please |
Test build #89667 has finished for PR 21110 at commit
|
Merged to master. |
This allows sockets to be bound even if there are sockets
from a previous application that are still pending closure. It
avoids bind issues when, for example, re-starting the SHS.
Don't enable the option on Windows though. The following page
explains some odd behavior that this option can have there:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms740621%28v=vs.85%29.aspx
I intentionally ignored server sockets that always bind to
ephemeral ports, since those don't benefit from this option.