Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[develop] Disconnect monitor will pick up normal shutdowns #223

Closed
mschubert opened this issue Dec 5, 2020 · 3 comments
Closed

[develop] Disconnect monitor will pick up normal shutdowns #223

mschubert opened this issue Dec 5, 2020 · 3 comments
Labels

Comments

@mschubert
Copy link
Owner

Q(function(x) Sys.sleep(x), x=c(0, 10), n_jobs=2)

first job is shutting down, disconnect monitor may (will?) treat it as error and fail

@mschubert mschubert added the bug label Dec 5, 2020
@mschubert mschubert changed the title [develop] Disconnect monitor will pick up normal shutdowns Disconnect monitor will pick up normal shutdowns Dec 27, 2020
@mschubert
Copy link
Owner Author

mschubert commented Dec 27, 2020

This was merged to master as well, please use CRAN version until this is resolved

master reverted

@mschubert mschubert changed the title Disconnect monitor will pick up normal shutdowns [works on CRAN] Disconnect monitor will pick up normal shutdowns Dec 27, 2020
@mschubert
Copy link
Owner Author

To add some more explanation here, every time you get a

1 peer(s) lost

without a worker crashing first, that is this bug.

This is caused by the fact that the monitor signal (my SO question here)

ZMQ_EVENT_DISCONNECTED
The socket was disconnected unexpectedly. The event value is the FD of the underlying network socket. Warning: this socket will be closed.

also happens on a normal closing (or closing after explicitly disconnecting) the socket. So it looks like I need to track whether a disconnect is clean or not on the application level, where peer identities are normally abstracted away by the ZeroMQ sockets I'm using.

So this requires a fair bit of rewrite on the monitoring logic it seems

@mschubert mschubert changed the title [works on CRAN] Disconnect monitor will pick up normal shutdowns [develop] Disconnect monitor will pick up normal shutdowns Mar 25, 2021
@mschubert mschubert added breaking and removed bug labels Mar 25, 2021
mschubert added a commit that referenced this issue Apr 10, 2021
@mschubert
Copy link
Owner Author

fixed with bundled libzmq

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant