allow worker to work when a host has multiple IP addresses

[swadey]

This is the body of the email i sent to julia-dev:

I've been trying to run a couple of parallel things and I've come across two problems:

1. I'm unable to start julia with workers when I'm connected to my institute's vpn. I get this error:

$ julia -p 4
could not connect to 192.168.1.106:9012, errno=60

in Worker at multi.jl:110
in Worker at multi.jl:115
in start_remote_workers at multi.jl:947
in addprocs_local at multi.jl:1008
in process_options at client.jl:202
in _start at client.jl:255

The problem seems to be that when julia launches workers, it expects them to report their IP addresses (even if the worker is started locally). If a machine has multiple addresses, the first non loopback adapter is chosen (via a call to getipaddr() which c-calls getlocalip()), which results in:

$ julia --worker
julia_worker:9021#192.168.1.106

The problem is that many VPNs (including our) don't allow bridging, so when the VPN is connected, connections to the primary adapter are blocked.

I worked around this by adding a mode argument to --worker:

--worker local
--worker remote

start_worker() checks this flag to decide whether to use "localhost" or the result of getipaddr().

[ViralBShah]

Is it possible to create this pull request without all the Merges? I guess they can be squashed, or it may just be simpler to create a new pull request.

[swadey]

sure, let me look into it.

thanks,
wade

On Wed, Jan 2, 2013 at 9:16 PM, Viral B. Shah [email protected]:

Is it possible to create this pull request without all the Merges? I guess
they can be squashed, or it may just be simpler to create a new pull
request.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1879#issuecomment-11833133.

[vtjnash]

This is interesting, but I think it is incorrect and should be reverted. With this patch, if i understand correctly, I think it is now impossible to use both local workers and remote workers (although I didn't test it). The better behavior might be to store all available (non-loopback) ip-addresses and try each in turn from the destination node.

[swadey]

Hi Jameson, I don't think that's a problem. Only in local mode does localhost get used as the addr for a worker. In the remote case, things work exactly as they did before.

[vtjnash]

In local-only and remote-only mode, it would be OK. However, the reported ip address is used by all computers, so in a mixed environment, this will break existing usage. There is an implicit assumption in the rest of the code that all workers can talk to any other worker given the ip address reported by the worker.

[ViralBShah]

@vtjnash Do you still think we should revert/rework this?

[swadey]

Jameson is right, this breaks mixed mode environments and should be reverted (since I think the VPN scenario is less common). Now that I'm back in the office, I was able to setup a test with our cluster and it does, in fact, break mixed mode configurations.

I'm not sure how to solve this in general. The problem is the assumption that getipaddr() returns an address that is reachable by everyone. Perhaps, getipaddr() could check if it's able to connect to a specific address before returning the address?

[aviks]

Servers generally use something like a --bind-to option in situations like this, for the user to specify which interface to bind onto. So in situations where the default behaviour is incorrect due to complex configuration, you can specify explicitly the interface to use. So for a fully local setup, --bind-to lo . For mixed mode environments with a vpn, you can specify --bind-to vpn-if ..etc..

That may be a generic solution to this problem, rather than simply splitting into local and remote.

[swadey]

That seems like a good solution.

On Fri, Jan 4, 2013 at 11:27 AM, Avik Sengupta [email protected]:

Servers generally use something like a --bind-to option in situations
like this, for the user to specify which interface to bind onto. So in
situations where the default behaviour is incorrect due to complex
configuration, you can specify explicitly the interface to use. So for a
fully local setup, --bind-to lo . For mixed mode environments with a vpn,
you can specify --bind-to vpn-if ..etc..

That may be a generic solution to this problem, rather than simply
splitting into local and remote.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/1879#issuecomment-11888952.

[ViralBShah]

Should we then revert this commit and get the generic solution in place as suggested by @aviks? @vtjnash Is that the way to go?

[vtjnash]

Yes, that sounds good. Instead of remote and local, indicate the adapter to bind to. Unfortunately, it still won't work over a VPN that doesn't allow loopback, but that can probably be dealt with separately (if it is needed).