BTL TCP on macOSX broken in v4.0.x and master

[gpaulsen]

As also reported by @rhc54 and @jsquyres in https://www.mail-archive.com/[email protected]//msg20758.html

 mpirun -np 2 --mca btl "tcp,self" ./imb.x
[GRP-MBP][[39843,1],0][btl_tcp_endpoint.c:731:mca_btl_tcp_endpoint_start_connect] bind() failed: Invalid argument (22)
[GRP-MBP:04122] *** An error occurred in MPI_Bcast
[GRP-MBP:04122] *** reported by process [2611150849,0]
[GRP-MBP:04122] *** on communicator MPI_COMM_WORLD
[GRP-MBP:04122] *** MPI_ERR_OTHER: known error not in list
[GRP-MBP:04122] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[GRP-MBP:04122] ***    and potentially your MPI job)

[gpaulsen]

haven't tried 3.1.x or earlier yet.

[bosilca]

#5819 is what you are looking for.

[gpaulsen]

I'll give it a try tonight.

I did learn that -mca btl self,tcp -mca btl_tcp_if_include lo0 helps it pass for mpi hello world, but more complext apps like IMB or rings still failes in MPI_Bcast or MPI_Send.

[bosilca]

This might be the right time to have a clear discussion about what we want to support and what we don't. The only reason to have multiple IPs on the same interface is for load balance and QoS, 2 things that we mildly care about (there is also because "we can", but let's ignore this for now). If we drop all addresses but the one on the first device, and use multi-links we will be able to achieve the same outcome but with a selection logic greatly simplified, and a cleaner code base. I added this topic to the developer meeting agenda in few weeks.

[gpaulsen]

I've cherry-picked this to the v4.0.x branch, and verified that I can run on MACOSX with or without the if_include, with only btl self,tcp. with both hello_world and IMB.

[jsquyres]

I confirm that this is a problem on both master and v4.0.x. It does not happen on v3.1.x.

I'd argue that this is a blocker for v4.0.0.

[jsquyres]

BTW, it's not clear yet that #5819 is the correct fix (i.e., it fixes the issue, but there's still some debate as to whether it is the correct fix or not).

[gpaulsen]

Fixed on master and v4.0.x (I also verified), and not an issue on v3.1.x. Closing this issue. Thanks to all who helped out on this one.