RFC: Define wait(req) to use threadcall

[simonbyrne]

Prompted by this discussion I tried to see if I could use @threadcall to wait on nonblocking MPI operations inside tasks.

And it seems to work! I've defined this to be a method of Base.wait, in that it acts mostly the same way (i.e. will yield until it is complete). Obviously one problem is that it is limited by the size of the libuv thread pool, but I don't think that would be a big issue.

If people are in favor, I can define analogous waitany/waitall/waitsome operations.

cc: @giordano @stevengj @vchuravy @fverdugo

[stevengj]

Couldn’t waitall be defined as a wait method that takes an array of requests, maybe with a flag argument for waitall vs waitany?

[simonbyrne]

One issue is that on 32-bit windows we need to use the stdcall convention: this is usually handled by the @mpicall/@mpichk macros, but @threadcall doesn't seem to support nonstandard calling conventions.

On the other hand, it passed tests so 🤷 ?

[simonbyrne]

(alternatively, we could disable this on 32-bit Windows, like we do the custom operator stuff)

[vchuravy]

It would be good to have this be configurable, I have to think a bit more about this, but MPI implementation are free to not be threadsafe (or use libraries under the hood that are not threadsafe, I suspect that half the UCX builds are built without support for threads...)

…

On Wed, Feb 24, 2021, 23:29 Simon Byrne ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In src/pointtopoint.jl <#452 (comment)>: > + errcode = @threadcall((:MPI_Wait, libmpi), Cint, + (Ptr{MPI_Request}, Ptr{Status}), + req, stat_ref) (alternatively, we could disable this on 32-bit Windows, like we do the custom operator stuff) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#452 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABDO2XUWEFAODNEL422JR3TAXG37ANCNFSM4YFNR2XA> .

[fverdugo]

Nice to see this!

Is this approach composable? In some situations, we need to wait for a precondition (Request 1) in order to start a non-bloking communication that will generate Request 2. It would be nice to be able to compose these two requests in a asynchronous Task (or whatever object) that waits for Request 1, starts the non-blocking communication and waits for Request 2.

[simonbyrne]

It would be good to have this be configurable, I have to think a bit more about this, but MPI implementation are free to not be threadsafe (or use libraries under the hood that are not threadsafe, I suspect that half the UCX builds are built without support for threads...)

Yes, the user would be required to initialize using MPI.Init_thread(MPI.THREAD_SERIALIZED) or MPI.Init_thread(MPI.THREAD_MULTIPLE).

Is this approach composable? In some situations, we need to wait for a precondition (Request 1) in order to start a non-bloking communication that will generate Request 2. It would be nice to be able to compose these two requests in a asynchronous Task (or whatever object) that waits for Request 1, starts the non-blocking communication and waits for Request 2.

Yes, you can do something like:

request1 = MPI.Irecv!(...)

commtask = @async begin
    wait(request1)
    request2 = MPI.Send(...)
    wait(request2)
end
# do other work
wait(commtask)

Alternatively, it might be better to just use Julia threading for this:

request1 = MPI.Irecv!(...)

commtask = Threads.@spawn begin
    MPI.Wait!(request1)
    request2 = MPI.Send(...)
    MPI.Wait!(request2)
end
# do other work
wait(commtask)

The main difference is that the first is blocking a libuv thread, the second is blocking a Julia thread. I'm not sure what other implications of this are.

[simonbyrne]

After further thought, I don't think this is a good idea.

I think it would be better to define wait as a simple busy loop with yield, e.g.

function Base.wait(req::MPI.Request)
   while !MPI.Test(req)
       yield()
   end
end

This should work, since

If an MPI_TEST that completes a receive is repeatedly called with the same arguments,
and a matching send has been started, then the call will eventually return flag = true, unless
the send is satisfied by another receive. If an MPI_TEST that completes a send is repeatedly
called with the same arguments, and a matching receive has been started, then the call will
eventually return flag = true, unless the receive is satisfied by another send

[fverdugo]

@simonbyrne thanks for the new idea.

In my code I am finally doing something like

t = @async begin
   while !MPI.Test(req)
       yield()
   end
end

and then one can consume task t as usual in Julia. Perhaps we don't even need to define the new method for wait. Just add some comment on the docstrings of Test and TestAll regarding on how to use them in combination with @async.

[simonbyrne]

Just add some comment on the docstrings of Test and TestAll regarding on how to use them in combination with @async.

That's a good idea, would you mind opening a draft PR?

[fverdugo]

That's a good idea, would you mind opening a draft PR?

Yes! I'll do that.