Initial sender/receiver adaptations for tile algorithms

[msimberg]

This is not meant to be anywhere near complete. It mostly contains a bunch of TODOs and notes for myself about what needs/can be adapted where. However, I'm opening this to start exposing others to the sender/receiver interfaces, and try to collect feedback on what sort of utilities/interfaces/functionality is useful/acceptable/harmful in your opinion.

I've changed the gemm task in one of the triangular solver implementations to use senders/receivers and it passes tests.

I see two primary benefits to using senders/receivers in DLAF:

• More flexible sender/receiver customization points should allow us to remove some of the hacks currently in place e.g. for tile lifetime management, which can with senders/receivers be managed in real customization points, rather than our ad-hoc dataflow_finalize.
• Potential optimizations when chaining work (I think there are actually very few, if no cases where this applies in DLAF), and potentially fewer heap allocations (although dataflow already does a decent job of this).

Short summary of the "sender adapters" that appear here:

• template <Backend B, typename Sender> auto transform(Sender&& sender) {...}: this is a backend-specific version of future::then(executor, ...)/dataflow(executor, ...)/execution::transform (the last one is from P1897). The reason it does not take an executor/scheduler/policy or similar is that the current proposals don't really talk about the kind of customization we want to do (inserting cuda streams etc.) and what is there is semantically a bit shaky/bound to change at least for this use case. The custom CUDA/MPI executors are already a bit questionable.
• keep_future: currently the default behaviour of a future as a sender is to get the value from the future and pass that to the continuation. keep_future is an alternative which passes the future itself, similar to how future::then takes a callable which takes a future as its argument. We might swap the default to be the keep_future behaviour. This functionality is mostly intended for the transition phase from futures to senders, and should not be needed in the end.
• when_all_lift: P1897 proposes a when_all that takes a pack of senders and triggers a continuation when all the input senders are done. However, it takes only senders, which means that if one has ready values one would have to wrap them in execution::just (this is equivalent to how one would have to do dataflow(f, make_ready_future(3)) if dataflow only supported futures as arguments). It would be nice if the proposed when_all was a bit more flexible in this regard (including accepting void senders), and we'll follow up on that. If the standard when_all ends up accepting only senders, this is still a useful extension for HPX/DLAF.

[msimberg]

bors try

[bors]

try

Build failed:

• ci/gitlab/full-pipeline

[msimberg]

Still a draft, but I've updated a few things based on some of the things I presented yesterday.

I've introduced a policy class, templated on the backend. The policy is supposed to be similar to an HPX execution policy, but is not currently one (not necessary at the moment, not for use with HPX parallel algoritms). In the future it could be a proper execution policy, or it could be replaced by an HPX execution policy. The reason for making it a proper class, rather than just the Backend enum is that it can hold properties, such as priorities. Since the executor concept in the latest P0443 is very simplified compared to the executors we're currently using I'm not using the new executors for anything here.

I've also added a few more overloads to the tile algorithms (trsm, gemm, etc.). That means that they currently have the following overloads:

• plain blocking CPU overload dispatching to blaspp/lapackpp
• plain asynchronous GPU overload dispatching to cuBLAS/cuSOLVER, which takes a stream/handle
• an overload that takes a policy and is blocking (this is equivalent to e.g. hpx::for_loop(hpx::execution::par, ...)
• an overload that takes a policy and one sender and returns a sender (somewhat close to hpx::dataflow(hpx::for_loop, hpx::execution::par, ...) except it has lazy semantics; with senders it would be used e.g. as hpx::sync_wait(hpx::for_loop(hpx::execution::par, hpx::execution::when_all(...)))
• an overload that takes only a policy (this is only for use with operator|, e.g. hpx::execution::when_all(...) | hpx::for_loop(hpx::execution::par) | hpx::execution::sync_wait())

With these changes I think the first two overloads could be moved to internal, and the trsm_o etc. callable objects would never show up in the algorithms (they can be kept around for convenience, but still be internal).

Overall I myself quite like this direction. It aligns quite well with where I think the standard library is possibly headed. The policy overloads are also higher level than the dataflow + executor versions, and gives us room to change things below those overloads (either because we want to or because the proposals change). The higher level ideas are unlikely to change as much. We've also started looking at adding exactly these kinds of overloads to the HPX parallel algorithms.

I think from an algorithm-writers perspective the most important thing is how e.g. this looks: https://github.com/msimberg/DLA-Future/blob/e578e959819142021f12439414a30d7aad193e78/include/dlaf/solver/triangular/impl.h#L39-L42. If this looks like a decent direction I'll start tidying up loose ends (for this iteration...).

Let me know what you think (including @aurianer, @biddisco).

[albestro]

I think from an algorithm-writers perspective the most important thing is how e.g. this looks: https://github.com/msimberg/DLA-Future/blob/e578e959819142021f12439414a30d7aad193e78/include/dlaf/solver/triangular/impl.h#L39-L42. If this looks like a decent direction I'll start tidying up loose ends (for this iteration...).

I like how the code looks, I think it is quite clear and easy to read.
For me it's still a long way to fully understand all these things in detail, so maybe I'm not very helpful on commenting the internal mecanisms, but from the user-perspective they're really nice.

Thanks for the effort!

[msimberg]

bors try

[bors]

try

Build failed:

• ci/gitlab/full-pipeline

[msimberg]

bors try

[msimberg]

bors try

[bors]

try

Already running a review

[bors]

try

Build failed:

• ci/gitlab/full-pipeline

[msimberg]

bors try

[bors]

try

Merge conflict.

[msimberg]

bors try

[bors]

try

Build failed:

• ci/gitlab/full-pipeline

[msimberg]

bors try

[msimberg]

bors try

[bors]

try

Already running a review

[msimberg]

bors try

[msimberg]

I've added (back in many cases) the priorities as explicit arguments to helper functions.

To summarize, the algorithms that have been senderified in this PR are:

• solver/triangular
• factorization/cholesky
• eigensolver/gen_to_std
• copy for Matrix

As a prerequisite all tile algorithms used in the above matrix algorithms have overloads that work with senders and policies.

[msimberg]

bors try

[bors]

try

Already running a review

[msimberg]

bors try-

[teonnik]

LGTM!

[msimberg]

bors try

[bors]

try

Build succeeded:

• ci/gitlab/full-pipeline

[rasolca]

@msimberg Are you going to try @albestro 's suggestion (https://godbolt.org/z/6TcGvr7e8 i.e. delete instead of unimplemented macro) or the PR is ready to merge?

[msimberg]

@msimberg Are you going to try @albestro 's suggestion (https://godbolt.org/z/6TcGvr7e8 i.e. delete instead of unimplemented macro) or the PR is ready to merge?

See my response here: #371 (comment). I tried it in a standalone setting and it does in principle the right thing. However, my preference is still the static_assert because it let's us put a pretty obvious error message there.