Streamline communications

[ahoenselaar]

• Calculate the sequence of sends and receives once after connect_the_chunks
• Initiate asynchronous receives as early as possible
• Initiate asynchronous sends immediately after copying data into the communication buffers
• Prioritize chunks based on the data transfer size
• Introduce an RAII-based communications manager class that waits for requests to finish upon destruction
• Store nonzero communication sizes in a hash map instead of an array of size #chunks x #chunks

Performance increase by about 15% in smaller sims with light communications (small transfer sizes), e.g. a 2D sim with 12 processes on a single physical node.
Not as noticeable for larger 3D sims (e.g. 24 processes across 2 physical nodes), likely because the increased communication efficiency is overshadowed by the effects of uneven chunk balancing.

Future improvements:

• Start processing received messages immediately by using MPI_Testsome instead of MPI_Waitall. This should overlap network communications more with the copying of data into the field array.
• Combine the sequential boundary updates of {PH, H}, {PE, E} into a single [Set up Rx, copy data, trigger Tx, process Rx data] cycle
• Prioritize send operations by target process instead of source chunk. This requires more careful management of the subranges in the comms buffer array.

[ahoenselaar]

Are there any review comments?