No pinned flags in setup_trans

[lukasm91]

I suggest that we remove pinned flags from setup_trans, mainly because there are no allocations in setup_trans anymore that are in the critical path.

The pinned flag used to be here for a few intermediate buffers that were copied forth and back in each iteration. Those are now gone.

If we are not removing pinned, we have to add it to a few more files to make sure we use the same API for deallocation. This is one of the reasons trans_release/trans_end is not working right now, and the error messages are far from intuitive.

[wdeconinck]

I definitely don't like the "-gpu=pinned" flag and would like to avoid it if possible.
Did you also verify with an experiment that this modification has no negative impact on performance within the iterations loop?

[marsdeno]

I tested this in the run-up to merge of GPU code path to develop, and my commit must somehow have got lost somewhere, sorry.
In any case in my tests things worked fine with no effect on performance.

[lukasm91]

Yes, I confirm, also didn't see an impact on performance, which makes sense because setup_trans has no allocations that are moved between CPU and GPU in the critical path (this was different in earlier versions because the CPU-GPU staging buffers where allocated in this routine).

[wdeconinck]

Does this now mean that this flag is also no longer required on the link line?

[lukasm91]

Does this now mean that this flag is also no longer required on the link line?

In the current code, yes, because the only transfers in the critical path that benefit from pinned memory are copying input/output buffers from outside ectrans, and those have to be allocated accordingly, but outside of ectrans. If we were to optimize the "no cuda aware MPI case" on GPU, we would need to compile and link that file with -gpu=pinned, but that is probably a stretch, and is also not a regression.