[Performance] AllReduce performance debugging

[chhwang]

This issue is suggested by @saeedmaleki for qualifying #83. Desired enhancements:

1. Improve reduce sum throughput (should achieve near HBM bandwidth)
2. Communication-computation overlapping
3. (For LL) changing the loop order, incremental flags

[chhwang]

3. is a TODO for Add packet copy (LL) for AllReduce #85

[chhwang]

@saeedmaleki For LL AllReduce, it seems we cannot avoid signalPacket() even if we use incremental flags. LL won't check these flags before writing, so the writer may overwrite unread data on the remote. If we let LL check flags before writing, it is nothing different from signalPacket().

[saeedmaleki]

@saeedmaleki For LL AllReduce, it seems we cannot avoid signalPacket() even if we use incremental flags. LL won't check these flags before writing, so the writer may overwrite unread data on the remote. If we let LL check flags before writing, it is nothing different from signalPacket().

I have a trick! We can use two different buffers and set of packets and alternating between them from one kernel to another. This way we are guaranteed not to overwrite the data.

[chhwang]

@saeedmaleki For LL AllReduce, it seems we cannot avoid signalPacket() even if we use incremental flags. LL won't check these flags before writing, so the writer may overwrite unread data on the remote. If we let LL check flags before writing, it is nothing different from signalPacket().

I have a trick! We can use two different buffers and set of packets and alternating between them from one kernel to another. This way we are guaranteed not to overwrite the data.

This is now implemented in #85. Big latency gain :)

[chhwang]

All issues tackled in #85.