Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement MMIO with put and signal #55

Merged
merged 12 commits into from
Apr 19, 2023
Merged

Implement MMIO with put and signal #55

merged 12 commits into from
Apr 19, 2023

Conversation

Binyang2014
Copy link
Contributor

This implementation using mmio to copy data from two devices. The process is:

GPU 1 ---->GPU 2 -----> GPU 3 ---> ... ---> GPU 8----> GPU 1

Test results for 1n-8p 1KB data, ring send-recv:

# minBytes 1024 maxBytes 1024 step: 1048576(bytes) warmup iters: 10 iters: 50 validation: 1 graph: 15, kernel num: 0
#
#
# Initializing MSCCL++
# Setting up the connection in MSCCL++
# Launching MSCCL++ proxy threads
#
#                                    in-place                       out-of-place          
#       size         count     time   algbw   busbw  #wrong     time   algbw   busbw  #wrong
#        (B)    (elements)     (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
        1024           256    37.81    0.22    0.19      0
Stopping MSCCL++ proxy threads
# Out of bounds values : 0 OK
#

Result for nccl using simple protocol for send-recv test

#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
        1024           256     float     sum      -1    20.44    0.05    0.05      0    25.15    0.04    0.04    N/A
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0.0454108 
#

Will optimize it in following PR

src/include/mscclpp.h Outdated Show resolved Hide resolved
src/init.cc Outdated Show resolved Hide resolved
src/include/mscclpp.h Show resolved Hide resolved
src/include/mscclpp.h Show resolved Hide resolved
src/include/mscclpp.h Outdated Show resolved Hide resolved
@chhwang chhwang requested a review from olsaarik April 12, 2023 03:35
@Binyang2014 Binyang2014 merged commit 3d767f3 into main Apr 19, 2023
@chhwang chhwang deleted the olli/smcopy branch April 23, 2023 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants