[REVIEW]Multi-trainers cugraph-DGL examples

[VibhuJawa]

This PR adds a working Multi-GPU Graph (on 2 dask workers) being trained/loaded on multiple pytorch trainers. (3)

Todo:

• Verify works on multiple trainers and multiple dask workers
• Show scaling as you increase training GPUs

At 1 second we become bottlenecked by sampling dask cluster, but we see perf improvement by going from 1 GPU->2GPU.
On OBGN-Products

| Number of Training GPUs | Time per epoch |
|-------------------------|----------------|
| 1                       | 2.3 s          |
| 2                       | 0.582 s        |
| 4                       | 0.792 s        |

This PR depends upon: #3393
CC: @rlratzel , @alexbarghi-nv , @BradReesWork

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (branch-23.04@1543356). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files

@@               Coverage Diff               @@
##             branch-23.04    #3212   +/-   ##
===============================================
  Coverage                ?   56.27%           
===============================================
  Files                   ?      153           
  Lines                   ?     9662           
  Branches                ?        0           
===============================================
  Hits                    ?     5437           
  Misses                  ?     4225           
  Partials                ?        0           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[alexbarghi-nv]

@VibhuJawa is this is still WIP? I thought you said this was ready for review.

[VibhuJawa]

@VibhuJawa is this is still WIP? I thought you said this was ready for review.

Sorry for the confusion. This is still WIP. Will let you know when it is ready for review

[VibhuJawa]

Please note that this example is failing because of a bug that i am trying to triage. That said, the bug is independent of the PR so we can probably review it.

[alexbarghi-nv]

Looks ok, just had a couple comments that don't need to hold up merging this PR.

[alexbarghi-nv]

/merge