Fix CUDA_VISIBLE_DEVICES tests

[pentschev]

After recent changes in Distributed, particularly dask/distributed#4866, worker processes will now attempt to get information from PyNVML based on the index specified in CUDA_VISIBLE_DEVICES. Some of our tests purposely test device numbers that may not exist in some systems (e.g., gpuCI where only single-GPU is supported) to ensure the CUDA_VISIBLE_DEVICES of each worker indeed respects the ordering of dask_cuda.utils.cuda_visible_devices. The changes here introduce a new MockWorker class that will monkey-patch the behavior of NVML usage of distributed.Worker, which can then be used to return those tests to a working state.

[pentschev]

Failures are unrelated: #637 (comment), probably still needs rapidsai/cudf#8426 to be in.

[galipremsagar]

rerun tests

[quasiben]

rerun tests

[quasiben]

I retargeted this PR to 21.08

[pentschev]

rerun tests

[pentschev]

I retargeted this PR to 21.08

Why? If we're pinning today's Dask release for 21.06, then we need this in.

[pentschev]

I forgot this requires dask/distributed#4873, CI won't pass until that's merged.

[jrbourbeau]

dask/distributed#4873 is in now

[pentschev]

Thanks @jrbourbeau , rerunning!

[pentschev]

rerun tests

[pentschev]

I'm still trying to reproduce this, I attempted doing so also running the Local CI scripts, but all tests pass then as well on a DGX-1. I think the issue may be due to gpuCI having only a single GPU, I'll try that on a machine with only one GPU.

[codecov-commenter]

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.08@0c4c59d). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 34f4caa differs from pull request most recent head 5a175f0. Consider uploading reports for the commit 5a175f0 to get more accurate results

@@               Coverage Diff               @@
##             branch-21.08     #638   +/-   ##
===============================================
  Coverage                ?   90.45%           
===============================================
  Files                   ?       15           
  Lines                   ?     1645           
  Branches                ?        0           
===============================================
  Hits                    ?     1488           
  Misses                  ?      157           
  Partials                ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0c4c59d...5a175f0. Read the comment docs.

[pentschev]

rerun tests

[quasiben]

@gpucibot merge

[pentschev]

Thanks @quasiben !