Test broadcast merge in local_cudf_merge benchmark

[rjzamora]

Includes the necessary changes to test dask#7143. More specifically, this adds the following options:

• --base-chunks : Number of base-DataFrame partitions (default: n_workers)
• --other-chunks : Number of other-DataFrame partitions (default: n_workers)
• --broadcast-join : Use broadcast join when possible
• --shuffle-join : Use shuffle join (takes precedence over '--broadcast-join')

[quasiben]

run tests

[rjzamora]

Now that dask#7143 was merged, it is now safe to consider this PR for integration.

[pentschev]

Retargeting seems to have no conflicts, let's see what the tests say. 🙂

[jakirkham]

Could you please run black locally Rick and push the changes? Currently failing the style job. Idk if the others will run without that

[pentschev]

Idk if the others will run without that

They still run, so we can wait another hour or so to see the results, but feel free to restyle it anyway in the meantime Rick.

[jakirkham]

Ah ok. If they still run, great let's wait

Know that previously changelog (ofc no longer included) and style were required to pass before running tests. Given this, couldn't tell if the jobs here were actually queued to run or maybe we were running into some edge case in gpuCI where they only appear queued (but don't actually run).

[codecov-io]

Codecov Report

Merging #507 (77346b7) into branch-0.19 (f99a037) will increase coverage by 1.39%.
The diff coverage is 0.00%.

@@               Coverage Diff               @@
##           branch-0.19     #507      +/-   ##
===============================================
+ Coverage        62.37%   63.77%   +1.39%     
===============================================
  Files               22       22              
  Lines             2517     2523       +6     
===============================================
+ Hits              1570     1609      +39     
+ Misses             947      914      -33     

Impacted Files	Coverage Δ
dask_cuda/benchmarks/local_cudf_merge.py	0.00% <0.00%> (ø)
dask_cuda/explicit_comms/dataframe/shuffle.py	97.94% <0.00%> (+0.68%)	⬆️
dask_cuda/_version.py	44.80% <0.00%> (+0.71%)	⬆️
dask_cuda/utils.py	90.86% <0.00%> (+1.02%)	⬆️
dask_cuda/proxy_object.py	91.36% <0.00%> (+1.11%)	⬆️
dask_cuda/cli/dask_cuda_worker.py	96.96% <0.00%> (+1.51%)	⬆️
dask_cuda/device_host_file.py	90.90% <0.00%> (+2.27%)	⬆️
dask_cuda/initialize.py	92.59% <0.00%> (+3.70%)	⬆️
dask_cuda/proxify_device_objects.py	90.16% <0.00%> (+9.83%)	⬆️
dask_cuda/get_device_memory_objects.py	89.04% <0.00%> (+19.17%)	⬆️
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f99a037...b80c70a. Read the comment docs.

[pentschev]

I left a minor nitpick, but other than that LGTM. Thanks @rjzamora !

[pentschev]

Thanks @rjzamora !

[jakirkham]

Looks like we are seeing some failures in test_dataframe_merge (subprocess is exiting with a non-zero exit code)

[rjzamora]

Looks like we are seeing some failures in test_dataframe_merge (subprocess is exiting with a non-zero exit code)

Hmm - It doesn't seem like that test is running the benchmark modified in this PR, is it?

[jakirkham]

Was this from a change in Dask or Distributed then? Both of those are planning to be released tomorrow. So if they have issues, we should try to identify those quickly

[jakirkham]

Just to update this thread, Rick did find a recent Dask change that is causing failures here as mentioned in this comment ( dask/dask#7305 (comment) ) in the PR with the change. We have surfaced this in the Dask release issue ( dask/community#129 (comment) ). Trying to figure out now how best to address this before the Dask + Distributed release tomorrow

[pentschev]

I'm attempting to fix the issue mentioned above in dask/dask#7325 .

[jakirkham]

rerun tests

Edit: As Peter's fix has landed. Thanks Peter! 😄

[jakirkham]

@gpucibot merge