[BUG] Unable to Send dask_cudf.core.Scalar Object Between Workers

[alexbarghi-nv]

We found an issue while working on #3669 where the Scalar object returned from min() and max() had to be sent to the client first before other workers.

We tried using Client.replicate, from_delayed after calling client.submit with the min/max as arguments to create a new cudf DataFrame on each partition, creating a new DataFrame and expanding it, and a variety of other solutions that should have worked, but didn't. In all these cases, the min/max value was not sent to all workers, and in some cases, it was not sent to any workers at all, even after calling persist() and wait.

This is likely due to a bug in dask, or potentially something else we're not doing correctly. In any case, this needs further investigation since it could cause issues on multi-node workflows.

[alexbarghi-nv]

Not an appropriate use of dask, closing this issue.