Release 2021.03.0

[jrbourbeau]

It's been a few weeks since the last release. I'd like to release Dask and Distributed version 2021.02.1 this Friday if others are okay with it

cc @jakirkham @kkraus14 @jsignell

[kkraus14]

We're seeing a flaky incorrect result from dask.dataframe.describe in dask-cudf unit tests that seems to be a cupy bug, but we're still triaging. Hopefully should have more clarity in the next couple of days, but as of now no objections.

[jrbourbeau]

Thanks for the heads up @kkraus14 👍

[jrbourbeau]

Flagging a couple of other issues:

• Unstable dask causes unexpected compute results from xarray dask#7263
• Integer overflow in TCP comm distributed#4538

[jsignell]

It'd be good to fix or revert dataframe __setitem__: dask/dask#7261

[jakirkham]

We're seeing a flaky incorrect result from dask.dataframe.describe in dask-cudf unit tests that seems to be a cupy bug, but we're still triaging. Hopefully should have more clarity in the next couple of days, but as of now no objections.

For context there seem to be two things. A CuPy bug ( cupy/cupy#4451 ), which will be fixed in CuPy 9 and we are working to backport to CuPy 8 ( cupy/cupy#4742 ). A Dask-cuDF bug ( rapidsai/cudf#7402 ), which is still being investigated.

[jrbourbeau]

Thanks for the update @jakirkham -- it sounds like there's nothing for us to do about these issues on the dask / distributed side of things.

There are a few issues, specifically dask/dask#7263, dask/distributed#4538, and dask/dask#7261, that might cause friction with releasing tomorrow. Perhaps this is a good time to push releasing back a week to give time to resolve these lingering issues. Do others have any objections to releasing next Friday instead of tomorrow?

[jakirkham]

Correct

Followed up with Florian in issue ( dask/distributed#4538 ), but it sounds like he can’t reproduce the error. Given this, not sure what else there is to do there

[jrbourbeau]

Followed up with Florian in issue ( dask/distributed#4538 ), but it sounds like he can’t reproduce the error

Awesome, thanks for staying on top of that issue! Yeah, I agree if we can't reproduce the issue then there's not much we can do about it.

For dask/dask#7261 we could, as @jsignell suggested, revert the setitem changes in dask/dask#7033. However I get the sense we still need to form consensus on dask/dask#7263

[jakirkham]

Thanks for keeping track of all of things James! 😄

Got it. So issue ( dask/dask#7263 ) is the tricky one. Read through it and agree we should fix that before we release, but don't have any concrete suggestions. Pushing out the release date to next week makes sense.

[jsignell]

It'd be good to get this one in as well dask/dask#7260

[jakirkham]

How are things looking? Curious as we were planning on releasing this coming Friday

[jrbourbeau]

Here's the current status:

• There's a proposed solution in Unstable dask causes unexpected compute results from xarray dask#7263 which no one has objected to, so I think @pentschev is up for implementing the fix
• I follow-up on Integer overflow in TCP comm distributed#4538 to see if we're able to reproduce the TCP integer overflow. I'm inclined to release if we're not able to reproduce.
• As far as I can tell there's not been progress on Reintroduce __setitem__ which more closely matches NumPy dask#7261, but that's okay as we can revert Extend __setitem__ to more closely match numpy dask#7033 before releasing if needed

[jakirkham]

Thank you for tracking these James! 😄

[jakirkham]

We just stumbled across an issue on CI in RAPIDS that tracks back to a recent Dask change. Please see this comment for details ( dask/dask#7305 (comment) ). Thanks @rjzamora for identifying this!

[jakirkham]

The issue Rick identified is being addressed by Peter in PR ( dask/dask#7325 )

[jakirkham]

Ok I think we have all the release blocker fixes in. Though please recheck.

Just went back through James' list and all the patches have been merged. We also fixed the issue Rick identified.

We have not been able to reproduce issue ( dask/distributed#4538 ). So that has been ignored for now.

[jrbourbeau]

Yep, all the blocker fixes are in. Thanks everyone who's been working on them!

There's one last loose end to tie up before releasing -- we need to bump the minimum version of distributed over in dask. @jakirkham would you mind taking a quick look at dask/dask#7328 which does this

[jakirkham]

Good point. Thanks for remembering that. Reviewed 😀

[jrbourbeau]

Thanks all! 2021.03.0 is out