Release 2024.4.1

[jrbourbeau]

Release version	2024.4.1
Planned release date	2024-04-04
Status	On-track
Release manager	@jrbourbeau

Best effort

Try to close before the release but will not block the release

• ...
• ...

Blocker

Issues that would cause us to block and postpone the release if not fixed

• ...
• ...

Comments

The 2024.4.0 release, import dask.dataframe raises an error on Python 3.11.9 (other Python versions are okay) dask/dask#11035. Things have been fixed on main and I'd like to push out a new release later today with the fix.

cc @fjetter @phofl @rjzamora @quasiben @hendrikmakait @crusaderky

[rjzamora]

Just a note that dask/dask#11035 may not have correctly resolved the problem. We are now seeing <function PandasDelegate._add_delegate_accessors.<locals>._create_delegator_method.<locals>.f at 0x7f0ecacbf560> - Will try to follow up soon.

[jrbourbeau]

Thanks for the heads up @rjzamora. I'll hold off on releasing until we're sure things have been resolved.

[rjzamora]

Okay - It looks like the problem I reported above was likely due to a corrupted environment. Someone internal reported that they were hitting this new TypeError. However, they are no longer able to reproduce the problem after a fresh build. We could always remove the new TypeError to be safe, but I'm not sure it's necessary.

[crusaderky]

Best effort:

• Restart workers when worker-ttl expires distributed#8538
• Don't deep-copy read-only buffers on unpickle distributed#8609

[phofl]

We got the same report from someone else dask/dask#11038

cc @rjzamora so I think the failure was legit?

[rjzamora]

@phofl - Yes, the DatetimeAccessor error was very real. We were also seeing a PandasDelegate error after 11035 was merged, but it seems to have been a local environment problem.

[jrbourbeau]

Okay, so sounds like dask/dask#11035 is needed and there's not a known related follow up needed either. I'll wait a bit for further feedback and then will start pushing this release out

[rjzamora]

@jrbourbeau - I submitted dask/dask#11039 to make sure we don't run into the hypothetical problem I mentioned. Feel free to ignore, but might be worth merging first.

[rjzamora]

On a related note: I wonder how many users/organizations are temporarily stuck on 2024.1.1 until they have had time to migrate to the new dask-expr API?

For the impending rapids 24.04 release, we are pouring many engineering hours into finding a way to patch/vendor our own version of dask-2024.1.1 with the python 3.10 fix. We originally considered this to be our own problem to deal with. However, I now strongly suspect that many other organizations would benefit from a patched 2024.1.1 release. Any thoughts on this?

[jrbourbeau]

FYI 2024.4.1 is out on PyPI (conda-forge + docker soon)

[jrbourbeau]

FYI it appears there were some CDN issues with conda-forge today and dask-core=2024.4.1 still isn't available to download (cc @jakirkham in case I'm misunderstanding). I'll try bumping the remaining feedstocks tomorrow

[jrbourbeau]

Closing as 2024.4.1 is out on conda-forge and docker registries now

On a related note: I wonder how many users/organizations are temporarily stuck on 2024.1.1 until they have had time to migrate to the new dask-expr API?

@phofl probably has a better sense than I do for this. I've mostly seen folks turning query planning off who run into issues for one reason or another (though issues have been quickly resolved in dask-expr too)

Maybe @quasiben @charlesbluca can do something similar to #316 to help. Regardless, this seems like a separate issue, so I'm going to close this issue out (hope that's okay)