NEP 29 - Dropping Python version support

[jacobtomlinson]

Next week numpy will be dropping Python 3.6 support as per the NEP 29 schedule.

Next week NEP 29 suggests projects start dropping support for Python 3.6.

I think Dask as a project should adopt NEP 29. What does the community think?

[gforsyth]

I'm strongly in favor of this.

[TomAugspurger]

+1

…

On Tue, Jun 16, 2020 at 11:24 AM Gil Forsyth ***@***.***> wrote: I'm strongly in favor of this. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIU7HWGO5GKFFLANUBLRW6MFRANCNFSM4N7YGQSQ> .

[Carreau]

Next week numpy will be dropping Python 3.6 support

Just to clarify this is not what the NEP says. It is and informational nep, and numpy/NEP is a good place to have community wide discussion and guideline. Per the NEP:

This is an unusual NEP in that it offers recommendations for community-wide policy and not for changes to NumPy itself

I actually doubt numpy will drop 3.6 in next version as it is a relatively low level dependency and they will likely be one of the last to actually remove support for 3.6 – though as a community adopting the NEP give a signal to numpy that they can drop 3.6.

Though I have been following nep29 with IPython so will drop 3.6 (at least in requirement), and I'm not going to actively make IPython incompatible with it, but I have no plan to make effort to keep it compatible.

[jakirkham]

My understanding from talking with them recently is they are dropping Python 3.5 (not 3.6) in NumPy 1.19 (due out soon). Not aware of Python 3.6 being dropped by NumPy yet. NumPy master still shows it as supported.

[rth]

I think this NEP could have been worded more neutrally. It is a proposed timeline but it's not actually being followed by core scientific python packages who co-signed it. Most dropped Python 3.5 less than a year ago, and use a slightly more conservative support window.

Also if Python 3.6 is dropped today, it would mean dropping PyPy support as well, since there is no Python 3.7 PyPy release yet. Not sure when one would happen.

[Carreau]

Most dropped Python 3.5 less than a year ago, and use a slightly more conservative support window.

Yes, roughly when the NEP was introduced and merged (in october).
So technically it's the first time that we go through a stop of support for a Python version because of this NEP.

I think this NEP could have been worded more neutrally. It is a proposed timeline but it's not actually being followed by core scientific python packages who co-signed it.

hum, well, Matplotlib, scikit-learn, IPython, Scipy only support 3.6+ (haven't checked all the rests) so I'm not sure how you can say it's not being followed. The nep say that starting with releases after next week you should consider dropping 3.6 so that leave plenty of time until next releases to stop support.

Also if Python 3.6 is dropped today, it would mean dropping PyPy support as well, since there is no Python 3.7 PyPy release yet. Not sure when one would happen.

Sure, but that does not mean people could not install old versions.

[rth]

Most dropped Python 3.5 less than a year ago, and use a slightly more conservative support window.

To be more precise numpy, scipy and jupyter notebook/lab haven't yet made a stable release (if you don't count RC) that dropped even Python 3.5 I think. More details in numpy/numpy#14086 (comment)

[MSeal]

To be more precise numpy, scipy and jupyter notebook/lab haven't yet made a stable release (if you don't count RC) that dropped even Python 3.5 I think.

Well a number of the backend libraries in Jupyter did drop python 3.5 support as they added async capabilites because it was too buggy without 3.6+ and 3.5 is nearing end-of-life. So those libraries aren't updating on jupyter notebook py3.5 deployments anymore.

[rth]

it would mean dropping PyPy support as well,

Sure, but that does not mean people could not install old versions.

Yes, but it would mean that it's not possible to run CI for PyPy on master to avoid regressions: some of the packages do that e.g. numpy, scikit-learn.

[Carreau]

Yes, but it would mean that it's not possible to run CI for PyPy on master to avoid regressions: some of the packages do that e.g. numpy, scikit-learn.

It's understandable that it's annoying, but isn't "master" of PyPy a "3.7" ? And I could see how we can workaround on CI to still install the libraries on PyPy master.

If PyPy master is not a "3.7" and libraries are not going to be compatible with 3.6, then why even test on master ? Unless I'm not following.

Regardless I'm not sure that's true, NEP 29 speaks of "support" and only suggest to setup python_requires, and that for "support"ed version one should build binary artifact. It does not define what "Not supporting" mean; that can be a gradation that goes from being actively incompatible to simply not publishing binary artifact.

Yes python_requires make it harder to install from source; it is critical when doing source release to not get random users that will start to build the sdist when pip install old python versions. But it does not make it impossible to install, nor does it tell you that the package should be incompatible.

And with this gradation there is a lot of flexibility from continuing to test on CI, with or without allow failure. or whatever author desire. – but that's IMHO a development workflow detail.

IMHO even if a project does build artifacts and set python_requires to a lower version but explicitly state in it's documentation that they are not supporting 3.6, it is in the spirit of NEP 29 and would be great.

[rth]

but isn't "master" of PyPy a "3.7" ?

I meant e.g. running numpy master with latest stable version of PyPy.

For the other suggestions I agree. A soft deprecation with delaying changing python_requires a bit would certainly make things easier.

[mrocklin]

Do we know if there is a schedule for PyPy 3.7? If it is likely to come out in a few months then I personally would be happy waiting to drop Python 3.6. I don't observe much pain around Python 3.6 today.

[jacobtomlinson]

@Carreau updated original post to more accurately reflect NEP 29

[tacaswell]

I am strongly in favor of dask dropping py36 for it's next release 😄 . Matplotlib is in the process of getting a 3.3 tagged and out the door (which will support py36 and is now about a month and a half behind our plans) but the next version (mpl34) will be py37+.

As @Carreau points out this is the first time we are really going through one of these thresholds sense getting the NEP29 finalized. NEP29 is meant to be a floor for the support window and to provide a framework for when to have these discussions.

[jakirkham]

Do we know if there is a schedule for PyPy 3.7? If it is likely to come out in a few months then I personally would be happy waiting to drop Python 3.6. I don't observe much pain around Python 3.6 today.

Maybe @mattip would know? 🙂

[Carreau]

I don't observe much pain around Python 3.6 today.

I don't either, except CI Time, especially on windows, and I believe that reactivity of CI is one of the key things for contributor experience. I don't know how many contributor use the "Edit-Push-CI-Loop", but I'm one of them on some projects with complex testing and shaving even 5 minutes sometime would be great 😛

[tacaswell]

I don't observe much pain around Python 3.6 today.

Part of the point of NEP29 is to take the subjective nature out of the decision. If we drop version of Python "when there is pain" it decoheres across the eco system, everyone says "well I won't drop until XYZ drops", and it becomes a giant wibbly-wobbly mess of prolonged discussions.

By having a calendar based scheme the goal was to both give users more certainty and the ability to plan on one hand and on the other avoid having long opinion based discussions of when we should drop a given version of Python on every project every year.

[mrocklin]

I'm totally fine dropping Python 3.6, but I do want to make sure that the PyData community is sensitive to PyPy.

Years ago a lot of us said "those PyPy people don't get SciPy/PyData at all. They kind of went off on their own and didnt really make any effort to engage." (Or at least this is what I thought at the time.) I think that today we should be mindful of PyPy and see if there is a way to accomodate their schedule so that there is a smooth transition.

I briefly looked through the PyPy page for a way I could ask this question, but it appears that I would need to sign up for a new thing in order to do this, which I chose not to do. I suggest that we wait a bit and see if @mattip responds here to get his sense of things.

[tacaswell]

I do want to make sure that the PyData community is sensitive to PyPy.

I agree, if a pypy release that supports py37 is on the horizon I think holding off on a py37+ release of our projects until after at least an pypy RC is out is both reasonable an with in the spirit of NEP29.

[jakirkham]

IIUC PyPy migrated from Bitbucket to Heptapod recently. Guessing due to Bitbucket sunsetting Mercurial support. After some digging I found this. It may be out-of-date (as noted on the Wiki homepage). AFAICT no timeline is listed. Maybe this is useful?

[mattip]

There is no timeline for a PyPy3.7 release, but NEP 29 provides motivation to move forward with at least an alpha release soonish. I will discuss it with the PyPy team. The py3.7 wiki page mentioned above is actually up-to-date. Most of the missing pieces are corner cases but we will not know which projects hit the corners until we try. Which Dask project uses PyPy in its CI?

[mrocklin]

We don't currently test against PyPy in CI. Every once in a while a PyPy user comes by and informs us that we broke something. Usually this is pretty easy to fix.

…

On Wed, Jun 17, 2020 at 5:10 AM Matti Picus ***@***.***> wrote: There is no timeline for a PyPy3.7 release, but NEP 29 provides motivation to move forward with at least an alpha release soonish. I will discuss it with the PyPy team. The py3.7 wiki page <https://foss.heptapod.net/pypy/pypy/wikis/py3.7%20status> mentioned above is actually up-to-date. Most of the missing pieces are corner cases but we will not know which projects hit the corners until we try. Which Dask project uses PyPy in its CI? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTAAN3XN55R32MUJTJLRXCXC7ANCNFSM4N7YGQSQ> .

[jakirkham]

Ah ok. Thanks for clarifying Matti!

Yeah I think this comes up periodically with Distributed as some people run it with PyPy.

[jakirkham]

@mattip, we had a similar conversation today about Python 3.6 support in conda-forge's meeting today. Brought up PyPy in that context so that core is aware. Am curious where things are today? 🙂

[mattip]

Thanks for taking PyPy into consideration. TL;DR: I hope to release a 3.7 PyPy this month.

I think we are pretty close. The only issue with the 3.7 label is one about PyThread_tss which is a new 3.7 C-API feature. There are no big blockers on the buildbot failure status page, I think all of these are niche corner cases or changes in error messages.

We had an issue with our macOS tester: the buildbot worker was offline for all of July, but now it is back (as of yesterday) and I will start a 3.7 build soon.

[quasiben]

@mattip thank you for pushing on pypy3.7. I see there was an alpha 3.7 release last month. Do you have a sense of when 3.7 will be fully released ?

[mattip]

That was not actually released. I am now working on the official 3.7 alpha. We will wait for feedback, and then probably do another release at the end of the year or beginning of 2021 (unless there is a catastrophic failure or astonishing success).

[quasiben]

That was not actually released.
Ah, good to know!

Thanks for the response @mattip

[TomAugspurger]

Just for reference, pandas master is 3.7+ now: pandas-dev/pandas#35214. Pandas 1.2 (due sometime in November) will support 3.7, 3.8, and 3.9.

[mrocklin]

I recommend that we drop Python 3.6 support starting in the next Dask version, 2020.11

[jakirkham]

Yeah if our dependencies are dropping Python 3.6, it makes it harder to support.

[jakirkham]

Should add RAPIDS 0.15 dropped Python 3.6.