pip 21.0 release (Q1 2021)

[pradyunsg]

Oh yea, it's happened. We have two separate release issues open at the same time.

This one is a big one, coz we're dropping Python 2 support. :)

[pradyunsg]

I'm becoming a little wary that I'm becoming the SPOF for our release processes and, I'd like to avoid that. :)

@pypa/pip-committers I'd like for someone else to be the RM for this release. I'm happy to have a voice/video chat if that'd be helpful for whomsoever becomes the RM, but I think everything is pretty much documented already.

(for the uninitiated: RM is short for release manager, who would be the person in charge of making the call of what goes into the release closer to the release date and does the actual magic of generating the release artifacts for PyPI)

[pfmoore]

I could do it, but it's extremely likely that I'd have to do the release over a weekend and then be (mostly) unavailable until the following weekend (because work doesn't provide me with any open source time, and I expect things to be busy Q1 next year). Specifically, I would not be able to respond to the sort of post-release fallout we've had this time, and I worry that the removal of Python 2 support, and the probable deprecation of the search command, might generate similar levels of issues.

I'm very happy for someone else to take it if they want, though.

[sbidoul]

I'm a bit in the same situation as Paul. I could do it but, due to limited availability, I'd prefer to handle a "simpler" release than 21.0, if such as thing ever exists :)

[pradyunsg]

@uranusjr would you be available to manage this release?

[uranusjr]

Chinese New Year is 12th February, and weeks before that are usually crunch time. So I’ll likely not be able to react in timely fashion during the time. It may work for me if the release can happen on 11th February or later.

[pradyunsg]

Ohkay... It seems like I'm in the best position (availability wise) to be wearing the hat again for 21.0. If no one is opposed to that, I'll assign myself to this issue before the end of the year.

That said, I do think that someone else will need to be the RM for 21.1, because I'm pretty certain that I won't be as available as I am right now, when that release is due.

[pfmoore]

We may just have to accept that we do a release with the resources we have and deal with the consequences - @pradyunsg like you say, having you as the SPOF isn't either sustainable nor fair on you.

If you want me to, I'll take 21.0. We just need to be aware that you've been unusually available post-release these last few releases, and we don't guarantee that going forward.

BTW, what's the expectation on when we do 21.0? Normally it would be January, do we want to aim for that given that 20.3 was pretty late? I'm inclined to say yes, we should get back on track. I'd also say that 21.0 should be back to "release from master on the planned date" and not try to wait for anything specific (like completion of the removal of Python 2 support).

[pradyunsg]

Works for me. :)

I'll be fairly available in Jan FWIW, and I do agree that a release in early Jan works best.

---

I think we'd probably want to defer removing the legacy resolver in 21.0 though (to 21.1 or 21.2) given that there's still a bunch of bugs open for it, and we've just not had enough time for folks to adjust perhaps. That's a #6536 discussion but whoever wears the RM hat needs to decide on that so that we can do a round onlf communicating about it. :)

I think that communication is gonna be the main thing this release, especially coz we won't have any funded folks to actively look into communicating about it.

Personally, I'm not too concerned about Python 2. And, well, pip search can stay for a while. :P

[pradyunsg]

FWIW, in my opinion, we'd dropped support for Python 2 when we updated the python_requires and removed it from the CI. That's all that really matters from a user's PoV.

All the modernisation and trimming (shoutout to @hugovk and @jdufresne on that front) is entirely a maintainer-only thing, and I don't expect there to be any user-facing implications of those not being "done", unless we make significant improvements (or make a mistake).

[pradyunsg]

Oh, and heads up on a fun thing that needs to be handled/deferred: we should update pip's vendoring configuration, coz we're going to be able to vendor newer versions of setuptools, which would put additional files in the vendoring directory that the current configuration won't delete.

[uranusjr]

This might sound crazy, but what if we skip a quarter. Remembering 20.3 was already delayed a month for various reasons, 20.3 and 21.0 would still only be ~120 days apart if we target the first half of April.

[pradyunsg]

To be frank, I just want to have a release that drops Python 2 in Jan 2021, because we've explicitly documented and communicated that, and because I want to be able to close bug reports with "no Python 2, thanks". :)

I would be fine if we upload a 21.0 that is literally 20.3.x, but with updated python_requires and classifiers. Actually, I would be on board if the RM (which seems to be @pfmoore) decided to do it that way. :P

[pfmoore]

OK. I'm thinking that I will do the release on the weekend of 23/34 January. That's not set in stone yet, I may move it a week either way depending on my availability, but it'll do for now as a plan 🙂

I will be releasing whatever's on master at the time of release - we have a "Release 21.0" milestone, but I intend to treat this as a reminder list for the authors of the relevant PRs, not as an action list for the RM, so I'll ping them closer to the release to check if they are expecting to be ready, and if not, I'll simply bump the items to 21.1.

I'd like 21.1 to be a relatively "quiet" release - we'll be dropping Python 2 support which may catch some people out, but otherwise I'd like to focus on any remaining tidy-up from the new resolver release, and apart from that stick to "business as usual" changes. Our users have had a fair bit of change to deal with recently, and I think that a quiet start to 2021 would be welcomed by all!

One thought - presumably we'll need a final Python 2 version of get-pip.py. Has anyone done that as part of the PRs dropping Python 2 compatible code, or is that still to be done? I don't keep track of the get-pip repo, so I'm not aware of what's going on there. If not, does anyone want to pick this up or is it something I should add to the list of things for the release? I assume it would have to be based off 20.3.3, not master.

[pradyunsg]

We'd need to update get-pip.py's generator script.

https://github.com/pypa/get-pip/blob/master/tasks/generate.py#L121

I'd like 21.1 to be a relatively "quiet" release

I think you meant 21.0 here? ;)

[pfmoore]

< I think you meant 21.0 here?

Correct. I wish we'd decided to number releases as 1,2,3,4 rather than 0,1,2,3, but hey, too late now...

[pfmoore]

I've just added reminders to all issues/PRs marked against the 21.0 milestone. If anyone has anything else they'd like to get into 21.0, you have a bit under 2 weeks to get it merged to master...

[hugovk]

pip search deprecation? It looked like there was initial agreement to include a deprecation warning in 21.0 (#5216), but @nlhkabu asked to share some user research first.

According to https://status.python.org/incidents/grk0k7sz6zkp, the PyPI issue is still not resolved.

[pfmoore]

Vendoring - @pradyunsg am I right in thinking that re-vendoring is done by manually updating versions in the vendor.txt file, and then running nox -s vendor? There's no automatic process for updating the versions in vendor.txt?

The big question here is whether we update setuptools, as I think we pinned 44.0.0 because of Python 2 support. So we could now update it, but is there a risk in doing so? I'm inclined to do it in a separate PR and see what CI thinks. But if anyone who's more directly involved with removing Python 2 support wants to chip in, that would be fine too 🙂

The only other changes here are point releases in msgpack and requests (and its dependencies), which are recent and so I assume OK.

[pfmoore]

#9460 created for the non-controversial vendoring changes.

[uranusjr]

I browsed through the changelog quickly, and it seems the pkg_resources package has not changed much, pypa/setuptools#1563 is the only change relevant to pip.

Alternatively we could keep setuptools < 45 (there is a 44.1 IIUC) and work on replacing it entirely instead.

[pradyunsg]

I'm OK to keep the older setuptools/pkg_resources for now. If we're upgrading, the thing to take care of is that we'd need to account for changes in setuptools' structure/installed-artifacts in >= 50.0, since it's got a new _distutils_hack package and .pth file and stuff.

[pfmoore]

Should be a case of bumping up the version numbers now, I think.

Well, no, because the tests fail on 3.9 due to the cryptography pin. I'm not keen on messing with the test suite in the week before a release, TBH, so let's park this until after the release is done.

[pradyunsg]

Do I also need to create a new template script?

Your call TBH. We only need to do that if we move the main function. We haven't, so there's no need to.

OTOH, creating a new template now would mean that if we do move main() in a future refactor, we don't need to do the split then.

[pfmoore]

Apparently there's also something needed in "salt states" to add a Python 2.7 directory to bootstrap.pypa.io. I've no clue about this 🙁 @ewdurbin from what I can see you did the last one of these (for Python 2.6) - would you be able to do it again?

[ewdurbin]

I believe the necessary directory needs to be added to get-pip then the necessary symlinks can be created in the psf-salt repo to point to them.

[pfmoore]

OK, I'll do the get-pip change in the next day or two, but I don't know what's needed in the salt repo, so if I ping you will you be able to do that bit? (I can create a PR that blindly creates a 2.7 copy of the bit you highlighted, but I'm a bit scared of breaking something as I know nothing about salt...)

[ewdurbin]

Either way is fine @pfmoore! If you'd prefer I make the change, just mention me here or on the PR to get-pip requesting it.

[pfmoore]

@ewdurbin Sorry! I never got the chance to make the necessary PR for the salt repo. Is this something you could do? I'm in the process of doing the new pip release now.

[ewdurbin]

@pfmoore it's done and is live.

[pfmoore]

@ewdurbin Awesome! Thanks for doing that so quickly 🙂

[pfmoore]

Pip 21.0 has now been released.

One issue, around the classifiers (a previous commit had left them incorrect). We should probably have a lint check that the classifiers are valid, to avoid this sort of issue in the future.

[pfmoore]

Note for myself. I still need to create a PR for CPython to include 21.0 with ensurepip. I'll hold off on that until next weekend to see what happens with #9506 (we may do a 21.0.1).

[groodt]

Thank you @pfmoore and all the volunteers! I appreciate your work.

[pfmoore]

Looks like packaging may have a release ready for this weekend, which will address #9506. Assuming they do, I'll release 21.0.1 this weekend from master. Anything else that needs to go into 21.0.1 should be in master by then.

I don't intend to release a 21.0.2 (barring unexpected disasters) so anything that doesn't make 21.0.1 will need to wait for 21.1 (in April).

[pfmoore]

OK, release 21.0.1 has been published!

Thanks @henryiii for assistance with the vendoring, that made my life (and weekend!) a lot easier 🙂

[henryiii]

Fantastic! Thank you so, so much for your support! We should have cibuildwheel ready to produce Universal2 wheels very soon now!

[brainwane]

If we are not removing the old resolver in any 21.0.x release, we should update the deprecation timeline at https://pip.pypa.io/en/latest/user_guide/#deprecation-timeline .

[pfmoore]

The 21.0 release cycle is complete now, there's no expectation of a 21.0.2, so yes, removal of the old resolver will happen in a future release. I suspect we should avoid being specific about when, simply because it depends on some volunteer doing the work, so we can't say when it will actually happen.

Maybe we should update the deprecation to note that as of 21.0 the old resolver is no longer supported, and will be removed without further warning in some future release (when resources permit)? That probably reflects the reality better than trying to predict when it will happen.

As an immediate measure, someone could submit a PR to remove the --use-deprecated=legacy-resolver flag. That would be a smaller change than full removal (not trivial, because it would also need to remove any tests that still check the old resolver) and has the same effect as far as end users are concerned. If submitted (and merged) now, it would be released with 21.1.

[MrMino]

@brainwane @pfmoore see #9631

[pradyunsg]

Funny how I never closed this. :)