pip 10 Coming - Lots of API Changes

[cooperlees]

We should double check pip-compile etc. still works with the announced upcoming pip 10 release. A lot of APIs have been moved to internal. We should see what breaks with the pip-tool suiete.

Environment Versions

OS Type - Any
Python version: 3.6.3
pip version: HEAD
pip-tools version: HEAD

Steps to replicate

1. Install HEAD build of pip and see wht happens

Expected result

Some API moves and maybe change what APIs are called from pip-tools.

[estan]

Just as some extra piece of info, here's the link to the thread on distutils-sig@ about the coming PIP 10:

https://mail.python.org/pipermail/distutils-sig/2017-October/031642.html

[vphilippon]

Just a quick update: I did a quick test, installing the head of pip's master branch, and pip-compile does not work straight out of the box with pip 10 as of now (that was to be expected).
There will be work to do.

[pradyunsg]

I partially refactored the internal code for pip's resolution logic. That's still WIP so, there's that.

Let me know how I can help. :)

[vphilippon]

Let me know how I can help.

Send a PR to make pip-tools work with pip 8.0.1 to 10* 😄. (Yeah, I know, that's a big one)
Otherwise, we now have your name if we need a contact, that's a good start, thanks!

* That's an ideal case, but I'm aware we might have to drop some past version support. I'd rather not drop pip 9 support right away though, and with the major refactor, that might a big part of the issue. sigh

[pradyunsg]

Send a PR to make pip-tools work with pip 8.0.1 to 10

I am kinda short on free time right now so I think that's unlikely. :/

we now have your name if we need a contact

Sure! Gimme a mention and I'll respond when I can. :)

---

I noticed that you're reaching deep into pip's internals so, whoever does this is gonna have a lot of fun. :P

[vphilippon]

@pradyunsg Its allright, I just couldn't let the opportunity pass and not ask :).

Actually, if I'm not mistaken, you're working on bringning dependency resolution to pip, right? Is that something that's aimed for pip 10? That would change the game a lot here.

Also, I've been looking for a "pip 10 change resume" to figure out what changed and how things are working now. I you have some resources to point me to, I'd take it.

[pradyunsg]

I just couldn't let the opportunity pass and not ask :)

I imagine so. :P

Is that something that's aimed for pip 10?

In a word, no.

I wanted to but it turned out to be more work than I had anticipated and then I ran out of time (paid and then soon after free; thanks to college).

I'm gonna try a different approach -- implementing the resolver independently of pip and then fitting that into pip but that's for early next year; not pip 10.

_{Rant hidden}

There's a 30 page report that I have to submit in the morning tomorrow but I got know about it day before and everything that I did yesterday for it (after finishing 2 other assignments) was incorrect because I was given wrong instructions. So, now I have to do it all from scratch again and sleep in 10 hours. ugh.

I've been looking for a "pip 10 change resume" to figure out what changed and how things are working now.

I don't think there's one. pip doesn't document internal changes. There's PRs tagged kind - refactor by me over on pip's tracker. You'd probably want to look at them. (now that I look at them, they're a good summary)

---

In general, anything touching RequirementSet would have to change because RequirementSet is no longer the God Class that it was. It's still god-ly -- not as much has before though. My understanding is, that's all of pip-tools.

Basically, things still have to move out of RequirementSet, till it basically becomes a set of requirements with some helpers to query items in the set; all operations on them would move out into other places. The current codebase is in a limbo where some stuff has been moved decoupled but some hasn't. (For example there's stuff still on RequirementSet that should really be on Resolver but well, I didn't get to it because of a lack of time mostly.)

pip 10 will have this partially; even patch versions of pip 10 might change this situation (because it's pip._internal stuff) so, you're probably gonna have to deal with that too. :-(

[vphilippon]

Allright, so the best short-medium term idea I have right now would be to vendor pip 9.0.1 as a whole and use it for the whole dependency resolution part.

(Bonus point: I'm thinking about vendoring pretty much any dependency to stop having issues with pip-sync risking breaking itself.)

I highly doubt anyone will have the time to redesign all of this to work with pip 10, and I'm not even touching the subject of retro-compatibility. Also the risks of breaking stuff are..."fairly high" (cough it's going to be a firetrucking mess cough)

[davidovich]

I wholeheartedly agree with you @vphilippon. This is a great idea, considering that pip 10 will probably offer an alternative, and that the external interfaces to pypi will probably stay compatible with pip9, I think absorbing pip will let pip-tools continue to be relevant by itself for the phase out period. Le mer. 22 nov. 2017 à 18:18, Vincent Philippon <[email protected]> a écrit :

…

Allright, so the best short-medium term idea I have right now would be to vendor pip 9.0.1 as a whole and use it for the whole dependency resolution part. (Bonus point: I'm thinking about vendoring pretty much any dependency to stop having issues with pip-sync risking breaking itself.) I highly doubt anyone will have the time to redesign all of this to work with pip 10, and I'm not even touching the subject of retro-compatibility. Also the risks of breaking stuff are..."fairly high" (*cough* it's going to be a firetrucking mess *cough*) — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#580 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABX6ckXOkFv_8VlYBUSAGLMuRRY-eDxTks5s5KvggaJpZM4QA6-4> .

[merwok]

As a user, I would be happy if pip-tools had a strict dependency on pip 9.

[davidovich]

As a user, I would be happy if pip-tools had a strict dependency on pip 9.

As a typical setup, pip is only installed once in the system environment (aside from virtualenvs of course, but talking from a boostapping view). When the user will have upgraded to pip10, it will break pip-tools for that user. If we vendor pip and that user gets the new pip-tools version, it will isolate him from any other change.

Right now, pip-tools has many workarounds and special cases to support varying pip versions, vendoring will help eliminate those.

[pradyunsg]

Vendoring sounds good to me. It does save pip-tools from some of the hacks it does.

That does come with the issue that the new pip features -- including PEP 518 support, better PEP 508 support -- and (lots of)? bug fixes won't be available to users of pip-tools.

To be honest, I don't see a cleaner way forward on this, other than someone putting in a lot of hours, which is not gonna be me so I'm not gonna complain about it.

[vphilippon]

@davidovich

This is a great idea, considering that pip 10 will probably offer an alternative [...]

Just to prevent any bad surprise, as @pradyunsg mentionned, pip 10.0.0 will likely not provide dependency resolution. This adds even more importance to have pip-tools work even if the user has pip 10 installed (which you already agreed on, I'm simply pointing out the additional need for support, confirming that being blocked to pip 9 would be a pain for many.)

@pradyunsg

[Tons of nice pip 10 stuff] won't be available to users of pip-tools.

We can vendor pip 9.0.1 for the dependency resolution part, and still rely on pip 10 (or whatever pip the user has installed) for the pip-sync part (IIRC, nothing too funky happens in there, TBC), and let users benefit from most of pip 10.

For PEP 518, I'm not sure that would go affect dependency resolution, but I might be wrong.
For better PEP 508 support, that's a bummer, but it might be easier for us to actually patch pip 9.0.1 for that part (those are words that I may regret later).

[pradyunsg]

Okay -- just checked code...

PEP 518 would not affect dependency resolution and PEP 508 support doesn't affect dependencies, only top level requirements right now.

still rely on pip 10

Oh right. Cool. :)

---

It's just the bug fixes then, which, I guess, is cool.

[merwok]

As a typical setup, pip is only installed once in the system environment (aside from virtualenvs of course,

I’m never not using virtualenvs, so I didn’t consider that. Vendoring is always a defeat to me, but do what you must!

[taion]

If you're going to vendor pip, then there's no actual downside to calling into _internal in your vendored pip, since those imports won't change under you anyway.

Instead of vendoring pip==9.0.1, then, I think it might be better to vendor pip==10.0.0 (when it comes out), and just update the imports to pip._internal (as appropriate for the vendored version).

At a glance, it didn't look like the internal organization changed meaningfully, but I guess we can verify that empirically.

[vphilippon]

@taion The issue is that pip 10 also did breaking changes for us beyond the restructuration in pip._internal. I did some testing locally, using pip 10 and adding _internal in every import. That took care of the import issues, but the interface and role of the classes changed (namely RequirementSet, a core piece currently needed by pip-tools).

Of course, if we vendor pip, it will make it easier to eventually change pip-tools code in order to use pip 10 without having to worry with compatibility break with pip 8-9.
I would be happy if it could be done right away, but realistically, I highly doubt it will.
Once again, I would be happy to be proven wrong!

[taion]

Fair enough! I guess either way, it moves "upgrading to Pip 10" to something that can be handled via a PR, in case anybody needs the changes.