refactor pip's use of "#egg="

[qwcode]

I suspect pip doesn't really need the"egg=" value (assuming we make some modifications)

setuptools used it in two cases:

• installing single py files (which pip doesn't support I guess?)
• installing from svn urls

pip's support is like so:

1. officially documented for vcs urls (it determines the src or build dir, depending on whether it's editable, and also determine's it's identification in the dependency resolution process.)
2. although not documented, I've seen users (including myself) tack "egg=" onto archive urls (that currently don't get their sdist filenames parsed, as you would expect (see The --build-dir options doesn't work for local archive installs #804), so using "egg=" for them makes them have a set build dir)

3 changes are needed, to make pip not require "egg=":

1. do away with the --no-install/--no-download "workflow" (see refactor --no-install/--no-download/--download (and use mkdtemp build dirs) #906), which assumes fixed build dirs, and then drop fixed build dirs all together.
2. make it a tested feature that non-specifier requirements (i.e. vcs urls, archive urls and paths) get processed first (i.e. unpacked and have their name/version determined), so they can properly participate in the dependency resolution process. this is currently true but not very prominent in the code and not tested afaict.
3. when doing editable intalls from vcs urls, use some other approach to determine the "src" directory. Maybe just unpack it to /tmp, then determine the name, then move it back into src.

even if it turns out #egg=is not used in any way by pip, it may be important to document a convention as visual aid (possibly as #pkg=, see #1265), when the vcs url would otherwise contain no clear indication as to what it is. I know for example that various CM modules for chef and puppet make use of this indicator

[Ivoz]

@qwcode could you link to an example puppet / chef module that makes use of that currently?

[qwcode]

• this PR (Extract egg/package name from URLs. poise/python#85), but it's unmerged
• puppet's core pip provider: https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pip.rb#L77
• and my company has an internal puppet module (that might go open source) that relies on it.

[qwcode]

cc @coderanger

[coderanger]

To be fair, the reason I've not merged that PR is because I don't want to be parsing pip-specific URL syntax.

[qwcode]

the module we use at work uses a python helper module (that uses pip to handle the parsing), so no rewriting of pip in ruby.

but regardless, the issue is whether the vcs requirement form should require or at least allow an identifier tag like #egg= (or #dist=).

if you're only thinking about what pip needs to do an install, then you could argue it's not necessary (and worse actually misleads pip). you could just say that pip should always be cloning and using the real name and version from setup.py to determine how it gets processed relative to the other requirements.

but for a CM system that might want to support the vcs url form as a valid way to declare a resource (i.e. not just package specifiers 'pkg==version'), then having an identifier like #egg==name-version is pretty handy to quickly validate whether the resource is currently satisfied on the system.

[Ivoz]

So, I've realised that although pip might deprecate making use of that field, there's no-one saying we should disallow the fragment being in the url, right?

[qwcode]

but I'm thinking it's more than just allowing arbitrary fragments.
we should document the convention.

moreover, pip might start using the conventions itself for fast determination of fulfillment:

• if a '@' ref is present that's a literal version/commit, check that before updating the src clone.
• or, if #[egg|dist]=<name>-<version> is present, check that name and version before re-updating the clone.

[dstufft]

I think that this is still an issue, I'm not sure it makes sense to get rid of it, but at the very least we should rename it since we have deprecated eggs and the terminology is confusing to people.

[xavfernandez]

I think it should (very) slowly be deprecated in favor of the PEP-508 blessed pkg_name @ url.
(or am I missing something ?)

[dstufft]

Yea that's likely a reasonable outcome.

[pradyunsg]

So... the actionable part of this would be to deprecate #egg= in favour of the PEP 508 syntax?

Would this be a documentation only change?

[pradyunsg]

Ping?

[xavfernandez]

Maybe a warning to suggest the new PEP 508 syntax ? And a very long deprecation period...

[pradyunsg]

Cool. Let's do this then.

[pradyunsg]

This issue is a good starting point for anyone who wants to help out with pip's development -- it's simple and the process of fixing this should be a good introduction to pip's development workflow. See the discussion above to understand what the desired fix is.

Feel free to give me a mention "@pradyunsg" if you have any doubts or questions.

[gutsytechster]

So, the solution to it is to deprecate the use of #egg= format with a deprecation cycle? May I know how do we perform a deprecation cycle?

[uranusjr]

Merging in #9429, which is basically another reason why #egg= should be removed (internally) and discouraged.

Proposed solution from #9429 (comment):

• If a requirement’s does not contain an egg fragment, use it as it.
• If a requirement contains an egg fragment, and is not in PEP 508 format, convert it into PEP 508 with a warning.
• If a PEP 508 direct URL requirement’s URL part contains an egg fragment containing an identical package name, from the egg fragment with a warning.
• If a PEP 508 direct URL requirement’s URL part contains an egg fragment containing a different package name, fail with an error (requirement invalid).

[sbidoul]

I've given some more thought to this, and I think the first thing to do is to support -e "name @ vcs url". Indeed, currently the #egg= fragment is mandatory for -e vcs-url, because pip needs a name to create the checkout directory under src. When that is done we should have alternatives to egg fragments for all use cases.

[uranusjr]

I opened #9471 for -e "NAME @ URL" support.

[piotr-dobrogost]

If a PEP 508 direct URL requirement’s URL part contains an egg fragment containing a different package name, fail with an error (requirement invalid).

Wouldn't ignoring fragment with a warning be more sensible here?
Just treat PEP 508 direct URL as overriding old approach with an egg fragment and keep going.

[uranusjr]

Cross-linking #10002 (comment) for my thoughts about warnings (tl;dr I don’t think we should do this for now).

[DiddiLeija]

Hello. It seems like this issue has been fragmented into many issues that were already resolved. Can we close this one?

[pradyunsg]

This is still definitely relevant. I've poked at pypa/packaging#264 again, to get the ball rolling again for #9471.

[woodruffw]

Cross-referencing: there's a discussion here on removing egg= fragments entirely: https://discuss.python.org/t/killing-off-the-egg-fragment-once-and-for-all/21660

[lrq3000]

It was suggested in the forum's discussion to remove the egg= fragment entirely on the premise that the @ syntax has equivalent functionalities, but this is not the case. I would like to suggest to not deprecate egg= fragments entirely until the @ syntax is really equivalent, which it currently is not because it lacks support for PEP 508 URLs as editables. In particular, this means that if the egg= fragment is removed in the current state, it will be impossible to specify extras on an editable PEP508 URL, eg, the following is currently impossible:

pip install --upgrade --editable "mypkg[dist] @ git+https://github.com/username/mypkg" --verbose

The only way it can work currently is as follows:

pip install --upgrade --editable "git+https://github.com/username/mypkg#egg=mypkg[dist]" --verbose

To my knowledge, there is no other way than using an egg fragment to specify an extra, so if this gets removed before an equivalent implementation in @, this would be a net loss in functionality.

[uranusjr]

The PEP 508 syntax should provide equivalent functionalities, the fact that --editable does not support the syntax is an implementation issue that should be fixed before/when the egg fragment syntax is dropped, not a design issue.

[lrq3000]

Cross-referencing #11617 since that's the one that is linked to in the deprecation warning for the issue I mentioned above.

[lrq3000]

Ah, it seems that it's not possible to add cross-references on locked issues/PRs. Then I would suggest to change the deprecation warning:

Using pip 23.1.2 from /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pip (python 3.11)
DEPRECATION: myproject.git#egg=myproject[test] contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at `https://github.com/pypa/pip/issues/1161`

To:

Using pip 23.1.2 from /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pip (python 3.11)
DEPRECATION: myproject.git#egg=myproject[test] contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at `https://github.com/pypa/pip/issues/1161` and `https://github.com/pypa/pip/issues/1289`

Ie, adding a link to the current issue, since #1161 has no relevant information for the case of --editable + egg fragment.

[mr-c]

Hello from 2024, will installing an editable from a VCS URI really be removed in pip 25?
Many of the projects I maintain have a release testing script where we want to have an editable install of a VCS (git) checkout, including the specification of [extras].

https://github.com/common-workflow-language/cwl-utils/blob/67dbfa590c733b6ad5bf9139866e42cb5cd02aa2/release-test.sh#L64
https://github.com/common-workflow-language/schema_salad/blob/b9f2d76b4accfb633aab5b6bdb42b3fe93d5f39e/release-test.sh#L64
https://github.com/common-workflow-language/cwltool/blob/5e41bda52c8b24643aa4b4de86172365bd1c8ff7/release-test.sh#L69

Thanks

[pfmoore]

To my knowledge, there is no other way than using an egg fragment to specify an extra

Just to clarify, extras are allowed in (URL specifiers)[https://packaging.python.org/en/latest/specifications/dependency-specifiers/]:

url_req       = name wsp* extras? wsp* urlspec wsp+ quoted_marker?

will installing an editable from a VCS URI really be removed in pip 25?

As noted (above)[https://github.com//issues/1289#issuecomment-1510735040], support for the url_req format in editable installs is an implementation issue. We can't guarantee what will happen over the next year, but if we don't implement that support, we'll likely just update the deprecation to push it back a bit.

But at some point, you will need to switch from #egg to url_req. When that point is depends on resources at pip's end, though.