pip freeze messes up egg version with git branches, raising ValueError: 'Expected version spec in'

[rbu]

I have a requirements.txt containing this line:

-e git://github.com/mfogel/django-timezone-field#egg=django-timezone-field

When dumping pip freeze, it writes this:

-e git://github.com/mfogel/django-timezone-field@82a9c16eb7955e7d4f037ececf867d10fd5565ae#egg=django_timezone_field-origin/develop

This cannot be installed with pip, it raises:

Exception:
Traceback (most recent call last):
  File "lib/python2.7/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "lib/python2.7/site-packages/pip/commands/install.py", line 220, in run
    for req in parse_requirements(filename, finder=finder, options=options):
  File "lib/python2.7/site-packages/pip/req.py", line 1454, in parse_requirements
    line, comes_from=comes_from, default_vcs=options.default_vcs if options else None)
  File "lib/python2.7/site-packages/pip/req.py", line 85, in from_editable
    res = cls(name, comes_from, source_dir=source_dir, editable=True, url=url, prereleases=True)
  File "lib/python2.7/site-packages/pip/req.py", line 43, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "lib/python2.7/site-packages/pkg_resources.py", line 2914, in parse
    reqs = list(parse_requirements(s))
  File "lib/python2.7/site-packages/pkg_resources.py", line 2839, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "lib/python2.7/site-packages/pkg_resources.py", line 2807, in scan_list
    raise ValueError("Expected "+item_name+" in",line,"at",line[p:])
ValueError: ('Expected version spec in', 'django_timezone_field-origin/develop', 'at', '/develop')

$ pip --version
pip 1.4 from ...virtualenv/lib/python2.7/site-packages (python 2.7)
$ easy_install --version
setuptools 0.9.8

[rosscdh]

+1 am getting this issue as well.. exact same versions

[mrmachine]

Me too. Is it a bug or a new backwards incompatible feature? Pip used to dump these with a "-dev" suffix. Now it seems to use "remote/branch" (or sometimes, just "branch").

[ericbuehl]

+1

[lsemel]

+1

[typeshige]

+1

[vitawasalreadytaken]

+1, same versions. It could be fixed in setuptools by changing the regex DISTRO = re.compile(r"\s*((\w|[-.])+)").match to DISTRO = re.compile(r"\s*((\w|[-./])+)").match in pkg_resources.py to allow for slashes. But I guess it would be better if pip didn't put slashes in egg names in the first place.

A quick fix is of course to remove the slashes from your pip freeze output.

[cjerdonek]

I also hit this in pip 1.5.

[cjerdonek]

It looks like this is where pip adds the egg name:

req = '%s@%s#egg=%s' % (svn_location, rev, cls.egg_name(dist))

The egg_name is originally defined here in setuptools's pkg_resources module:

def egg_name(self):
    """Return what this distribution's standard .egg filename should be"""
    filename = "%s-%s-py%s" % (
        to_filename(self.project_name), to_filename(self.version),
        self.py_version or PY_MAJOR
    )

And here it looks like pip is already stripping off the last -py... part of that:

def egg_name(dist):
    name = dist.egg_name()
    match = re.search(r'-py\d\.\d$', name)
    if match:
        name = name[:match.start()]
    return name

So maybe the fix is as easy as having pip strip off a bit more from the end.

[mwaterfall]

I've just closed #1125 as it's a dupe of this issue.

[cjerdonek]

Is it a bug or a new backwards incompatible feature? Pip used to dump these with a "-dev" suffix. Now it seems to use "remote/branch" (or sometimes, just "branch").

It looks like this change may have been introduced back in May 2013 in commit fa81a41. For example, you can see one of the test expectations changing in that commit from -dev to -0.1.1. And see this line:

+            full_egg_name = '%s-%s' % (egg_project_name, names_by_commit[current_rev])

The commit message sounds like it was meant to be only a refactoring ("Combine Git get_tag_revs and get_branch_revs into single get_refs method."), so I wonder whether this change in functionality was indeed deliberate.

[carljm]

The change in functionality was deliberate in that commit (clearly, since a test was changed), although it's true that the commit message didn't reflect that as it should have. In the course of a refactoring I came across functionality that had been apparently intended by broken/dead code paths in the existing code, and restored it. However, I failed to check the case of a branch name including remote name, and apparently failed to notice that the previous dead code path had tried to handle that case with an removal of an "origin/" prefix (which I think is overly simplistic, since it assumes the only possible remote name is "origin").

I think the right solution here is, when constructing full_egg_name from egg_project_name and names_by_commit[current_rev], to split names_by_commit[current_rev] on slashes and use only the final portion.

[cjerdonek]

Thanks for replying so quickly, @carljm.

I think the right solution here is, when constructing full_egg_name from egg_project_name and names_by_commit[current_rev], to split names_by_commit[current_rev] on slashes and use only the final portion.

It seems somewhat common to have branch names with a slash in it. To name one example, Django has a series of branches of the form stable/1.6.x. Can you suggest a solution that accommodates cases like that?

[carljm]

@cjerdonek Short of a change to the regex in setuptools as suggested above (which may have other consequences I'm not aware of, and I'm not sure would be accepted by the setuptools maintainers), it's not possible to maintain such branch names unmodified. This egg version is not functionally important; using anything other than "dev" here is simply an attempt to give a better human-readable clue as to which version of the package is in use. In the case of Django you mention, the resulting egg name would be "Django-1.4.x" if the branch was "stable/1.4.x", which still seems reasonable and more useful than "Django-dev".

Another option would be to translate slashes to some other allowed character (underscore?); it's not clear to me that "Django-origin_stable_1.4.x" is really better than "Django-1.4.x", though. Perhaps there are other cases where it would be preferable, but my guess is that in general a) the remote name is not very useful there, and b) in most cases where branch names with slashes are in use, the final portion is probably adequate as a human-readable hint.

[cjerdonek]

Okay, good to know that the name isn't functionally important. For readability/usability, my instinct would be to lean towards stripping the remote name off the front and then convert slashes to underscores. So in the example, it would be ""Django-stable_1.4.x". Locating the corresponding branch name is more immediate if it at least begins with the same characters (otherwise you have to scan every name for the final segment).

[carljm]

Sure, that seems fine to me. According to @mrmachine above, the remote name is only sometimes included, so in order to implement this approach it would be necessary to figure out when it is or isn't, so the remote name alone can be reliably stripped if present.

[cjerdonek]

By the way, the pip documentation can probably use some clarification in this area. For example, the VCS Support section says:

The url suffix "egg=<project name>" is used by pip in it’s dependency logic to identify the project prior to pip downloading and analyzing the metadata.

which doesn't leave open the possibility of suffixes of the form "egg=<project_name>-<project_description>" or say what the "suffix of the suffix" is for. This section has a similar issue because all of the examples are of the form:

$ pip install -e git+https://git.repo/some_pkg.git#egg=SomePackage          # from git

[aj-may]

+1

[lsolanka]

Hi all, is there any fix available for this issue?

[msabramo]

On Python > 3.2, it's even worse as the generated egg name is non-deterministic and can vary from run to run - see #1867

[pirate]

+1

[fermayo]

+1

[cjerdonek]

I'll start working on this and see if I can make some progress.

[cjerdonek]

I made a minimal fix with some tests for this issue: #2063

The fix simply translates slashes in the existing egg "surname" to underscores -- as discussed in this comment: #1083 (comment)

[paulmelnikow]

I'm happy to test this fix, probably will be able to do that tomorrow. Thanks!

[slorquet]

Hello,

I got this issue while trying to install an egg file that was generated in a subdirectory

pip install somesubdir/somepackage.egg

failed

cd somesubdir
pip install somepackage.egg

worked.

edit: whatever, pip does not work at all this way ! I need easy_install to install from an egg file.