pip install -e resets to origin/master

[lorin]

If I do this:

pip install -e git://github.com/nimbis/django.git#egg=Django-dev

Then pip will grab the default branch from that repository (1.4.1-patched) and install it, as you can see in the src directory where it's installed:

cd ~/.virtualenvs/nimbis/src/django
[(1.4.1-patched) ~/.virtualenvs/nimbis/src/django]
$ git log | head -n4
commit a5d7c7b3a2d6a729bf5ede2254b5c75be9da94d7
Author: Lorin Hochstein <...>
Date:   Mon Jul 30 21:44:20 2012 -0400

If I do it again, the HEAD becomes the same as the remote master branch, although it still uses 1.4.1-patched (default branch) as the branch name.

$ pip install -e git://github.com/nimbis/django.git#egg=Django-dev
[(1.4.1-patched) lorin@nibbler ~/.virtualenvs/nimbis/src/django]
$ git log | head -n4
commit e567f439bdfc60e16c465220fdaa8ea8a0fae936
Merge: c0748a6 226a3e7
Author: Alex Gaynor <...>
Date:   Sun Jul 29 17:07:55 2012 -0700

[qwcode]

which version of pip?

[lorin]

pip 1.1

[qwcode]

yes, I see exactly what you describe. sure looks like a bug.

but know that you can use the '@' parameter to reliably get the 1.4.1-patched branch on repeated installs.
pip install -e git://github.com/nimbis/[email protected]#egg=Django-dev

[lorin]

Yeah, that's what I'm doing now.

[mcdonc]

FWIW, I added a pdb.set_trace() to req.py update_editable() (line 380), and traced this down to the point where vcs/git.py(110)obtain() does if self.check_destination(dest, url, rev_options, rev_display):. Upon the second run of install, the values of dest, url, rev_options and rev_display are ('/home/chrism/projects/pip/testenv/issue647/src/django', 'git://github.com/nimbis/django.git', ['origin/master'], ''). rev_options is ['origin/master'] because there is no revision in the URL and that's its hardcoded default in the no-url-revision case. check_destination isn't quite as innocuous as its name; it winds up calling self.update(dest, rev_options) where rev_options is origin/master. I don't quite have a theory yet about how to fix it, but that's where the bug is.

[mcdonc]

Currently pip is inconsistent because it checks out the HEAD symbolic ref (aka the "default branch") when not instructed to do otherwise, but updates the master (not the default branch) when not instructed to do otherwise. Its checkout vs. update strategies are inconsistent.

There are two competing actions that could be taken to make pip consistent here:

• pip could always use the default branch (aka "HEAD symbolic ref", see http://feeding.cloud.geek.nz/2008/12/setting-default-git-branch-in-bare.html) of the remote and when
  updating a local checkout
  from the remote, it should fetch from the same default branch.
• pip should always use the master of the remote when first checking out the branch, and if a branch
  or other rev is not specified, it would then assume it needs to update from master.

The former is harder than the latter. I don't know any way to get the HEAD symbolic ref from a remote repository without actually just re-checking-out the entire repository and seeing what its initial branch name is. The latter is easier, but it might be confusing for people that use gitflow or some other model where the default branch is not master.

If there's a way to get the HEAD symbolic ref from a remote repo, we can easily issue a command to get it and use it. @lorin, do you know how to do that by any chance? If not, we'll likely either need to both check out and update master when otherwise unspecified, or close this as a wontfix.

[lorin]

@mcdonc I think you can use "git remote show" to do this, and it will tell you what the remote head branch is:

For example:

$ git remote show origin | head -n4
* remote origin
  Fetch URL: [email protected]:nimbis/django.git
  Push  URL: [email protected]:nimbis/django.git
  HEAD branch: 1.4.1-patched

[svetlyak40wt]

Seems, that this bug is still there.

[svetlyak40wt]

I mean, it is still resets to the given branch or revision even if there is uncommited changes. This is very bad behaviour — yesterday I lost few hours of work this way.

[guettli]

I think we have this problem, too.

The checkout of "pip install -e git+https:..." is broken.

I looked at the source vcs/git.py, and I guess this comment says it all:

    # refs maps names to commit hashes; we need the inverse
    # if multiple names map to a single commit, this arbitrarily picks one

I don't know why pip does a lookup of GIT-HASH-ID --> REV if I do install a git branch.

One GIT-HASH-ID can be referenced in several branches.

If two branches refer to the same hash-id, the guess is a guess and not deterministic.

---

What can I do to fix this? At the moment I don't get what pip does at all.

[cumber]

To set up development environments, we use a pip requirements file with lines like:

-e git+ssh://host/repository.git#egg=repository

There are a bunch of repositories that go into the environment. In development I'll use manual git commands to create branches in the particular repositories I'm working on. Every now and then, it's useful to re-run the pip install command that set up the dev environment (to pick up changes like new scripts installed to bin, deleted files, or new dependencies; things that aren't always automatically handled just by changing the files inside an editable-installed package).

If I have any non-master branches still checked out, pip hard resets those to origin/master; this results in losing uncommitted edits as @svetlyak40wt noted, but also any committed changes on the branch that aren't included on another branch (as well clobbering the branch pointer itself).

The way I would have expected pip to behave is one of these three:

1. Error out complaining about the wrong branch being checked out
2. Fetch the remote version of the locally checked out branch, and reset the local branch to that
3. Check out local master, and reset that to the newly fetched origin/master

[cumber]

The description on the pull request and the linked issue sound like the thing that was fixed is that pip would use origin/master when doing an editable install using an existing checkout, even though the default branch might not be origin/master. Is that all that's been fixed?

For me, the most problematic part of this issue wasn't the potential mismatch of origin/master and origin/HEAD, but rather that it updates the existing checkout by blindly doing a hard reset. That's a problem because it doesn't check what the locally active branch is; if you're actually using the checkout to do work in (and if you're not, what's the point of an editable install?) then the branch active in the local checkout is quite likely to be a topic branch, or to be ahead of the default branch on the remote. In those cases the hard reset loses committed work (as well as any uncommitted changes, of course).

Has this been addressed by the change that closed this issue? If not, should I report a new issue to get that looked at specifically, or should we reopen this one?

[cjerdonek]

Hi @cumber, I totally agree with you that a hard reset doesn't seem like the right approach for the reasons you mentioned. It seems a bit dangerous in fact because, like you said, you can even lose work. It's one of the things I noticed while preparing some of my refactoring PR's in this area of the code. It's also one of the things I was planning to address after some of my PR's are accepted, but so far it seems to be slow going for PR's to be merged.

On the subject of the issue tracker, while there are a few related issues I've come across, I don't think it's been specifically reported yet. The closest one I think is issue #2037 (since it's mentioned as a side comment), but that's still different. I'd recommend creating a new issue for it. I wouldn't reopen this issue because this issue was solving a different problem.

[cumber]

Thanks for clarifying, @cjerdonek. I had misinterpreted this issue as being about the hard reset. I'll raise a new issue then.