Pipfile.lock changes local editable dependency to VCS

[gsakkis]

1. I have defined my project as editable in Pipfile with myproject = {path=".", editable=true}.
2. Without an initial Pipfile.lock and an existing virtualenv, pipenv install generates a Pipfile.lock with

"myproject": {
    "editable": true,
    "path": "."
},

3. Running pipenv lock now changes the Pipfile.lock to:

    "myproject": {
        "editable": true,
-       "path": "."
+       "git": "[email protected]:xxxxx/myproject.git@xxxxxxxxxxxxxxxxxxx"
    },

Expected result

The Pipfile.lock shouldn't have changed.

Describe your environment

Pipenv version: 8.3.2

[ipmb]

FWIW, I just tested this on 8.3.2 and was not able to replicate it:

$ rm -rf $(pipenv --venv) Pipfile.lock
$ cat Pipfile
[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

"myproject" = {path = ".", editable = true}



[dev-packages]
$ pipenv install
Creating a virtualenv for this project…
⠋Using base prefix '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/pete/.virtualenvs/myproject-sF_Aa0AH/bin/python3.6
Also creating executable in /Users/pete/.virtualenvs/myproject-sF_Aa0AH/bin/python
Installing setuptools, pip, wheel...done.

Virtualenv location: /Users/pete/.virtualenvs/myproject-sF_Aa0AH
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (d7c923)!
Installing dependencies from Pipfile.lock (d7c923)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run the following:
 $ pipenv shell
$ grep myproject -A3 Pipfile.lock
        "myproject": {
            "editable": true,
            "path": "."
        }
$ pipenv --version
pipenv, version 8.3.2

[gsakkis]

The issue shows up when running pipenv lock after pipenv install.

[techalchemy]

@gsakkis I wasn't able to reproduce this. Out of curiosity, what was your project folder called?

[gsakkis]

Ok, I looked into it closer and it turns out it manifests only when there is at least one other VCS dependency in the Pipfile. Sample Pipfile to reproduce:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
myproject = {path=".", editable=true}
requests = {git="https://github.com/requests/requests.git", ref="24092b11d74af0a766d9cc616622f38adb0044b9"}

If the requests dependency is removed or changed to a non-git ref, the Pipfile.lock doesn't change after pipenv lock (i.e. it preserves "myproject": {"editable": true, "path": "."})

[techalchemy]

@gsakkis if you install a local path in editable mode, it will pin the package version in the lockfile. In this case, since the local path is a git repository and has a setup.py file, it installs myproject as the local path and as part of its resolution of myproject it pins the git repository and reference as well. If you don't install in editable mode, this resolution doesn't occur in the lockfile.

[gsakkis]

@techalchemy pinning the local path in editable mode would be ok If it weren't for the fact that

1. it seems to happen if (and only if) other (unrelated) dependecies in Pipfile are VCS references
   and
2. pipenv install is inconsistent with pipenv lock; the first one doesn't pin, that second does (subject to 1)

[gsakkis]

Why was this closed?

[hnykda]

I object about closing this as well @techalchemy ! This is definitely not an expected behavior and took me a while to find out what's going on. If I don't have a VCS dependency, it behaves differently than when I have.

I have a pretty standard project: few public dependencies, the package I develop in a private repo and one of public dependencies needs to be installed from github becuase it doesn't have the latest features in PyPi release.

Just because of adding this simple dependency, I am not able to use pipenv install, because it also tries to download my package from (private) github repository. All is fine when I am not downloading the version of the public project from the repo.

My Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
redis = "*"
rq = "*"
"s3fs" = "*"
hug = "*"
elasticsearch = "*"
gunicorn = "*"
pandas = "*"
numpy = "==1.14.0"
rq-dashboard = {git = "https://github.com/eoranged/rq-dashboard", ref = "0b4949cbce05383277ed0de1ac3b85f0c94ca207"}
google-cloud-storage = "*"
requests = "*"
python-json-logger = "*"
"e1839a8" = {path = ".", editable = true}


[dev-packages]
pytest = "*"
pycodestyle = "*"
"autopep8" = "*"
isort = "*"
vulture = "*"
pytest-mock = "*"
pytest-cov = "*"

[requires]
python_version = "3.6"

while my Pipfile.lock:

...
        "abss": {
            "editable": true,
            "git": "[email protected]:something/abss.git",
            "ref": "2b4a825288f77bf9a28a49e2e6035277f075bef6"
        },
...

and my setup.py:

from setuptools import find_packages, setup

setup(
    name='abss',
    version='1.0.0',  
    packages=find_packages(),
    description="blabla",
    python_requires=">=3.6",
    url='https://github.com/something/abss',
)

I had to uncomment url in my setup.py to make it work, which is of course wrong!

[techalchemy]

@hnykda we have about 3 other issues tracking this, can you include this information in one of the open ones? I know this is a problem and I know it’s really annoying (believe me, I use pipenv every day, I know). This one is just a tiny bit complicated I think, and we’ve been focused on the other fixed we needed.

[hnykda]

Cool. it's great if you could link related issues before closing it, I thought it's just abanonded and no one really cares.

I was trying to replicate this on some minimal example and I wasn't successful, unfortunately. Is there anything particular you want me to test?

[techalchemy]

@hnykda I think #1837 and #1690 are both the related issues and I have a clear idea about what is wrong, but a lot of dread about the fix... your case may be a touch different than all of these issues which is probably why you are having trouble reproducing -- It's not the url in the setup.py that matters but instead the use of pipenv update or something along those lines, basically it will put a second entry for your local path, but it will be a VCS entry instead of a local path entry. This isn't fixed because the code here tends to have cascading failures every time I change it (that's why this is a bug now) but I believe it is fixed in the complete refactor in #1962 if you want to take that for a spin

[baxen]

@techalchemy I found this thread while investigating the same issue. Below I've included a minimal example that I was going to post in case it helps, but I can confirm though that this is fixed by the refactor you linked to. Thanks for the solution and excited to see it go live 😁

The original post I had for reference:

git clone https://github.com/Suor/funcy.git
cd funcy

Using this Pipfile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
"e1839a8" = {path = ".", editable = true}
toolz = {git = "https://github.com/pytoolz/toolz.git"}

[requires]
python_version = "2.7"

If the virtual environment is completely clean (e.g. after running pipenv --rm), then pipenv lock will create a lock with the local install of funcy and a VCS install of toolz, as desired.

But once I actually install and then redo the lock

pipenv install
pipenv lock

only then does it add this entry to Pipfile.lock:

        "funcy": {
            "editable": true,
            "git": "https://github.com/Suor/funcy.git",
            "ref": "aa9dac86244313b47cfeff1918e076b1e0bf7776"
        },

This results in plenty of other issues depending on individual setups. Like @hnykda mentioned above it can cause problems with private repos, and it also can cause unfortunate issues when rebasing in git if the ref is deleted. I also find that this can lead to local edits to the code being ignored inside the virtualenv.