process-dependency-links not respected for subdependencies when generating pipfile.lock

[lbenezriravin]

When trying to install a private dependency with a private subdependency that has a private subdependency of its own, pipenv doesn't follow the specified dependency_links for the "subsubdependency" when generating Pipfile.lock.

$ python -m pipenv.help output

Pipenv version: '11.10.1'

Pipenv location: '/home/user/.local/lib/python2.7/site-packages/pipenv'

Python location: '/usr/bin/python'

Other Python installations in PATH:

• 2.7: /usr/bin/python2.7

• 2.7: /usr/bin/python2.7

• 3.6: /usr/bin/python3.6m

• 3.6: /usr/bin/python3.6

• 2.7.15: /usr/bin/python

• 2.7.15: /usr/bin/python2

• 3.6.5: /usr/bin/python3

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.17-300.fc27.x86_64',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Thu Apr 12 18:19:17 UTC 2018',
 'python_full_version': '2.7.15',
 'python_version': '2.7',
 'sys_platform': 'linux2'}

System environment variables:

• PYTHONDONTWRITEBYTECODE
• PIP_PROCESS_DEPENDENCY_LINKS
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/home/user/.local/bin:/home/user/bin
• SHELL: /bin/bash
• LANG: en_US.UTF-8
• PWD: /home/user/test/

---

Contents of Pipfile ('/home/user/test/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]
private_package = {ref = "master", git = "ssh://[email protected]/private/dependency.git", editable = true}

[requires]
python_version = "2.7"

---

Expected result

The Pipfile.lock should be correctly generated.

Actual result

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
Could not find a version that matches privatesubsubdependency
There are incompatible versions in the resolved dependencies.

Steps to replicate

Create three repos, the first of which (privatedependency) points to the second (privatesubdependency) in its dependency_links, and the second of which points to the third (privatesubsubdependency).

pipenv install -e 'git+ssh://[email protected]/private/dependency.git@master#egg=dependency'

Everything will be installed correctly, but pipenv will complain when attempting to generate the Pipfile.lock. If I separately add privatesubdependency to the Pipfile, things start working OK, but it defeats the point of pipenv when I start having to manually manage my subdependencies.

I know that process-dependency-links is deprecated but the "official best ™️ " solution (maintain a private PyPI-compatible server) simply isn't feasible for my team right now. We're closely following threads like pypa/pip#3610 and pypa/pip#4187, but until a better alternative is proposed, we rely on this feature pretty heavily.

Thanks!

[kennethreitz]

assigning this one to @techalchemy

[techalchemy]

I am pretty sure this is the only way to do what you're trying to do right now, so I agree with your assessment so far

For the record you probably don't need quotes around your dependencies...

Point of curiosity -- can you just set PIP_PROCESS_DEPENDENCY_LINKS=1 as an environment variable and have this work? I'm not really sure how else we will accomplish this

[lbenezriravin]

If I specify a subdirectory in my dependency, the ampersand gets mangled by bash without quotes, so I always keep the quotes there so I don't forget :)

PIP_PROCESS_DEPENDENCY_LINKS=1 is already set. Pipenv/pip installs the private dependencies fine, it's only the lockfile that fails.

[techalchemy]

Note to self: this doesn't work with lockfiles because we pass --no-deps

[bjmc-globus]

Is a fix (or workaround) for this issue on the roadmap? Is there anything I could do to help? Do you need test cases?

[techalchemy]

test cases would probably be helpful, @bjmc-globus but I think we merged a fix for this actually if you want to test out the current version on master -- see #2643

[techalchemy]

Closing, this is removed in the latest pip, fixed before that in pipenv, thanks all!

[wreed4]

I'm not sure pipenv has absorbed the change in latest pip correctly. When trying to use the new syntax (which works correctly in pip), I get the following error:

env/versions/3.6.3/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 235, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/home/william/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 415, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(download_dir, ireq, wheel_cache)
  File "/home/william/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 297, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/william/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 335, in _resolve_one
    add_req(subreq, extras_requested=available_requested)
  File "/home/william/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 288, in add_req
    wheel_cache=self.wheel_cache,
  File "/home/william/.pyenv/versions/3.6.3/lib/python3.6/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 171, in from_req
    "dependencies" % req
pipenv.patched.notpip._internal.exceptions.InstallationError: Direct url requirement (like privatedependency@ git+https://[TOKEN]@[URL_REDACTED].git) are not allowed for dependencies