Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Environment variables in Pipfile does not resolve #2389

Closed
ilons opened this issue Jun 20, 2018 · 12 comments
Closed

Environment variables in Pipfile does not resolve #2389

ilons opened this issue Jun 20, 2018 · 12 comments
Labels
Status: Needs More Information This issue does not provide enough information to take further action. Type: Possible Bug This issue describes a possible bug in pipenv.

Comments

@ilons
Copy link

ilons commented Jun 20, 2018

When I add a second source in my Pipfile with a private pypi registry, and then try to run pipenv update (or anything else which tries to install from it), I'm getting an error.

I've found similar issue, both of which are closed:
#2351
#1906

If I set the username and password as part of the source url directly in the Pipfile it works as expected, also when I create a ~/.netrc file with the machine, login and password.

Expected result

I expect the packages defined in Pipfile to be installed

Actual result
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${PYPI_AUTH}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${PYPI_AUTH}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:
  privatepackage

Finding the best candidates:
User for pypi.domain.io: 
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.domain.io
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 85, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 74, in main
    system=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 65, in resolve
    allow_global=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 442, in resolve_deps
    pre,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 352, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 157, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 142, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

When possible, provide the verbose output (--verbose), especially for locking and dependencies resolving issues.

Steps to replicate
Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${PYPI_AUTH}@pypi.domain.io/pypi"
verify_ssl = true
name = "private-pypi"

[packages]
privatepackage = {version = "*", index = "private-pypi"}

[requires]
python_version = "3.6"

Command

env PYPI_AUTH="user:password" pipenv update

[Edit]: Updated Pipfile to fix spelling error.

@uranusjr
Copy link
Member

uranusjr commented Jun 20, 2018

Did you spell the variable wrong? PUPI_AUTH?

@ilons
Copy link
Author

ilons commented Jun 21, 2018

No, I didn't, it sneaked in when I created the issue.

I just did a second verification on it to be sure, and I can still reproduce, with the exception that I don't get it to work with netrc (as I thought I did initially).

When I put the user/pass directly in the Pipfile it works as expected.

Reproduced with:
mkdir test; cd test

Pipfile:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${PYPI_AUTH}@pypi.domain.io/pypi"
#url = "https://user:[email protected]/pypi"
#url = "https://pypi.domain.io/pypi"
verify_ssl = true
name = "private-pypi"

[packages]
six = "*"
privatepackage = {version = "*", index = "private-pypi"}

[requires]
python_version = "3.6"
Running:

PYPI_AUTH="user:pass" && pipenv update || echo "${PYPI_AUTH}"

Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
elf.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

user:pass

@techalchemy
Copy link
Member

Pipenv doesn’t parse netrc files, if that’s relevant; it only reads environment variables

@ilons
Copy link
Author

ilons commented Jun 22, 2018

Don't mind netrc, that was merely a distraction of mine it seems.
My problem is that pipenv does not read env variables as expected.

@techalchemy
Copy link
Member

techalchemy commented Jun 22, 2018

Based on the error I think it is residing the variable but downstream there is an expectation of these being separate and for security reasons we only interpolate at the last possible moment. The lack of a separator is potentially throwing things off. Can you try passing these in separately as ${USER}:${PASS}?

@shawnzhu
Copy link

shawnzhu commented Jun 25, 2018

I can reproduce this issue on release 2018.05.18. However, it works when embedding the credential in source URL inside Pipfile directly like that in #2389 (comment)

Update: I can reproduce this issue on release 2018.06.25 (released 4 hours ago) too.

PS: ~/.netrc works when no authentication specified for index-url for pip9 since requests supports it

I've spent some time to diagnostic this problem via running pipenv lock --verbose and found out the problem would be pipenv/utils.py where it uses index-urls from Pipfile.lock directly without evaluating it via os.path.expandvars(), so when it performs GET request against pypi source, it doesn't send a header named Authorization at all.

@techalchemy
Copy link
Member

@shawnzhu the approach we recommend and have recommended is ${USER}:${PASS} in your source header, otherwise these values are not properly split when they are parsed by urlparse. I am still waiting to hear whether using this approach provides resolution or not -- without that info, I'd say that we are not really looking further at this (which is why you can reproduce whatever the issue is)

@techalchemy techalchemy added Type: Possible Bug This issue describes a possible bug in pipenv. Status: Needs More Information This issue does not provide enough information to take further action. labels Jun 26, 2018
@ilons
Copy link
Author

ilons commented Jun 27, 2018

Passing them separately still gives the same issue:

Pipfile
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://${USER}:${PASS}@pypi.domain.io/pypi"
#url = "https://${PYPI_AUTH}@pypi.domain.io/pypi"
#url = "https://user:[email protected]/pypi"
#url = "https://pypi.domain.io/pypi"
verify_ssl = true
name = "private-pypi"

[packages]
six = "*"
privatepackage = {version = "*", index = "private-pypi"}

[requires]
python_version = "3.6"
Command

USER=user && PASS="pass" && pipenv update --verbose || echo "Got ${USER}:${PASS}"

Output
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead.
Running $ pipenv lock then $ pipenv sync.
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${USER}:${PASS}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple --extra-index-url https://${USER}:${PASS}@pypi.domain.io/pypi

                          ROUND 1                           
Current constraints:
  privatepackage
  six

Finding the best candidates:
User for pypi.domain.io: 
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.org
INFO:pip9._vendor.requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): pypi.domain.io
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 85, in <module>
    main()
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 74, in main
    system=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/resolver.py", line 65, in resolve
    allow_global=system,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 442, in resolve_deps
    pre,
  File "/usr/local/lib/python3.6/site-packages/pipenv/utils.py", line 352, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 193, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 257, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 157, in find_best_match
    all_candidates = self.find_all_candidates(ireq.name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 142, in find_all_candidates
    candidates = self.finder.find_all_candidates(req_name)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 456, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 606, in _get_pages
    page = self._get_page(location)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 723, in _get_page
    return HTMLPage.get_page(link, session=self.session)
  File "/usr/local/lib/python3.6/site-packages/pipenv/patched/notpip/index.py", line 832, in get_page
    "Cache-Control": "max-age=600",
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 488, in get
    return self.request('GET', url, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 386, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/sessions.py", line 602, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/_vendor/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/pipenv/vendor/pip9/download.py", line 181, in handle_401
    username = six.moves.input("User for %s: " % parsed.netloc)
EOFError: EOF when reading a line

Got user:pass

@techalchemy
Copy link
Member

You should update your pipenv version, but I also can't replicate this at all --

 /t/test2  cat Pipfile
[[source]]
url = "https://${FAKE_PYPI}/simple"
verify_ssl = false
name = "kenneth-mirror"

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.6"
 /t/test2  set -gx FAKE_PYPI pypi.kennethreitz.org
 /t/test2  pipenv install six
Creating a virtualenv for this project...
Pipfile: /tmp/test2/Pipfile
Using /home/hawk/.pyenv/versions/3.6.4/bin/python3.6m (3.6.4) to create virtualenv...
⠋Running virtualenv with interpreter /home/hawk/.pyenv/versions/3.6.4/bin/python3.6m
Using base prefix '/home/hawk/.pyenv/versions/3.6.4'
New python executable in /home/hawk/.virtualenvs/test2-h6PFdzq5/bin/python3.6m
Also creating executable in /home/hawk/.virtualenvs/test2-h6PFdzq5/bin/python
Installing setuptools, pip, wheel...done.
Setting project for test2-h6PFdzq5 to /tmp/test2

Virtualenv location: /home/hawk/.virtualenvs/test2-h6PFdzq5
Installing six...
Looking in indexes: https://pypi.kennethreitz.org/simple, https://pypi.org/simple
Collecting six
  Downloading https://pypi.kennethreitz.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: six
Successfully installed six-1.11.0

Adding six to Pipfile's [packages]...
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Updated Pipfile.lock (b69ba7)!
Installing dependencies from Pipfile.lock (b69ba7)...
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternativaly, run a command inside the virtualenv with pipenv run.

This works with pipenv update as well. We have tests running in CI that test against this. I can't replicate the behavior :/

@ilons
Copy link
Author

ilons commented Jul 5, 2018

@techalchemy Do you expect the changes made in #2484 will fix my issue as well?

@techalchemy
Copy link
Member

Those are released so if it helps it will work in the newest release

@ilons
Copy link
Author

ilons commented Jul 10, 2018

So I got around doing as suggested and updated pipenv version, and did a few tests to see where this started working.

Fedora 28, Python 3.6.5, pip 9.0.3:
11.10.0: Failed
11.10.1, 11.10.2, 11.10.3, 11.10.4, 2018.5.18, 2018.6.25, 2018.7.1: Working

Shame on me that I did not update to newest version before testing.

It works well in the past couple of releases.

@ilons ilons closed this as completed Jul 10, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Needs More Information This issue does not provide enough information to take further action. Type: Possible Bug This issue describes a possible bug in pipenv.
Projects
None yet
Development

No branches or pull requests

4 participants