Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

issue with pipx dependency conflicting with package dependency, Click #233

Closed
AlJohri opened this issue Oct 9, 2019 · 11 comments · Fixed by #252
Closed

issue with pipx dependency conflicting with package dependency, Click #233

AlJohri opened this issue Oct 9, 2019 · 11 comments · Fixed by #252
Labels
help wanted Extra attention is needed

Comments

@AlJohri
Copy link
Contributor

AlJohri commented Oct 9, 2019

I'm running into an issue with the clokta package. It depends on Click. It seems like it is pipx thinks it is already installed because Click is a dependency of pipx itself.

Requirement already satisfied: click in /usr/local/Cellar/pipx/0.14.0.0/libexec/lib/python3.7/site-packages (from clokta) (7.0)
╰─ pipx install clokta --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /Users/johria/.local/pipx/venvs/clokta
pipx > (run:97): running /usr/local/opt/python/bin/python3.7 -m venv --without-pip /Users/johria/.local/pipx/venvs/clokta
pipx > (run:97): running /Users/johria/.local/pipx/venvs/clokta/bin/python -m pip install clokta
Collecting clokta
  Using cached https://files.pythonhosted.org/packages/7c/de/02c8f5a51ebacea04e52745852a3dc691d40e78d0fdf0be9f53ea2855561/clokta-4.1.0-py3-none-any.whl
Collecting enum-compat (from clokta)
Collecting boto3 (from clokta)
  Using cached https://files.pythonhosted.org/packages/42/b5/03da45c451b4e7d31bc42e3b844cbb2e9fc6e8757705504d828a7295b27e/boto3-1.9.245-py2.py3-none-any.whl
Collecting beautifulsoup4 (from clokta)
  Using cached https://files.pythonhosted.org/packages/3b/c8/a55eb6ea11cd7e5ac4bacdf92bac4693b90d3ba79268be16527555e186f0/beautifulsoup4-4.8.1-py3-none-any.whl
Collecting keyring (from clokta)
  Using cached https://files.pythonhosted.org/packages/b1/08/ad1ae7262c8146bee3be360cc766d0261037a90b44872b080a53aaed4e84/keyring-19.2.0-py2.py3-none-any.whl
Collecting requests (from clokta)
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting configparser (from clokta)
  Using cached https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Requirement already satisfied: click in /usr/local/Cellar/pipx/0.14.0.0/libexec/lib/python3.7/site-packages (from clokta) (7.0)
Collecting botocore<1.13.0,>=1.12.245 (from boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/0e/84/b13b233a344de26796b7b457571c16f2b3d1fab8fb48931208d650cc4d97/botocore-1.12.245-py2.py3-none-any.whl
Collecting s3transfer<0.3.0,>=0.2.0 (from boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting soupsieve>=1.2 (from beautifulsoup4->clokta)
  Using cached https://files.pythonhosted.org/packages/5d/42/d821581cf568e9b7dfc5b415aa61952b0f5e3dede4f3cbd650e3a1082992/soupsieve-1.9.4-py2.py3-none-any.whl
Collecting entrypoints (from keyring->clokta)
  Using cached https://files.pythonhosted.org/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.245->boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting docutils<0.16,>=0.10 (from botocore<1.13.0,>=1.12.245->boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/22/cd/a6aa959dca619918ccb55023b4cb151949c64d4d5d55b3f4ffd7eee0c6e8/docutils-0.15.2-py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.245->boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: enum-compat, six, python-dateutil, jmespath, urllib3, docutils, botocore, s3transfer, boto3, soupsieve, beautifulsoup4, entrypoints, keyring, idna, chardet, certifi, requests, configparser, clokta
Successfully installed beautifulsoup4-4.8.1 boto3-1.9.245 botocore-1.12.245 certifi-2019.9.11 chardet-3.0.4 clokta-4.1.0 configparser-4.0.2 docutils-0.15.2 entrypoints-0.3 enum-compat-0.0.2 idna-2.8 jmespath-0.9.4 keyring-19.2.0 python-dateutil-2.8.0 requests-2.22.0 s3transfer-0.2.1 six-1.12.0 soupsieve-1.9.4 urllib3-1.25.6
pipx > (_symlink_package_apps:551): Same path /Users/johria/.local/bin/clokta and /Users/johria/.local/pipx/venvs/clokta/bin/clokta
  installed package clokta 4.1.0, Python 3.7.4
  These apps are now globally available
    - clokta
done! ✨ 🌟 ✨

This is installed via the latest homebrew release.

$ clokta --version
clokta, version 4.1.0
@AlJohri
Copy link
Contributor Author

AlJohri commented Oct 9, 2019

I got around the issue temporarily by force installing it into the virtualenv: $HOME/.local/pipx/venvs/clokta/bin/python -m pip install click.

@cs01
Copy link
Member

cs01 commented Oct 9, 2019

I just ran pipx install clokta --verbose but did not get that warning. Maybe it has something to do with pipx being installed with homebrew. Have you tried installing pipx with pip, then installing clokta?

@AlJohri
Copy link
Contributor Author

AlJohri commented Oct 11, 2019

@cs01 yes, it does not occur with pipx installed via pip. however, I'm not sure its entirely a homebrew issue

going back to the homebrew installed version, here are some debug details:

$ which pipx
/usr/local/bin/pipx
$ pipx --help
...
Virtual Environment location is /Users/johria/.local/pipx/venvs.
Symlinks to apps are placed in /Users/johria/.local/bin.
These locations can be overridden with the environment variables
PIPX_HOME and PIPX_BIN_DIR, respectively. (Virtual Environments will
be installed to $PIPX_HOME/venvs)
...
$ echo $PIPX_HOME # blank
$ echo $PIPX_BIN_DIR # blank
$ pipx list
venvs are in /Users/johria/.local/pipx/venvs
apps are exposed on your $PATH at /Users/johria/.local/bin
   package envdir 1.0.1, Python 3.7.4
    - envdir
    - envshell
   package homebrew-pypi-poet 0.10.0, Python 3.7.4
    - poet
    - poet_lint
   package nbdime 1.1.0, Python 3.7.4
    - git-nbdiffdriver
    - git-nbdifftool
    - git-nbmergedriver
    - git-nbmergetool
    - hg-nbdiff
    - hg-nbdiffweb
    - hg-nbmerge
    - hg-nbmergeweb
    - nbdiff
    - nbdiff-web
    - nbdime
    - nbmerge
    - nbmerge-web
    - nbshow
   package python-language-server 0.28.3, Python 3.7.4
    - pyls
$ pipx install clokta --verbose
pipx > (run_pipx_command:134): Virtual Environment location is /Users/johria/.local/pipx/venvs/clokta
pipx > (run:97): running /usr/local/opt/python/bin/python3.7 -m venv --without-pip /Users/johria/.local/pipx/venvs/clokta
pipx > (run:97): running /Users/johria/.local/pipx/venvs/clokta/bin/python -m pip install clokta
Collecting clokta
  Using cached https://files.pythonhosted.org/packages/7c/de/02c8f5a51ebacea04e52745852a3dc691d40e78d0fdf0be9f53ea2855561/clokta-4.1.0-py3-none-any.whl
Collecting configparser (from clokta)
  Using cached https://files.pythonhosted.org/packages/7a/2a/95ed0501cf5d8709490b1d3a3f9b5cf340da6c433f896bbe9ce08dbe6785/configparser-4.0.2-py2.py3-none-any.whl
Collecting keyring (from clokta)
  Using cached https://files.pythonhosted.org/packages/b1/08/ad1ae7262c8146bee3be360cc766d0261037a90b44872b080a53aaed4e84/keyring-19.2.0-py2.py3-none-any.whl
Collecting requests (from clokta)
  Using cached https://files.pythonhosted.org/packages/51/bd/23c926cd341ea6b7dd0b2a00aba99ae0f828be89d72b2190f27c11d4b7fb/requests-2.22.0-py2.py3-none-any.whl
Collecting enum-compat (from clokta)
Collecting beautifulsoup4 (from clokta)
  Using cached https://files.pythonhosted.org/packages/3b/c8/a55eb6ea11cd7e5ac4bacdf92bac4693b90d3ba79268be16527555e186f0/beautifulsoup4-4.8.1-py3-none-any.whl
Requirement already satisfied: click in /usr/local/Cellar/pipx/0.14.0.0/libexec/lib/python3.7/site-packages (from clokta) (7.0)
Collecting boto3 (from clokta)
  Using cached https://files.pythonhosted.org/packages/ef/f3/4ef8264ed1cf4628513de5537464da1cf60b4f8e7551c4f2611e1e3cb4bd/boto3-1.9.247-py2.py3-none-any.whl
Collecting entrypoints (from keyring->clokta)
  Using cached https://files.pythonhosted.org/packages/ac/c6/44694103f8c221443ee6b0041f69e2740d89a25641e62fb4f2ee568f2f9c/entrypoints-0.3-py2.py3-none-any.whl
Collecting certifi>=2017.4.17 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/18/b0/8146a4f8dd402f60744fa380bc73ca47303cccf8b9190fd16a827281eac2/certifi-2019.9.11-py2.py3-none-any.whl
Collecting chardet<3.1.0,>=3.0.2 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<2.9,>=2.5 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests->clokta)
  Using cached https://files.pythonhosted.org/packages/e0/da/55f51ea951e1b7c63a579c09dd7db825bb730ec1fe9c0180fc77bfb31448/urllib3-1.25.6-py2.py3-none-any.whl
Collecting soupsieve>=1.2 (from beautifulsoup4->clokta)
  Using cached https://files.pythonhosted.org/packages/5d/42/d821581cf568e9b7dfc5b415aa61952b0f5e3dede4f3cbd650e3a1082992/soupsieve-1.9.4-py2.py3-none-any.whl
Collecting s3transfer<0.3.0,>=0.2.0 (from boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/16/8a/1fc3dba0c4923c2a76e1ff0d52b305c44606da63f718d14d3231e21c51b0/s3transfer-0.2.1-py2.py3-none-any.whl
Collecting jmespath<1.0.0,>=0.7.1 (from boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/83/94/7179c3832a6d45b266ddb2aac329e101367fbdb11f425f13771d27f225bb/jmespath-0.9.4-py2.py3-none-any.whl
Collecting botocore<1.13.0,>=1.12.247 (from boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/86/b2/ce485bd915a1570911765e28d3cca075ff8a920eec11f7a7d0c9725fb730/botocore-1.12.247-py2.py3-none-any.whl
Collecting python-dateutil<3.0.0,>=2.1; python_version >= "2.7" (from botocore<1.13.0,>=1.12.247->boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl
Collecting docutils<0.16,>=0.10 (from botocore<1.13.0,>=1.12.247->boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/22/cd/a6aa959dca619918ccb55023b4cb151949c64d4d5d55b3f4ffd7eee0c6e8/docutils-0.15.2-py3-none-any.whl
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.247->boto3->clokta)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Installing collected packages: configparser, entrypoints, keyring, certifi, chardet, idna, urllib3, requests, enum-compat, soupsieve, beautifulsoup4, six, python-dateutil, docutils, jmespath, botocore, s3transfer, boto3, clokta
Successfully installed beautifulsoup4-4.8.1 boto3-1.9.247 botocore-1.12.247 certifi-2019.9.11 chardet-3.0.4 clokta-4.1.0 configparser-4.0.2 docutils-0.15.2 entrypoints-0.3 enum-compat-0.0.2 idna-2.8 jmespath-0.9.4 keyring-19.2.0 python-dateutil-2.8.0 requests-2.22.0 s3transfer-0.2.1 six-1.12.0 soupsieve-1.9.4 urllib3-1.25.6
pipx > (_symlink_package_apps:551): Same path /Users/johria/.local/bin/clokta and /Users/johria/.local/pipx/venvs/clokta/bin/clokta
  installed package clokta 4.1.0, Python 3.7.4
  These apps are now globally available
    - clokta
done! ✨ 🌟 ✨

As you can see, pipx is running /Users/johria/.local/pipx/venvs/clokta/bin/python -m pip install clokta.

If I manually run, /Users/johria/.local/pipx/venvs/clokta/bin/python -m pip install clokta, I get the following output:

Requirement already satisfied: clokta in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (4.1.0)
Collecting click (from clokta)
  Using cached https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl
Requirement already satisfied: requests in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from clokta) (2.22.0)
Requirement already satisfied: boto3 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from clokta) (1.9.247)
Requirement already satisfied: enum-compat in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from clokta) (0.0.2)
Requirement already satisfied: keyring in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from clokta) (19.2.0)
Requirement already satisfied: beautifulsoup4 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from clokta) (4.8.1)
Requirement already satisfied: configparser in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from clokta) (4.0.2)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from requests->clokta) (1.25.6)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from requests->clokta) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from requests->clokta) (2019.9.11)
Requirement already satisfied: idna<2.9,>=2.5 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from requests->clokta) (2.8)
Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from boto3->clokta) (0.2.1)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from boto3->clokta) (0.9.4)
Requirement already satisfied: botocore<1.13.0,>=1.12.247 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from boto3->clokta) (1.12.247)
Requirement already satisfied: entrypoints in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from keyring->clokta) (0.3)
Requirement already satisfied: soupsieve>=1.2 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from beautifulsoup4->clokta) (1.9.4)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from botocore<1.13.0,>=1.12.247->boto3->clokta) (2.8.0)
Requirement already satisfied: docutils<0.16,>=0.10 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from botocore<1.13.0,>=1.12.247->boto3->clokta) (0.15.2)
Requirement already satisfied: six>=1.5 in ./.local/pipx/venvs/clokta/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore<1.13.0,>=1.12.247->boto3->clokta) (1.12.0)
Installing collected packages: click
Successfully installed click-7.0

As you can see, manually running /Users/johria/.local/pipx/venvs/clokta/bin/python -m pip install clokta causes Click to get installed.

I think this is a pipx issue, not a homebrew issue @cs01

@cs01 cs01 added the help wanted Extra attention is needed label Oct 11, 2019
@AlJohri
Copy link
Contributor Author

AlJohri commented Oct 11, 2019

@cs01, I figured it out.

The way the homebrew formula works is by manipulating the PYTHONPATH: https://github.com/Homebrew/homebrew-core/blob/master/Formula/pipx.rb#L47

Homebrew python packages can work in two ways: 1) using a proper virtualenv and 2) using a global vanilla python where the PYTHONPATH variable points it to the packages it needs. because of the nature of pipx, we use the second

I believe the correct solution will be to explicitly unset the PYTHONPATH within pipx

@AlJohri
Copy link
Contributor Author

AlJohri commented Oct 11, 2019

as a quick proof of concept, this patch fixes the issue:

diff --git a/pipx/util.py b/pipx/util.py
index 94732d8..ba27eee 100644
--- a/pipx/util.py
+++ b/pipx/util.py
@@ -93,11 +93,12 @@ def get_site_packages(python: Path) -> Path:
 def run(cmd: Sequence[Union[str, Path]], check=True) -> int:
     """Run arbitrary command as subprocess"""
 
+    env = {k:v for k,v in os.environ.items() if k != 'PYTHONPATH'}
     cmd_str = " ".join(str(c) for c in cmd)
     logging.info(f"running {cmd_str}")
     # windows cannot take Path objects, only strings
     cmd_str_list = [str(c) for c in cmd]
-    returncode = subprocess.run(cmd_str_list).returncode
+    returncode = subprocess.run(cmd_str_list, env=env).returncode
     if check and returncode:
         raise PipxError(f"{cmd_str!r} failed")
     return returncode

@laixintao
Copy link

Same issue here:

➜ [tmuxp4] ~ pipx runpip pingtop install click
Requirement already satisfied: click in /usr/local/Cellar/pipx/0.14.0.0/libexec/lib/python3.7/site-packages (7.0)

Pipx believes click is already installed, but it's pipx's click, not my tools' click.

INFO: my pipx is from brew.

➜ [tmuxp4] ~ brew info pipx
pipx: stable 0.14.0.0 (bottled)
Execute binaries from Python packages in isolated environments
https://github.com/pipxproject/pipx
/usr/local/Cellar/pipx/0.14.0.0 (107 files, 674.8KB) *
  Poured from bottle on 2019-10-15 at 10:38:23
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/pipx.rb
==> Dependencies
Required: python ✔
==> Analytics
install: 277 (30 days), 541 (90 days), 801 (365 days)
install_on_request: 273 (30 days), 537 (90 days), 796 (365 days)
build_error: 0 (30 days)

@laixintao
Copy link

@AlJohri any quickfix?

@AlJohri
Copy link
Contributor Author

AlJohri commented Oct 15, 2019

@cs01 what do you think of my proposed fix?

@cs01
Copy link
Member

cs01 commented Oct 15, 2019

It looks fine to me. If you make the change in a PR and the unit tests still pass then I will merge it.

@jayvdb
Copy link
Contributor

jayvdb commented Oct 23, 2019

ping @AlJohri ... your proposed change passes the existing unittests ... submit a PR?

It would be great if we could also have a test which demonstrates the problem and shows that it is fixed.

@AlJohri
Copy link
Contributor Author

AlJohri commented Oct 23, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants