pipx install from URL results in twice-cloned git repo

[Mattwmaster58]

Describe the bug
When running pipx install --verbose git+https://github.com/mygitrepo/here.git, the repo is cloned twice.

How to reproduce
Run pipx install --verbose git+https://github.com/yourgitrepo/here.git

Example output:

pipx > (run_subprocess:112): running /usr/bin/python -m venv --without-pip /tmp/tmpv8tj7pv7
pipx > (run_subprocess:112): running /tmp/tmpv8tj7pv7/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/matt/.local/pipx/shared/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /tmp/tmpv8tj7pv7/bin/python --version
pipx > (run_subprocess:112): running /tmp/tmpv8tj7pv7/bin/python -m pip list --format=json
pipx > (run_subprocess:112): running /tmp/tmpv8tj7pv7/bin/python -m pip install --no-dependencies git+https://github.com/Mattwmaster58/ftwar.git
Collecting git+https://github.com/Mattwmaster58/ftwar.git
  Cloning https://github.com/Mattwmaster58/ftwar.git to /tmp/pip-req-build-l1sp4c5w
  Running command git clone -q https://github.com/Mattwmaster58/ftwar.git /tmp/pip-req-build-l1sp4c5w
Username for 'https://github.com': Mattwmaster58
Password for 'https://[email protected]':
Building wheels for collected packages: ftwar
  Building wheel for ftwar (setup.py) ... done
  Created wheel for ftwar: filename=ftwar-0.0.1-py3-none-any.whl size=4970 sha256=3ab02c5acaa89e7907bb634b9d2bc52e3a9d60c41632cf792b40864335f75288
  Stored in directory: /tmp/pip-ephem-wheel-cache-72_5q14v/wheels/80/47/24/2c0770af0fdbf97d8d740332565bdfb8ee6b9bad14a39238f3
Successfully built ftwar
Installing collected packages: ftwar
Successfully installed ftwar-0.0.1
pipx > (run_subprocess:112): running /tmp/tmpv8tj7pv7/bin/python -m pip list --format=json
pipx > (install_package_no_deps:231): Determined package name: 'ftwar'
pipx > (_package_name_from_spec:104): Package name determined in 55.1s
pipx > (run_subprocess:112): running /usr/bin/python -m venv --without-pip /home/matt/.local/pipx/venvs/ftwar
pipx > (run_subprocess:112): running /home/matt/.local/pipx/venvs/ftwar/bin/python -c import sysconfig; print(sysconfig.get_path('purelib'))
pipx > (run_subprocess:112): running /home/matt/.local/pipx/venvs/ftwar/bin/python --version
pipx > (run_subprocess:112): running /home/matt/.local/pipx/venvs/ftwar/bin/python -m pip install git+https://github.com/Mattwmaster58/ftwar.git
Collecting git+https://github.com/Mattwmaster58/ftwar.git
  Cloning https://github.com/Mattwmaster58/ftwar.git to /tmp/pip-req-build-ja196apv
  Running command git clone -q https://github.com/Mattwmaster58/ftwar.git /tmp/pip-req-build-ja196apv
Username for 'https://github.com': Mattwmaster58
Password for 'https://[email protected]':
Collecting click>=7.1.1
  Downloading click-7.1.1-py2.py3-none-any.whl (82 kB)
     |████████████████████████████████| 82 kB 118 kB/s
Processing ./.cache/pip/wheels/08/8f/5f/253d0105a55bd84ee61ef0d37dbf70421e61e0cd70cef7c5e1/terminaltables-3.1.0-py3-none-any.whl
Collecting requests>=2.23.0
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
  Using cached urllib3-1.25.8-py2.py3-none-any.whl (125 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2019.11.28-py2.py3-none-any.whl (156 kB)
Collecting chardet<4,>=3.0.2
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Building wheels for collected packages: ftwar
  Building wheel for ftwar (setup.py) ... done
  Created wheel for ftwar: filename=ftwar-0.0.1-py3-none-any.whl size=4970 sha256=264614c49a3b6b02c5d778a90da7fbd456f662f941b06acb1d6e5d79760cceaa
  Stored in directory: /tmp/pip-ephem-wheel-cache-23sm4p6r/wheels/80/47/24/2c0770af0fdbf97d8d740332565bdfb8ee6b9bad14a39238f3
Successfully built ftwar
Installing collected packages: click, terminaltables, urllib3, certifi, chardet, idna, requests, ftwar
Successfully installed certifi-2019.11.28 chardet-3.0.4 click-7.1.1 ftwar-0.0.1 idna-2.9 requests-2.23.0 terminaltables-3.1.0 urllib3-1.25.8
pipx > (run_subprocess:112): running /home/matt/.local/pipx/venvs/ftwar/bin/python -c #!/usr/bin/env python3

Expected behavior
The repo should only be cloned a single time.

[cs01]

This is intentional to allow the user not to have to specify the package name.
See #147, #302.

[Mattwmaster58]

In that case, is there any way to make it so that it only clones once? And still determines the package name from the downloaded repo?

[Mattwmaster58]

Just looking at the source, could we maybe keep the dir from _package_name_from_spec always since it's only used in install, and then when it comes time to clone - check if the tempdir exists, and if so, use it?

[cs01]

Just looking at the source, could we maybe keep the dir from _package_name_from_spec always since it's only used in install, and then when it comes time to clone - check if the tempdir exists, and if so, use it?

That's a good suggestion, but it's nice to have all the venvs in once place, and named with the package name. It was also suggested to rename the temp venv after the name is determined, but that was shot down. #147 (comment). Maybe there is a way to do it, I don't think we tried to hard to make it happen. Feel free to hack on it and if you get something that works, submit a PR.

[Mattwmaster58]

I think the venv creation and source-fetching steps are 2 different things. In the logs in this issue, the source is cloned to /tmp/pip-req-build-l1sp4c5w and then later on it's cloned to /tmp/pip-req-build-ja196apv. It would appear we could just use dir 1 for everything.

The only pitfalls I can spot is that we'd have to cleanup the tempdir ourselves.

[cs01]

In the logs in this issue, the source is cloned to /tmp/pip-req-build-l1sp4c5w and then later on it's cloned to /tmp/pip-req-build-ja196apv. It would appear we could just use dir 1 for everything.

pipx creates a venv to isolate the installation, which is required to get the name. pip itself decides how to install things, including where to clone the repo to if a VCS specification is being installed, which are the two dirs you mentioned. Maybe there is a pip API that allows finer grained control of this, but it's not something pipx can control other than setting environment variables or passing arguments to pip.

[Mattwmaster58]

Oh, I see now. But couldn't the second python -m pip install git+https://github.com/Mattwmaster58/ftwar.git be replaced with python -m pip install /tmp/mypreviouslyclonedrepo Nevermind, I'm starting to understand the scope of the problem.

[Mattwmaster58]

Looking into it further, it looks like this would require quite a bit of work either on our side or pips.

[uranusjr]

One possible improvement for Git specifically is that the second call can clone the local repository instead of the original remote repository. But that smells like a hack to me…

[itsayellow]

Yes, originally we discussed ways to avoid installing twice, but this seemed the simplest way.

Also, the first time in the temp directory only the main package is installed (not the dependencies) so that speeds things up a little.

[itsayellow]

Closing this because it has been discussed here and before.