Git+SSH dependencies have subtle (yet critical) differences from git clone

[grahamc]

Reproduction steps:

1. Have a git repository which is private, and is a pip package.
2. Have a deploy key which can read the repository.
3. Confirm you can run git clone [email protected]:account/private-pip-package.git
4. Google around and discover that the instructions to use a git repo as a pip dependency boil down to very nearly "prefixing the repo with git+ssh:// and suffix it with #egg=private-pip-package."
5. End up with a pip dependency of git+ssh://[email protected]:account/private-pip-package.git#egg=private-pip
6. Try installing the package, but get a fatal: Could not read from remote repository. instead.
7. Run the gamut of tests (ssh [email protected] works, but ssh ssh://[email protected] fails ...)
8. Realize (after much hair-pulling) that instead of git+ssh://[email protected]:account/private-pip-package.git#egg=private-pip it should be git+ssh://[email protected]/account/private-pip-package.git#egg=private-pip (if you didn't catch that, its github.com/account instead of github.com:account)

I would propose that a simple note in the failure message, or checking on input, or what have you, about the : vs. / would be quite helpful.

[Ivoz]

After checking back on the docs, I noticed you can use that form of uri by [email protected]:account/repo (the last example documented).

[grahamc]

Maybe that only works for public repos? This is for a private repository.

[mgasner]

While this works for pip install git+ssh://[email protected]/account/private-pip-package.git#egg=private-pip, it doesn't help much when trying to use setup.py.

If you run setup(install_requires=['git+ssh://[email protected]/account/private-pip-package.git#egg=private-pip'], ...), you'll get an error like error in package setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers.

If you instead run setup(dependency_links=['git+ssh://[email protected]/account/private-pip-package.git#egg=private-pip'], install_reqs=['private-pip-package'], ...), you'll get an error like Could not find any downloads that satisfy the requirement private-pip-package (from package==1.2.3).

Perhaps you thought you'd be clever and do something like install_reqs=parse_requirements( 'requirements.txt')! Too bad, because #2244 (comment).

Is there really no lighter-weight, supported solution for requiring private repos than running your own PyPI?

[remohammadi]

If you instead run setup(dependency_links=['git+ssh://[email protected]/account/private-pip-package.git#egg=private-pip'], install_reqs=['private-pip-package'], ...), you'll get an error like Could not find any downloads that satisfy the requirement private-pip-package (from package==1.2.3).

Pip wants to know the version before fetching (git clone) the dependency. If we specify the version after the project name in the egg string, in this hacky form: egg=<project name>-<version>, then there will be an item in _find_all_versions()'s returned list, and everything goes fine. So if we want to dependent our package on the latest commit in the branch STABLE-X of a VCS repository, we can do this:

install_requires=[
    'private-pip-packag',
],
dependency_links=[
    'git+ssh://[email protected]/account/private-pip-package.git@STABLE-X#egg=private-pip-9999999',
],

But, it's too hacky! I think these days it's so reasonable to have a project dependant on the latest version of a branch of some VCS repository, and there should be no need on specifying a dummy version number as a workaround for this purpose.

[ajmazurie]

Hi guys,
I am having the very same problem of not being able to install a specific commit of a Github-hosted project using setuptools' dependency_links option, and was wondering if this was still considered a bug or a misunderstanding of setuptools.

Say, for example, that I want to install a specific commit of the project paramiko-expect; for example, https://github.com/fgimian/paramiko-expect/tree/943630ac499284e6441854a9c4ae1e04301bfdd9

Here is what works and what doesn't, using pip 8.1.2 and setuptools 23.1.0:

• 🆗 pip install "git+ssh://[email protected]/fgimian/paramiko-expect.git@943630a#egg=paramiko-expect-0.2"
• 🚫 (in dependency_links in setup.py) git+ssh://[email protected]/fgimian/paramiko-expect.git@943630a#egg=paramiko-expect-0.2
• 🚫 (in dependency_links in setup.py) https://github.com/fgimian/paramiko-expect/archive/943630ac499284e6441854a9c4ae1e04301bfdd9.zip#egg=paramiko-expect-0.2
• 🚫 (in dependency_links in setup.py) https://github.com/fgimian/paramiko-expect/zipball/943630a#egg=paramiko-expect-0.2

Note that the setup.py file then has paramiko-expect==0.2 in its install_requires field, and that I tried all possible variants of paramiko-expect versus paramiko_expect. The setup.py file is then used by typing pip install ..

Is there something I'm missing here?
Best, Aurélien

[BertrandBordage]

@mgasner @ajmazurie Same issue here… I almost got crazy by spending 3 hours testing all the possibly imaginable combinations for this, and couldn’t find a single one working.

Pip is always trying to find the private module on PyPI, which of course doesn’t work. Instead, it should simply install all dependency_links before checking that the correct version of each install_requires package is installed, and otherwise try to download it on PyPI.

Of course, installing what I put in dependency_links using pip install works.
And obviously, dependency_links items are ignored when they don’t match an install_requires entry, so we can’t even ask for the setup script to always reinstall the same module without checking it was installed.

[xavfernandez]

If you want pip to take dependenc_links into account, you should use the --process-dependency-links.

[ChillarAnand]

@xavfernandez --process-dependency-links is deprecated?

[xavfernandez]

@ChillarAnand it is, but without any deprecation timeline and available replacement so this is still the way to go.

[ghost]

soo, there is no way to install a git+https://{url/to/repo}@{branch}#egg={package}-{version} url in setup.py? my understanding was that your module's dependencies should go into setup.py and your pinned-for-release dependencies should go in requirements.txt. But If I cant load first party dependencies in setup.py, this work flow cant be done.. what?

Edit, I suppose this workflow is also complicated by the fact that pip doesn't preserve the source uri of its package, so pip freeze doesn't actually produce an install-ready file, since the "dependency links" equivalent names have to be manually inserted into the file. :(

[wobeng]

Any update on this issue? having the same problem

[max-sixty]

@wobeng What problem are you having? The solutions above worked for me

[wobeng]

I have a private package A on github and it requires a private package B on github. When I try to install A, it tries to installs B but stop short with an error saying Could not find a version that satisfies the requirement package B No matching distribution found for package B

[max-sixty]

What's the exact situation? What's the string in setup.py you're using?

[wobeng]

@MaximilianR are you on slack?

[max-sixty]

No

[wobeng]

setup(
    name='some-private library',

    // Omitted some less important stuff here...

    install_requires=[
        'some-git-dependency',
        'another-git-dependency',
    ],
    dependency_links=[
        'git+ssh://[email protected]/my-organization/some-git-dependency.git#egg=some-git-dependency',
        'git+ssh://[email protected]/my-organization/another-git-dependency.git#egg=another-git-dependency',
    ],
)

[max-sixty]

Try finishing egg with a number (I know it seems weird!), i.e. #egg=some-git-dependency-0

I've had success with http, but looks like others have with ssh. This is what works for me: 'git+https://{github_token}@github.com/company/{package}.git/@{version}#egg={package}-0'

[wobeng]

That means I will have to hard code token in the setup.py?

[max-sixty]

No, you can do github_token = os.environ['GITHUB_TOKEN']

[wobeng]

ok what about when you want to include a private package in requirements.txt? can you use os.environ?

[max-sixty]

I'm not sure about that actually, I'd be interested to know what works best there

[ssaumitra]

I am seeing this exact problem #2124 (comment) While @MaximilianR 's HTTPS solution might work, could someone make it work with SSH?

[TimZaman]

Still can't get this working properly.

[hnykda]

We were able to make it work only by using this solution: #3610 (comment)

[uranusjr]

I think this can be closed since we’re dropping the “SCP style” URL altogether. Only git+ssh://[email protected]/... will be supported going forward.

[sbidoul]

Maybe a little documentation work could alleviate the confusion mentioned by OP?

if you didn't catch that, its github.com/account instead of github.com:account

[gutsytechster]

For people facing issues with referring VCS URL in install_requires, this comment would resolve their query #8105 (comment)

Further, can it be considered a documentation issue? Will it make sense to add a note regarding #2124 (comment) here under VCS support section?

Also @uranusjr, why does your comment #2124 (comment) seems to conflict with the currently documented behaviour https://pip.pypa.io/en/stable/reference/pip_install/#git or is there something I am misunderstanding?

[uranusjr]

Not sure where the confusion is. The issue is about git+ssh://git@domain:path.git having subtle behaviour inaccuracies (compared to git+ssh://git@domain/path.git), but support for that URL scheme has been removed from pip (and the documentation reflects that).

[gutsytechster]

Ok, I was actually confused with this sentence

Only git+ssh://[email protected]/... will be supported going forward.

However, I think then this issue can be closed now. :)

[uranusjr]

See #7543 where the git@domain:path format was removed from documentation.

[aaronbriel]

Just to help out somebody who runs into an issue with git+ssh and stumbles across this thread -
For me, using -e git+ssh://github.com/account/private-pip-package.git#egg=egg_name in requirements failed with git ERROR: Command errored out with exit status 128, but using -e git+https://github.com/account/private-pip-package.git#egg=egg_name succeeded.