pip install from package from github, with github dependencies

[dneise]

• Pip version: 8.1.1
• Python version: python 3.5 (Anaconda)
• Operating System: Ubuntu 15.10

Description:

The story goes like this: We wrote a little app shifthelper, which calls us on the phone, in case our experiment (a telescope on a remote island) has a problem. Then we found that part of the app, should be pulled out as a library smart_fact_crawler, since other apps for the same experiment might be able to use it.

Neither the little app nor the little library are on pypi, since we strongly believe only about 20 people in the world can use it, which we all know. At the same time the repos are of course public github repos.

We would like to enable our non-developer friends to be able to install the app (including all dependencies) without prior explicit cloning it. Basically we want the installation to be a one-liner.

What already works:

One can pip install the dependency first and then install the app, like this:

pip install git+https://github.com/fact-project/smart_fact_crawler
pip install git+https://github.com/fact-project/shifthelper

Or one can clone the repo of the app only, pip install the requirements.txt and then pip install the app like so:

git clone https://github.com/fact-project/shifthelper
pip install -r shifthelper/requirements.txt
pip install ./shifthelper

What I've run:

I searched the web and found the "dependency_links" keyword argument for setuptools.setup(). Although I also found comments here and there, telling that this feature is to be considered depricated and might vanish soon.

So I've put this into the setup.py of the app

    # ...
    install_requires=[
        # ...
        'smart_fact_crawler',
    ],
    dependency_links=[
        'git+https://github.com/fact-project/smart_fact_crawler.git#egg=smart_fact_crawler'
    ],
    # ...

And tried to install the app like this:

pip install git+https://github.com/fact-project/shifthelper

but pip complained like this in friendly red letters ;-)

Collecting smart-fact-crawler (from shifthelper==0.6.0)
  Could not find a version that satisfies the requirement smart-fact-crawler (from shifthelper==0.6.0) (from versions: )
No matching distribution found for smart-fact-crawler (from shifthelper==0.6.0)

---

My question now is:

Is maybe my entire approach wrong as in: What you are trying is not usually the way people work. You should instead try: ...

---

Related SO questions are:

• 1_mine
• 2
• 3

[pradyunsg]

Try running the install command with --process-dependency-links.

[pradyunsg]

@pfmoore Sorry for the mention on a completely unrelated issue.

I think this issue can be safely tagged as no response (or similar) and closed.

[dneise]

Oh sorry, I missed the reply.

I tried --process-dependency-links earlier, as it is mentioned in the SO threads also.
But it did not work.
(have no details at hand at the moment .. its in the middle of the night in europe ... will post details in 10h or so)

[pradyunsg]

Sure. Could you give me a quick summary of your situation? I'd like to see what can be done...

[mostrows2]

--process-dependency-links doesn't work for me:
"DEPRECATION: Dependency Links processing has been deprecated and will be removed in a future release."

After this message the installation attempts to look for my dependency package in pypi.

[dan-coates]

@dneise Were you able to get this to work? I remember hearing that git dependencies don't work when you're installing from git. So if you pip install git+some-git-package and it itself has a git dependency, it won't work, even with --process-dependency-links. That seems to be what I'm experiencing.

[chris-hailstorm]

@dan-coates Can you point us to where you heard that? It might explain my current problem.

[garywu]

The problem is the format of your link:

    dependency_links=[
        'git+https://github.com/fact-project/smart_fact_crawler.git#egg=smart_fact_crawler'
    ],

change to this, then run pip with --process-dependency-links:

    dependency_links=[
        'git+https://github.com/fact-project/smart_fact_crawler.git@master#egg=smart_fact_crawler-0'
    ],

The reason is contrary to the documentation, pip needs:

1. a version number, hence: '-0' at the end
2. a branch or tag, hence: '@master'

[kbuilds]

@garywu 's solution helped me. Didn't find any mention of having to specify a version in each dependency link in the docs.

I can confirm that in v9.0.1 pip does need a version number, but does not require a @branch to be specified for each link

[debugtalk]

@dneise Thanks for your thorough research and detailed sharing, it helps me a lot.

[pradyunsg]

Is there anything actionable here?

[chris-hailstorm]

@dneise You asked, is there a better way?

I've given up on this "-e" / "dependency_links" approach because (1) the problem has been going on for a long time, and (2) even if it's fixed, it won't apply to many versions of pip in common use.

The solution I'm using is to publish the package to a private PyPi-style repo and use a normal pip install that's made aware of the private repo in addition to the main PyPi repo. This requires the user to add an "--extra-index-url" param on the pip install command line, or an "extra-index-url" param in one of the pip config files. Downside: extra params / config. Upside: it works. The downside is not so bad if you add the new param to a config file, since that's set-and-forget.

I chose Gemfury as my repo service - there are several others to consider.

[hnykda]

Please don't deprecate --process-dependency-links. We have like 5 packages hosted on our company github repo and installing them through this solution is much easier then having to have a local pypi (extreme overkill + maintenance cost)

This work for our scenario:

1. package is on github in a private repo
2. we want to install it into site-packages (not into ./src with -e)
3. being able to use pip install -r requirements.txt
4. being able to use pip install -e reposdir (or from github), where the dependencies are only specified in requirements.txt

requirements.txt

...
git+ssh://[email protected]/{acc-name}/{repo-name}.git@{commit}#egg={package-name}

setup.py

from setuptools import setup, find_packages


def install_deps():
    """Reads requirements.txt and preprocess it
    to be feed into setuptools.

    This is the only possible way (we found)
    how requirements.txt can be reused in setup.py
    using dependencies from private github repositories.

    Links must be appendend by `-{StringWithAtLeastOneNumber}`
    or something like that, so e.g. `-9231` works as well as
    `1.1.0`. This is ignored by the setuptools, but has to be there.

    Warnings:
        to make pip respect the links, you have to use
        `--process-dependency-links` switch. So e.g.:
        `pip install --process-dependency-links {git-url}`

    Returns:
         list of packages and dependency links.
    """
    default = open('requirements.txt', 'r').readlines()
    new_pkgs = []
    links = []
    for resource in default:
        if 'git+ssh' in resource:
            pkg = resource.split('#')[-1]
            links.append(resource.strip() + '-9876543210')
            new_pkgs.append(pkg.replace('egg=', '').rstrip())
        else:
            new_pkgs.append(resource.strip())
    return new_pkgs, links

pkgs, new_links = install_deps()

setup(
    install_requires=pkgs,
    dependency_links=new_links,
    ...
)

[pamolloy]

@hnykda Putting dependency links in requirements.txt seems a bit convoluted. Why not just put everything in setup.py? requirements.txt seems like it was intended for random extra packages that aren't dependencies or required for tests.

I believe dependency links were deprecated in favor of PEP 508 support, which was merged into pip on 2/5/2017 and will be included in the v10 release. Searching the issues of this repo for "dependency links" should provide plenty of other information.

Since the release cadence for pip is rather slow I imagine you could use dependency links to try out the PEP 508 support before the v10 release by pointing to master.

[hnykda]

@pamolloy Thanks, good point. Having standard requirements.txt was just one of the requirements from our devs.

I understood that @ cannot be used for dependencies - isn't that true? By the way, even the fact that we have to add --process-dependency-links is terrible :-/ .

[pamolloy]

We only use Python for a very small number of projects. Up until very recently we were using git submodules to track dependencies. The --process-dependency-links flag is actually helpful in our case because our primary target has other build requirements and we only want to use dependency links for CI. But I totally agree that it isn't a perfect solution. I find the handling of versions to be frustrating.

Oddly enough it looks like if you have a requirements.txt file you should be fine (see this comment in #4175 that I linked to above).

[rr-]

PEP 508 looked good until I learned its implementation merged to pip is good for nothing.

    install_requires=[
        …,
        'pip @ https://github.com/raphaelm/pyenchant/archive/2.0.0.zip',
    ],

Direct url requirement (like pip@ https://github.com/raphaelm/pyenchant/archive/2.0.0.zip) are not allowed for dependencies

[pradyunsg]

I'll close this a duplicate of #4187; the discussion on this issue ended up going in that direction.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.