Requirements.txt and git+ssh python packge urls

[elinep]

Hi,

In my python requirements.txt, I have some dependencies hosted on private git repositories.

# content of requirements.txt
git+ssh://MyGitServer/MyPackage.git@12365

I would like to clone and run experiments on multiple workers thanks to trains-agent.
It seems trains detects only package name and version and has no clue of the package source. Therefore workers fail to install my script dependencies.

Any hint?

Thanks

[bmartinn]

Hi @elinep, yes you should totally use trains-agent to do so :)

1. I'll check if we could get Trains to auto detect git+ssh packages,
2. Regrading the remote execution using trains-agent

• Clone the experiment you want to execute
• The new cloned experiment is in "draft" mode, which means everything is editable
• Go to Execution -> "Installed Packages" section and press on the edit button (it should hover over it, or just double-click on the text)
• Everything you write in the packages section is equivalent to writing it in a "requirements.txt". This means that in your case, just add the missing git+ssh:// line
• Trains-agent will install everything according to what we have in the "installed packages" section, so as long as you have the git/ssh keys installed on the remote machine, it should work out of the box. Notice that if you are running trains-agent in docker mode, your ~/.ssh folder will be mapped into the docker, so the keyring should already be available inside the docker :)

[elinep]

Ok, I confirm it works by manually editing the "installed packages" section.
Would be great if it could be handled automatically though ;)

Thanks

[bmartinn]

Awesome, I'm happy to hear the problem is solved.
I'm keeping this issue open until we have support for git/ssh package auto-detection :)

[bmartinn]

Well, solution is on its way... I assume...
Please 👍 on the original PIP issue and the PR

[bmartinn]

Woot Woot 🎊 pip PR merged pypa/pip#7612

[bmartinn]

Hi @elinep,
I forgot to update here, starting trains 0.15.0 trains and trains-agent fully support this feature
Just don't forget to upgrade pip > 20 😄

[elinep]

Hi @bmartinn

I missed your reply. While testing #219 (comment) I realized that the reported requirements on the UI does not show the git+ssh link. When remotely executing the task, trains-agent tries to install the package from the regular repository.

My requirements.txt is:

git+https://github.com/allegroai/trains.git@6dd7b4e02ec9201e2857ee4bfaa389e4cff04248
hydra-core==1.0.3

The reported requirements.txt is:

# Python 3.8.5 (default, Jul 28 2020, 12:59:40)  [GCC 9.3.0]
hydra_core == 1.0.3
trains == 0.16.3
# Detailed import analysis
# **************************
# IMPORT PACKAGE hydra_core
# script_hydra_trains_support_pattern_2.py: 3,4
# IMPORT PACKAGE trains
# script_hydra_trains_support_pattern_2.py: 2

And pip version is 20.2.4.

Did I miss something ?

[bmartinn]

@elinep , what are you getting on pip freeze ? (this is similar to how trains is analyzing the installed packages)

BTW:
I tested again on a clean environment, and it worked as expected.
One thing to note, pip introduced this capability in v20, that means that if a lower version was used to install trains, the information regrading the original source (i.e. git) is not stored anywhere.

[elinep]

It looks like wheel is also required for this feature. Updating pip and installing wheel before any git source packages solved the problem.

Thanks for your time @bmartinn