Handling of local Python dependencies

[Tenzer]

Hi

I have just got Dependabot set up on a Python project of ours, but it's giving us an error saying:

InstallationError('liblearn/python should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+',)

I've tried to reproduce the problem locally via the code in https://github.com/dependabot/dependabot-core/tree/master/helpers/python, and I am able to do so. The solution for this is to make sure the script runs inside the folder where the requirements.txt file is located, as since the dependency (specified as -e liblearn/python in our case) is a relative path to a folder within the same repository.

My Ruby is not strong enough to go through the code base to find out how to potentially fix this, and unfortunately it seems like it would be a non-trivial thing to fix as the error occurs inside the codebase for pip.

Looking at the pip codebase, I can see it takes a --skip-requirements-regex pattern for lines that should be skipped when reading the requirements file:
https://github.com/pypa/pip/blob/eeee9b81a224617057386bd621662b96d043562e/src/pip/_internal/req/req_file.py#L303-L313

It could probably be fixed using that for instance by skipping lines containing a slash (/), as it's not an allowed character in Python package names. I don't know if that sounds like a feasible solution or if another way to solve it might already be in the pipeline?

---

Another minor thing that I saw when testing the above issue out locally, was that any file which has a filename that ends with .txt is fed into pip, even if the file isn't related to Python at all. We for instance have a security.txt file inside our repository which is a static file used for https://securitytxt.org/. It's not a file that pip would understand so it would potentially also give us problems with Dependabot if it wasn't for the above problem.

Would it perhaps be an idea to change the patterns here to instead look for *req*.txt instead of *.txt?
https://github.com/dependabot/dependabot-core/blob/414903e909d21fac1cfdca6186751d238585dac3/helpers/python/lib/parser.py#L17-L18

[greysteil]

Thanks for such a thorough error report @Tenzer! I'm going to dig into this now.

[greysteil]

I think this is just a case of Dependabot not fetching path dependencies properly, which should be fixed by dependabot/dependabot-core@fc79abe. I'm deploying that now and will trigger a re-run on your repo once it's out.

On the .txt issue, I thought I had some handling for that, but perhaps not! We have a few folks use surprising names for their requirements file (pipenv.txt, for example), so can't scope it down easily, but we should be able to catch parser errors and ignore them for oddly named files. If you run into this can you ping @dependabot in the issue and I'll jump on it straight away?

[greysteil]

Just found the handling: it's here. Knew I'd written something around that (and that Dependabot wouldn't be working for lots of folks without it)!

[Tenzer]

Thanks for the quick turnaround, it seems to be working. There's now PRs made for me to look at. Thanks again :)

[greysteil]

Great! Apologies for the bug getting started. Do let me know any feedback as/when you have any.