Ignore updating certain dependency entries, line-based

[webknjaz]

In Python, we often have env dependencies for multiple envs listed in requirements files. It often affects test envs and/or building artifacts for different platforms.

Pip supports including dependencies based on certain conditions via env markers. Example:

cryptography<2.7; platform_machine != "x86_64"
cryptography==2.7; platform_machine == "x86_64"

This allows us to specify a custom version limit for env which newer dependency versions don't support (Cryptography dropped support for i686 arch in v2.7 but we still want to update version pins for x86_64).

So it'd be nice to have a way of instructing dependabot to not touch certain lines.

Prior art

In pyup, it's possible to use comments for this purpose and for setting the pin limits. Example:

cryptography<2.7; platform_machine != "x86_64"  # pyup: ignore

or

cryptography==2.6.1; platform_machine != "x86_64"  # pyup: <2.7  # cryptography v2.7 doesn't support i686 arch anymore

[greysteil]

Thanks for creating this. I've pushed a fix here.

In pyup, it's possible to use comments for this purpose and for setting the pin limits.

We've avoided implementing this in Dependabot because it doesn't feel right to have comments to Dependabot in your manifest file. If we have to resort to it we will, but for now I think ignoring < requirements will pretty much always be the correct thing for Dependabot to do. 🤞

[webknjaz]

@greysteil thanks for this!

I think ignoring < requirements will pretty much always be the correct thing

What I like about pyup is that it'd update the pin based on the comment if there was a patch version update still fulfilling the restriction. This way, pins are still set precisely while with < the deps may change over time.

Actually, secure pins is its own issue. It's recommended to use constraints files with hash pinning along with requirements.

There's this tool called pip-tools which helps us generate traversed dependency pins. So we can have a list of direct deps in requirements.txt and pinned ones in constraints.txt.

Example:

requirements.txt:

-c constraints.txt
dep1

constraints.txt:

dep1==v1 --hash=sha256:... --hash=sha256:
dep2==v2 --hash=sha256:... --hash=sha256:  # dep of dep1

With this pip install -r requirements.txt also compares check-sums of each dist downloaded.

I believe, dependabot doesn't support this workflow.

[greysteil]

Dependabot shells out to pip-tools directly when a requirements.in file is checked in, so it should be supporting pretty much anything pip-tools does. Do you have an example repo where it's not?

[webknjaz]

Oh, I didn't know that. I think there are some concerns here: pypa/setuptools#1566 (comment).

Also, what if I don't call that file requirements.in? Or if I have multiple such files?
Plus if you run pip-tools on your machine, it'd likely pick different OS-specific wheels unless you run it under different Pythons and OSs and then merge the results.

[greysteil]

It's not perfect but we have a bunch of logic to try and handle it. Mostly here. Picks up all .in files and checks if they look like they've got requirements in them. Also figures out the order to run in based on how they require each other. I generally suggest giving it a try and reporting here if anything doesn't work. 🙂

[ssbarnea]

Apparently the code vanished and I was not able to find any information regarding requirements.in inside the docs. A log of dark magic it seems.

[webknjaz]

@ssbarnea the new location is @ https://github.com/dependabot/dependabot-core/blob/20557d2/python/lib/dependabot/python/update_checker/pip_compile_version_resolver.rb#L23.