Support uv compiled requirements files

[avilaton]

What are you trying to accomplish?

Trying to draft support for using https://github.com/astral-sh/uv as a replacement for pip-tools in dependabot. The reason for this is that uv is much faster and many projects have already started switching to it. UV is a pip-tools compatible replacement written in rust.

This is a proposal for:

• Support python uv as pip-compile compatible replacement #10039

Anything you want to highlight for special attention from reviewers?

Even if uv isn't adopted right now, it might be the long term solutions for pip-compile slowness. We use it for generating requirements.txt and each time dependabot does it with pip-compile we get a slightly different output which we later correct manually.

How will you know you've accomplished your goal?

The change I introduced is fairly simple. First I look at the requirements file header to identify if uv was used to generate it. If it was, I change the command used from pyenv exec pip-compile to pyenv exec uv pip compile.

Checklist

• I have run the complete test suite to ensure all tests and linters pass.
• I have thoroughly tested my code changes to ensure they work as expected, including adding additional tests for new functionality.
• I have written clear and descriptive commit messages.
• I have provided a detailed description of the changes in the pull request, including the problem it addresses, how it fixes the problem, and any relevant details about the implementation.
• I have ensured that the code is well-documented and easy to understand.

[mgaitan]

we at @Shiphero are looking forward to using this functionality as soon as it becomes available. Currently we use dependabot only as an alert to recompile manually as we have to manage everything via uv (and it generates slightly different results than pip-compile).

[mgaitan]

@edgarrmondragon what is missing to merge this? it is a very valuable feature.

[beagold]

Crosslinking github/docs#34569

[avilaton]

Crosslinking github/docs#34569

Thanks! I had a look at it and it would have taken me weeks to get around to it.

[sachin-sandhu]

Hi @avilaton , thank you so much for the dev work, really appreciate it !!

I'm one of devs working on dependabot
I'm taking a look at it to make sure everything is good to go. Will come back with more info if needed.

[avilaton]

Hi @avilaton , thank you so much for the dev work, really appreciate it !!

I'm one of devs working on dependabot I'm taking a look at it to make sure everything is good to go. Will come back with more info if needed.

First time I contribute here so it is very likely we are missing something. Edit as you see fit, I just thought this was a good way to get the ball rolling.

[sachin-sandhu]

Hi @avilaton , looks like you added 2 new functions uv_pip_compile_options_from_compiled_file , pip_compile_options_from_compiled_file in pip_compile_version_resolver.rb, i think they are being referenced from pip_compile_file_updater.rb can you please fix this (IDE is showing it as no function exists in Dependabot::Python::UpdateChecker::PipCompileVersionResolver)

[avilaton]

Would be neat to have a way not to repeat this code, it seems to me as if it would be more resilient to have them centralized. What do you suggest we should do?

[sachin-sandhu]

@avilaton , if you think we might benefit from moving them to new location/class and see a additional functionality that could be used across. please do so.

[sachin-sandhu]

@avilaton , same as above

[jonjanego]

Hi @beagold, @avilaton and other contributors. How is progress going on these changes?

[beagold]

Hey!

I am waiting for a review on the documentation pr I made, and I believe @avilaton has some changes pending on this pull request (as requested by a dev here).

I could take a look at what is needed here and propose a patch. Will have a look later today!

[jonjanego]

Hey!

I am waiting for a review on the documentation pr I made, and I believe @avilaton has some changes pending on this pull request (as requested by a dev here).

I could take a look at what is needed here and propose a patch. Will have a look later today!

I know there are some internal gates on the docs side, but I can help us work through those. Want to make sure that the code changes are moving forward though :)

[avilaton]

Hi @beagold, @avilaton and other contributors. How is progress going on these changes?

I don't want to delay this a lot, I meant it to be a surgical change on the command and it snowballed into supporting other options which I don't fully understand. I need help here. The current pip-compile way of doing this has options parsed in both the file_updater and the version_resolver side of things, which are not quite the same.
I don't want to alter ANY code path taken by current pip-compile use-cases, this far exeeds what I was going for here.

[beagold]

@avilaton hi!

I think the tests are failing due to your recent changes (I believe they were already failing before tho).

Im sorry to hear that this got a bit out for control for you. Ruby is still a bit new to me, but will try to spend the day today figuring out how to run and test dependabot locally and what changes still need to be done.

Again, thanks for getting this rolling!

[RazerM]

pip-tools defaults to --emit-index-url, but will skip the default index.

uv defaults to --no-emit-index-url so I think this should be

Suggested change

	options << "--no-emit-index-url" unless requirements_file.content.include?("index-url http")
	options << "--emit-index-url" if requirements_file.content.include?("index-url http")

[RazerM]

I don't think there's any reason to pass this to uv, since it's just there for better UX

[imajes]

@avilaton, @beagold, @jonjanego, @sachin-sandhu, @RazerM -- to help out the team here, i applied the comments/fixed bugs in PR #10688. Hope that helps land this faster! Happy weekend :)

[imajes]

OK, quick update- this isn't going to work as is. There are other bugs in it also, and now taking a deeper look at the way pdm/uv/pip-compile/etc are supported (or not) here, that's showing up a bit more to do. will come back to it shortly. @jonjanego - can you reach out to me please? thanks!

[jonjanego]

Hi all; any further progress on this PR?

[avilaton]

It snowballed into something I cannot really keep up with. No further progress from my end. All I wanted was to swap pip install by uv pip install. Turned out to be pretty hard in the end apparently, not sure why we found so much overhead. UV support should be doable in full support by just copying what you have for poetry as they are the same from where dependabot can see them IMHO. El mar, 15 de oct de 2024, 11:45 a. m., Jon Janego ***@***.***> escribió:

…

Hi all; any further progress on this PR? — Reply to this email directly, view it on GitHub <#10040 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAQ3J7XVEFQVTXL3CHWR7LLZ3UTBJAVCNFSM6AAAAABJR3YZD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMJUGE2DANJZGI> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jerbob92]

I would like to take over this PR, @avilaton can you tell me which bugs are still in here?

[mistercrunch]

Hey, wondering if there could be a way to tell dependabot which command to run, maybe in a .dependabot file (or similar) in place of the normal uv pip compile. For example people might want to specify arguments to uv pip compile, or in our case we want to run multiple commands atomically (one for requirements/base.in and another for requirements/dev.in) for instance. More details here: astral-sh/uv#5487 (comment)

An alternative would be to implement semantics on the uv side, meaning we'd tell uv that uv pip compile should run a certain set of commands.

More generally, at the dependabot level, it'd be great to allow people to alter the behavior/command(s) execute by dependabot to support their workflow, whether it's passing specific arguments or running multiple commands.

[beagold]

Hey, wondering if there could be a way to tell dependabot which command to run, maybe in a .dependabot file (or similar) in place of the normal uv pip compile. For example people might want to specify arguments to uv pip compile, or in our case we want to run multiple commands atomically (one for requirements/base.in and another for requirements/dev.in) for instance. More details here: astral-sh/uv#5487 (comment)

An alternative would be to implement semantics on the uv side, meaning we'd tell uv that uv pip compile should run a certain set of commands.

More generally, at the dependabot level, it'd be great to allow people to alter the behavior/command(s) execute by dependabot to support their workflow, whether it's passing specific arguments or running multiple commands.

Probably not the place to ask for this, but dependabot will use the same flags that were used then the file was initially generated, so it has implicit support for this

[mistercrunch]

Probably not the place to ask for this

I agree but now that we're derailed I'll keep pushing :)

So dependabot will read the comment at the top of requirements.txt file to figure out what ran it initially and run the same command? Will it do it if there are many requirement output files, if so on the same PR or different PRs?

I remember digging in the docs to try and figure out how to pass specific arguments to pip-compile or specify multiple files and figured it wasn't possible. Good to know, not sure where it might be or should be documented as I think most software engineers wouldn't intuitively think that the bot is parsing the comments in a file to come up with a plan of what commands to execute.

Side note: it explains some of the magic behavior I've seen in some repos and found myself wondering what incantations were running behind the scene.

[ddelange]

fwiw, we get a backwards compatible output using --no-annotate --no-header:

uv pip compile requirements/*.txt -o requirements/requirements.lock --no-annotate --no-header