New goal for managing 3rd party dependencies

[kaos]

Intro

In order to help manage the packages and version specs in requirement files, a new goal could be introduced that
a) Updates the contents of requirement.txt files based on command line arguments, and
b) Regenerates affected lock files after such changes.

UX

Some ideas of what it could look like:

$ ./pants add --requirements=3rdparty/python/requirements.txt ansicolors@^1.1.0
13:48:27.15 [INFO] Added ansicolors<1.2.0,>=1.1.0 to 3rdparty/python/requirements.txt
13:57:16.34 [INFO] Completed: Generate lockfile for project-a
13:57:17.60 [INFO] Wrote lockfile for the resolve `project-a` to 3rdparty/python/lockfiles/project-a.txt
    Added ansicolors (1.1.8)

$ ./pants update --requirements=3rdparty/python/requirements.txt ansicolors
13:57:17.60 [INFO] Wrote lockfile for the resolve `project-a` to 3rdparty/python/lockfiles/project-a.txt
    Updated ansicolors (1.1.8 -> 1.1.12)

$ ./pants update --requirements=3rdparty/python/requirements.txt 
13:57:17.60 [INFO] Wrote lockfile for the resolve `project-a` to 3rdparty/python/lockfiles/project-a.txt
    Updated ansicolors (1.1.8 -> 1.1.12)
    Updated click (7.0 -> 7.1.2)

$ ./pants remove --requirements=3rdparty/python/requirements.txt ansicolors click
13:48:27.15 [INFO] Removed ansicolors, click from 3rdparty/python/requirements.txt
13:57:17.60 [INFO] Wrote lockfile for the resolve `project-a` to 3rdparty/python/lockfiles/project-a.txt
    Removed ansicolors (1.1.12)
    Removed click (7.1.2)

The --requirements flag is a bit of a nuance, which I hope we could get rid of (or at least alias to something shorter) using some kind of configuration or inference logic. It is not precisely clear to my at the moment how best to do this.

Consequences

I want to highlight two points here:

• There'll need to be a way to filter on resolve and/or 3rdparty package when generating lock files.
• The result of generating a lock file would need to return a diff of what has been changed, in order to give nice feedback to the user about what effect the updated requirements had.
  ✅ Print diff report when generating new lockfiles. #17347

Motivation

To get on par with poetry's dependency management features, so that I, as end user, can stick with one tool for the job, rather than two ;)
I'm lazy, and don't want to muck about in requirements.txt files by hand (and regenerating constraints files etc)

[kaos]

Considering that lock files contains interpreter constraints, these should be included in the output, so grouping by those, could look something like:

$ ./pants update --requirements=3rdparty/python/requirements.txt 
13:57:17.60 [INFO] Wrote lockfile for the resolve `project-a` to 3rdparty/python/lockfiles/project-a.txt
  python_version >= "2.7" and python_full_version < "3.0.0" or python_full_version >= "3.6.0"
    Updated ansicolors (1.1.8 -> 1.1.12)
    Updated click (7.0 -> 7.1.2)
  python_version >= "3.6" and python_full_version < "3.0.0" or python_full_version >= "3.3.0" and python_version >= "3.6"
    Updated ansicolors (1.1.8 -> 1.1.10)

Preferably, optionally using some terminal colors, to make the interesting bits stand out more, and the interpreter constraints lines rather dim, for instance, to make it more readable.

If this makes sense (not really used to these kinds of constraints).

[Eric-Arellano]

Thanks! I suspect we'll want to keep this on the generate-lockfiles goal as an option, like ./pants generate-lockfiles --resolve=data_science --upgrade='foo'. Avoid a proliferation of goals.

add and remove are a bit more tricky though...That would translate to: "create a new python_requirement_library target with this dependency in some BUILD file", and "delete all python_requirement_library targets with this project_name". Feasible, but desirable to be changing BUILD files like that programatically? (./pants tailor suggests maybe so?)

Note that add and remove would not mean "add or remove to this particular resolve", only "change the universe of what requirements Pants knows about." What goes into a resolve is computed by looking at every target that uses that resolve, and probably also #12848.

[kaos]

mm.. OK, I was unsure of the top level command names, and I see that what you suggest works, but it is not entirely user friendly to type.. I'm thinking that if we compare with git as example, there's low level "porcelain" commands that is the glue used by higher level "end user" commands. I kind of like that, you get the power at your hands when you need them, but there are high level ones for common tasks that are easy to use.

Regarding add and remove would be easier if targeting a requirements.txt file, that is picked up by some python_requirements target. Hmm... maybe going through python_requirement_library would be more robust, if we can trace them back to the originating requirements.txt, if there is one. A little trickier perhaps, yes, and actually not as important as update.

[Eric-Arellano]

OK, I was unsure of the top level command names, and I see that what you suggest works, but it is not entirely user friendly to type..

Agreed, but counterpoint is that it's not user friendly for ./pants help goals to be really long etc. There's a tradeoff with everything: here it seems to me less typing vs. more goals to deal with. I think it's valuable to keep dependency management limited to ./pants generate-lockfiles because that gives a namespace for dependency management related commands.

I know other tools like Yarn and Poetry do have add and update commands, although I think their purpose is different than Pants. They are fundamentally package managers, which also support things like running yarn test. Pants is instead fundamentally a build orchestrator, where dependency management is only one of many different things it does.

[cognifloyd]

This seems related to #15704