Suggestion: new command to bump versions of dependencies in pyproject.toml

[daisylb]

• I have searched the issues of this repo and believe that this is not a duplicate.

Issue

It would be awesome if Poetry had a command (let's call it upgrade) that bumped the version constraints of dependencies in pyproject.toml (as opposed to update, which afaict updates the lock file to the newest version within the constraint specified in pyproject.toml).

Some examples for how this command could behave:

• poetry upgrade django: Upgrade Django to the newest version that still works with other dependencies; equivalent to poetry remove django; poetry add django.
• poetry upgrade django djangorestframework: As above, but with more than one package at a time.
• poetry upgrade django=^2.1: Set the version of django to ^2.1, equivalent to poetry remove django; poetry add django=^2.1.
• poetry upgrade: Upgrade every dependency to the newest possible version. Equivalent to deleting the entire [tool.poetry.dependencies] section of pyproject.toml and running poetry add with a list of the names (but not versions) of every package previously in the list. (This one would be good for cookiecutter templates for projects, to make it easy to start a new project with the latest versions of everything.)

Currently, when I want to bump the version of something, I'm either running poetry remove ...; poetry add ... which moves the package to the bottom of the list in pyproject.toml, and results in uninstalling a bunch of dependencies which sometimes just get reinstalled again at the same version; or I'm manually editing pyproject.toml which means I have to look up the latest version manually, and I can't use Poetry's version resolution when I want to upgrade more than one package at a time.

[jgirardet]

It's a bit dangerous to upgrade everything to the last version. you might introduce many bugs doing this, without knowing where it comes from.

Anyway I also do this 😆

upgrade is very confusing with the update command.
We could have acommand per package and for all.

Maybe --reset-dependency package_name or/and --reset-dependency-all would be less confusing.
or : --find-latest, --upgrade-to-latest,--force-latest

[daisylb]

I agree; I only used ‘upgrade’ because I couldn’t think of something better :) It could also be added to ‘poetry add’ (either with or without a flag); currently that command just errors if you try to use it on a package that’s already in your dependencies.

…

On Tue, 2 Oct 2018 at 18:52, jgirardet ***@***.***> wrote: It's a bit dangerous to upgrade everything to the last version. you might introduce many bugs doing this, without knowing where it comes from. Anyway I also do this 😆 upgrade is very confusing with the update command. We could have acommand per package and for all. Maybe --reset-dependency package_name or/and --reset-dependency-all would be less confusing. or : --find-latest, --upgrade-to-latest,--force-latest — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#461 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAc7aLMzza48JHvRq92e6REbrmtJqsAzks5ugzAzgaJpZM4XDWit> .

[jgirardet]

I'll put a PR if @sdispater accepts the idea

[miracle2k]

The JS package managers make add do an upgrade of the version range, and that makes sense to me.

[kylebebak]

The JS package managers make add do an upgrade of the version range, and that makes sense to me.

@miracle2k
This is one of the many things that npm and yarn get right.

For example, if you yarn add abc, yarn might install abc version 2.2.1, and it will save this dep in package.json as "abc": "^2.2.1",.

In other words, when adding a package it uses the same caret prefix that poetry uses, and which permits subsequent updates to any version 2.X. I think this is a sensible default.

If you poetry add uWSGI, for example, it installs 2.0.18, but it saves this in pyproject.toml as uWSGI = "^2.0"... Why not save this as uWSGI = "^2.0.18"? This would behave the same under subsequent invocations of poetry update uWSGI, but it gives users immediate information about which version of the package is installed (2.0.18).

@jgirardet
I think a sensible implementation would involve a simple post-install step for both poetry add and poetry update. After running either of these commands, poetry could get the actual installed version of all dependencies and dev-dependencies, and update their versions in pyproject.toml, keeping the same caret or tilde prefix that the dep currently has.

In other words, if you have uWSGI = "^2.0.17" installed and you run poetry update uWSGI, it installs 2.0.18 and changes the line in pyproject.toml to uWSGI = "^2.0.18".

For wildcard (*) and gt, gte, lt and lte deps this behavior doesn't make sense, but caret and tilde requirements are by far the most commonly used.

[kylebebak]

If you make a script like this one at the root of the repo, make it executable, and run it with python3.7, it'll print the contents of pyproject.toml to the console after all ^ and ~ deps have been updated to their actual installed versions.

#!/usr/local/bin/python3.7
from typing import cast, Dict
import toml
import subprocess


def update_deps(name: str, version: str, t: Dict) -> Dict:
    def update(deps: Dict) -> None:
        for key in deps:
            v = deps[key]
            if type(v) is str and name.lower() == key.lower() and v[0] in ("~", "^"):
                deps[key] = f"{v[0]}{version}"

    update(t['tool']['poetry']['dependencies'])
    update(t['tool']['poetry']['dev-dependencies'])

    return t


with open('./pyproject.toml', 'r') as f:
    t = cast(Dict, toml.loads(f.read()))
    output = subprocess.run(["poetry", "show"], capture_output=True)
    lines = cast(str, output.stdout.decode()).split('\n')

    for line in filter(lambda l: bool(l), lines):
        name, version, *_ = line.split()
        t = update_deps(name, version, t)

    print(toml.dumps(t))

This is a goofy implementation that uses subprocess to call poetry show. It's just a proof of concept.

I think something like this happen after add and update are run, to update deps in pyproject.toml to their actual current versions.

[jmfederico]

If npm and yarn already do it in the add command, maybe keeping it consistent with what users already now would be better?

I also think that upgrade could be more confusing than useful. From other package managers, I would expect update/upgrade to just modify the lock file but not touch the dependencies definition file (tool in our case).

[sdispater]

If you use the latest beta release of the 1.0.0 version, you can upgrade your dependencies by using a specific constraint or the special latest constraint.

poetry add pendulum@latest
poetry add pendulum@^2.0.5

See #1221 for more information.

[max-wittig]

@sdispater But what about updating all dependencies at once?

Can we reopen this?

[Natim]

It is a bit tedious to go over all of them manually.

[dmwyatt]

While the command mentioned by @sdispater is nice, it doesn't go all the way to solving the problem talked about in this issue since you still have to go one by one and check each package.

[finswimmer]

I thinks the request about an upgrade command to update all dependencies to the latest available version is valid. So I reopen it.

[hay]

This would be a very useful addition. There's a tool for Node.js called npm-check-updates that might give some inspiration. I agree that an upgrade command would be confusing, but maybe something like poetry update ----upgrade-latest?

[donbowman]

npm also has
https://www.npmjs.com/package/npm-merge-driver

Next time your lockfile has a conflict, it will be automatically fixed. You don't need to do anything else.

which is nice.

reason being, is tools like dependabot keep it up to date, one by one, but, branches don't currently work w/ the poetry lock file since the hash line conflicts.

[Uninen]

This is something I need almost daily. I have a lot of projects that I want to keep up to date and Poetry is currently very cumbersome with that task. My ideal situation would be to have just one command to upgrade one or all dependencies (not at all interested bikeshedding about what that command would look like).

So, a very big +1 from me for this feature.

Also, it would probably be a good idea to document a suggested workaround/way to do this before we have the command. Using poetry add NameOfDependency@latest seems easy enough, and if there is no easy way to do this for multiple packages currently, maybe just mention it and link to this ticket or a PR with a "under development" note. Willing to write a PR for the docs if it would be accepted.

[Natim]

This is something I need almost daily.

Maybe you should consider to plug in @dependabot

[Uninen]

Maybe you should consider to plug in @dependabot

I do use Dependabot on projects that are on GitHub and Snyk on GitLab whenever I can, but I really wouldn't like to rely on third-party tools for something my package manager should handle. I mean, handling packages (dependencies) is literally the only thing I need the tool for.

[takeda]

@max-wittig Frankly, if someone needs to do such operation to all dependencies at once, they probably don't care about these constraints, and will be fine with no constraints (i.e. using "*" instead of "^1.2.3"), with that poetry update will do what you want. Your dependencies are still locked in the poetry.lock file.

[Natim]

@taketa Not really because that's something you may want to do on a regular basis weekly/monthly while making sure in the meantime nothing breaks because of a dependency change you haven't noticed and break something in production.

[max-wittig]

@takeda poetry update will only upgrade to minor and patch versions, not major.

@Natim Exactly, what he mentioned. We have renovate-bot for example, which automatically bumps versions. I don't want this happening automatically for major upgrades.

[takeda]

@max-wittig poetry update supposed to observe constraints in pyproject.toml ^1.2.3 allows to update to 1.3.4 but not too 2.3.4 (^ is more complex than that, but this is a ghist), if you use * there is no constraints.

[Natim]

@takeda if I understand correctly, you would set the version to * in pyproject.toml and then rely on poetry.lock to have your production build use the tested dependency and then run poetry update on time to time? It is a working workaround I guess.

What this issue is aiming for is to keep the pyproject.toml config file as a source of truth for the expected dependencies versions.

Thank you for the workaround though, it might work for my project.

[takeda]

@Natim yes, exactly. The reason for the constraints in pyproject.toml is to define such constraints that guarantee API compatibility.

Different authors have different ways they version their packages (for example if you use pytz, you probably want to have * there for it, since API never changes and your always want the latest time zone information) so that's where you use ^ and ~ (and also <, >, * more here: https://python-poetry.org/docs/dependency-specification/) to specify what versions are compatible with your application. If you have to do mass update that ignores these constraints, you aren't using them as intended.

[floatingpurr]

Interesting request. Such a command would be great!

[salvomcl]

Quick workaround:
poetry show -o -t | grep -v -e "--" | cut -d " " -f 1 | sed 's/$/\@latest/g' | xargs poetry add
This will upgrade all outdated top-level packages in a single transaction.
To continue even if a package fails installing, add -n 1 to xargs.

Disclaimer: this will incorrectly add new version of dev dependencies into normal dependencies.

Small improvement on the proposed solution to manage dev and non dev

poetry show -o -t --no-dev | grep -v -e "--" | cut -d " " -f 1 | sed 's/$/\@latest/g' > no_dev.txt
poetry show -o -t | grep -v -e "--" | cut -d " " -f 1 | sed 's/$/\@latest/g' > all.txt
join -v1 -v2 all.txt no_dev.txt > dev.txt
cat no_dev.txt | xargs poetry add
cat dev.txt | xargs poetry add --dev

Thank you @IceTDrinker very useful although it deletes extras dependencies

[IceTDrinker]

Thank you @IceTDrinker very useful although it deletes extras dependencies

yes I've made that discovery as well 😅 if you have something for it I would gladly take it :)

[sambyng-msft]

@MousaZeidBaker looks brilliant - thank you for sharing!

An option to --exclude certain packages would be nice. As a workaround, to do this i'm using --skip_exact option, and setting exact versions for packages i want to exclude.

However, sometimes I'll want to continue with minor/patch fixes within a major release and avoid skip_exact. So --exclude <PACKAGE_NAMES> would work well there.

[epage]

I think it could be useful to collaborate on this across ecosystems as consistency will help in understandability for users. I'm in a similar boat trying to figure out how to add first-party version-requirement upgrades to cargo, whether to put it in update or what to name a new command.

We have a collection of third-party plugins for editing the cargo manifest that I'm slowly moving into cargo, the first being cargo add which will be in the next stable release (though it doesn't have @latest syntax yet).

You can see my very rough notes on cargo upgrade at killercup/cargo-edit#685 while the official cargo issue is rust-lang/cargo#10498

[trallnag]

poetryup indeed works well. I installed it with pipx.

[epage]

For some reason I hadn't looked at poetryup before when comparing poetry with cargo upgrade (for Rust). I figured I'd do a comparison in case poetryup or poetry developers find something useful.

A subset of cargo-upgrade's output to compare with the above poetryup screenshot

• Both update the lock and requirements with compatible versions by default
• Both have a flag to upgrade past breaking changes (--latest vs --incompatible)
• Both can select and deselect specific dependencies, doing all by default
• poetryup let's you select subsets of dependencies with --group. I'm looking to add that to cargo upgrade
• poetryup --skip-exact will ignore dependencies pinned by the user while you have to opt-in with cargo upgrade --pinned
  • imo this should be opt-out (like cargo-upgrade) rather than opt-in (like poetryup )
• cargo upgrade has a --dry-run mode
• cargo upgrade reports the results in table to give more context, modeled after cargo outdated
• cargo upgrade treats all members of the project the same while poetry doesn't yet support sub-projects (Support subprojects in a poetry project #2270)

I've collected user care abouts in our thread on this topic but one in particular I want to highlight is

• Some library authors want to keep up on breaking changes but otherwise want to keep version requirements low to allow dependents to choose the version right for them (lower audit churn with fewer upgrades, workaround bugs, MSRV, etc)

We don't support this yet but I'm experimenting with this with Renovate (a more advanced Dependabot). With our current CLI, I would expect this to be cargo upgrade --incompatible --compatible=ignore && cargo upgrade --dep-kind dev. This is a bit verbose, both in flags and requiring two runs, and some see "compatible" and "incompatible" as opposites rather than selecting different upgrade candidates. Some have been requesting use-case specific flags like --only-incompatible. Unsure how this will shake out.

[MousaZeidBaker]

With the announcement of Poetry 1.2.0 and its support for plugin support, poetryup have been ported to poetry-plugin-up. Instead of being a standalone tool, the plugin provides an up command to Poetry itself in order to update dependencies and bump pyproject.toml. I strongly recommend using the plugin since it provides a more native experience and comes with new features such as dry-run, no-install etc.

Install the up plugin with:

poetry self add poetry-plugin-up

Usage:

poetry up --help

For more details visit poetry-plugin-up

[nfantone]

Have the need for bumping my pyproject.toml deps today and stumbled upon this thread. poetry-plugin-up worked great. Wish it was part of the core APIs.

[domoritz]

Up doesn't seem to work correctly (at least for me): MousaZeidBaker/poetry-plugin-up#48. Does it actually handle conflicts or just updates to the latest available versions?

[nbro10]

@domoritz The documentation says that poetry up a b would update a and b to the latest compatible versions ("Update dependencies to latest available compatible versions"), but in reality it does not. In fact, poetry up can fail because it seems it does not actually find the correct latest compatible versions (at least in some cases). See, for example, MousaZeidBaker/poetry-plugin-up#50

[wyattowalsh]

I recently created a Python3 script to update all Poetry packages to their latest version while preserving extras and dependency groups. Check it out here

[Elijas]

m.

It's a bit dangerous to upgrade everything to the last version. you might introduce many bugs doing this, without knowing where it comes from.

Anyway I also do this 😆

upgrade is very confusing with the update command. We could have acommand per package and for all.

Maybe --reset-dependency package_name or/and --reset-dependency-all would be less confusing. or : --find-latest, --upgrade-to-latest,--force-latest

Consider an old application with a minimal feature set that can easily be tested manually.

By not having a --force-latest you're compromising on security (old MAJOR and MINOR versions might not get security updates patches anymore), and the risk of breaking your application could be mitigated with those manual tests after the dependency version update.

[petros94]

This feature is open 6 years now, and people are still commenting. It's something that clearly the community needs.

[Natim]

Right now we use dependabot and renovate as an external weekly service that opens PR automatically with related updates.