Update installed dependencies in an environment

[lbowenwest]

Is there a way to update or install new dependencies in an environment without blowing it away and reinstalling everything from scratch?

If I run hatch env create again, I just get a message saying Environment `default` already exists.

Unless I've missed something, could there be a --force flag to the create command or similar to allow this?

[ofek]

No update, but they are guaranteed to be in sync https://hatch.pypa.io/dev/environment/#dependencies

[lbowenwest]

Ah must have missed that bit thanks. Running something like hatch run true does the trick with that

[velikodniy]

@ofek Probably, but it would be convenient to add a command for syncing. A typical use case is adding a dependency to a virtual environment used by the user's IDE. To have the dependencies installed, the user could sync the packages using hatch run true or something, but it looks like a hack from the user's perspective.

[dlukes]

Being in sync doesn't mean automatically updating to the latest version though, or does it? E.g. if I depend on numpy and have 1.23.5 installed in the virtualenv, and then 1.24 comes out -- Hatch won't update to 1.24, because as far as it's concerned, 1.23.5 already satisfies the dependency, so the deps already are "in sync".

This is unfortunate, because to make sure I test my code against the latest versions of the dependencies, I always need to recreate the environments.

Or am I missing something?

[ofek]

That is correct and I think the current behavior is the right way. As a user I would not expect syncing to mean an action that always downloads the latest version.

Similarly, if you have any Python installation and are using pip then installing a minimum version of something that has already been installed and has a newly released version will be a no-op because pip caches packages.

[dlukes]

As a user I would not expect syncing to mean an action that always downloads the latest version.

Oh yes, definitely! I didn't mean that it's unfortunate that syncing doesn't download updates; I meant it's unfortunate there's no way to get updates other than re-creating the environment.

In other words: syncing works reasonably as it is, but it's not really a complete solution to the problem of updating installed dependencies in an environment, as stated in the title of the post.

Sorry for the confusion.

[ofek]

Oh my bad I misinterpreted. Yes I will add such an update command as part of the lock file commands, when that happens

[dlukes]

I thought it might be linked to that :) Thanks for clarifying!

[mattijn]

Is there an issue that I can register to? We've a direct dependency on a gitub repo (see pyproject.toml file). That particular repo gets updates without bumping its version, but the syncing process of hatch (eg.when doing hatch run test) is not updating this package.

Btw, this is different behavior compared to python -m pip install -e .[dev] which does reinstall direct linked repositories.

[janLo]

Is there a way to force hatch to update environment dependencies (not project dependencies)? We constantly run in the issue that our CI uses a newer ruff version than our developers, causing issues to only surface first in the CI.

The dependency declaration is the default-generated "ruff>=0.0.243".

[janLo]

The problem with this is that someone need to constantly monitor the ruff releases and make sure that the minimal release is always in sync with the latest release and notify all developers after an update to rebase their pyproject.toml to get the changes to the minimal release to prevent surprises after git push.

The alternative is to specify a specific release and monitor upstream + update in a coordinated way. This means extra tooling as well. Especially as ruff is not the only tool in the box.

In my opinion a "update my environment" command or better a a config option like "always update when used" config option.

[ofek]

a config option like "always update when used" config option

That sounds reasonable

[FeryET]

Any update on this?

[antonymilne]

I'm also very interested in this. To me it feels similar to #603, and I wonder if there's some way of doing it already using pip environment variables. Did anyone try that already? e.g. something using PIP_UPGRADE or PIP_UPGRADE_STRATEGY. I guess PIP_FORCE_REINSTALL would work but is too aggressive for most people's needs.

[maxschulz-COL]

Just came across this, and would like to add that I am also very interested in it. Currently just killing and recreating the environment

[wolfch-elsevier]

On this page:
https://hatch.pypa.io/1.9/config/static-analysis/
...it claims you can upgrade ruff :

You can pin the particular version of Ruff by explicitly defining the environment dependencies:

[tool.hatch.envs.hatch-static-analysis]
dependencies = ["ruff==X.Y.Z"]

...so I added :

[tool.hatch.envs.hatch-static-analysis]
dependencies = ["ruff>=0.4.2"]

Then I found the hidden static analysis env for my checked out repo called "myproj"

$ ls -l /home/vscode/.local/share/hatch/env/.internal/hatch-static-analysis/
drwxr-xr-x 3 vscode vscode 4096 Apr 30 15:38 myproj
$ ls -l /home/vscode/.local/share/hatch/env/.internal/hatch-static-analysis/myproj
drwxr-xr-x 3 vscode vscode 4096 Apr 30 15:48 REtfwipp
$ ls -l /home/vscode/.local/share/hatch/env/.internal/hatch-static-analysis/hgraph/REtfwipp/ 
drwxr-xr-x 4 vscode vscode 4096 Apr 30 15:48 hatch-static-analysis

and then confirmed the version of ruff:

$ ls -l /home/vscode/.local/share/hatch/env/.internal/hatch-static-analysis/myproj/REtfwipp/hatch-static-analysis/lib/python3.11/site-packages
[...others..]
drwxr-xr-x 3 vscode vscode 4096 Apr 30 15:48 ruff
drwxr-xr-x 3 vscode vscode 4096 Apr 30 15:48 ruff-0.1.8.dist-info

So by default hatch-1.9.7 used ruff-0.1.8, so then I blew away that env and re-ran :
hatch fmt --formatter -- -v . expecting it to recreate that hidden internal env but now with ruff-0.4.2, since the documentation says I can do:

[tool.hatch.envs.hatch-static-analysis]
dependencies = ["ruff>=0.4.2"]

...but that's not how it worked - I still see ruff-0.1.8 :

$ ls -l /home/vscode/.local/share/hatch/env/.internal/hatch-static-analysis/hgraph/REtfwipp/hatch-static-analysis/lib/python3.11/site-packages
[...other files...]
drwxr-xr-x 3 vscode vscode 4096 Apr 30 16:01 ruff
drwxr-xr-x 3 vscode vscode 4096 Apr 30 16:01 ruff-0.1.8.dist-info

So there doesn't appear to be a way to upgrade ruff - right?

[ofek]

Try master? It should be working

[wolfch-elsevier]

Hey - thanks for the really quick reply! I can try master, but our CI will need to depend on a released version, but yeah, let me try...

[wolfch-elsevier]

I built it, but the version is hatch-1.9.2.dev12 yet the official release I have installed is 1.9.7 - I built it via hatch build.