More generic (declarative?) way to hotfix dependencies version

[Carreau]

This is more a question/thoughts than a bug.
prompted by #32 I find it non-trivial to hotfix in a reliable way the metadatas of already released packages. In particular I don't understand how or why the fix applied there will only applied to existing releases. I'm also going to guess that the need to update dependencies requirements of already release packages is only going to increase.

Is there any interest of having a declarative way of declaring those updated metadata ? For example:

- 'distributed <  2.11.0':
    - 'msgpack < 1.0.0'

Would hotfix all distributed versions < 2.11.0 to rely on msgpack <1.0.0.

It may likely be that i'm completely misunderstanding something there.

[beckermr]

So the way the code works now is that we directly edit a copy of the index. This copy is then diffed from the original one to produce the data required by conda for the hot fixes.

We could support a variant of the syntax you suggest but I don’t personally have the time to implement it. I don’t think the syntax you suggest will cover all cases so we will still need the general edits we do now.

Feel free to go ahead for simple stuff if you want!

If you do, I’d be useful to expand the test suite we have. There are a few basic tests of the code that does the diffs and ideally if we add support for such syntax we’d want more tests.

[asmeurer]

IMO the conda solver should always prefer the maximal build number for a version, even if its constraints are more restrictive. That way you could always hotfix metadata by making a new build.

[dhirschfeld]

IMO the conda solver should always prefer the maximal build number for a version,

@wolfv - isn't that what mamba does?

[wolfv]

yes, that's what mamba does :) it only gets complicated when variants are required, or when hacks are used (such as using build number 1000+ for the compiler migration, and then releasing new packages with a low build number).

[beckermr]

Also it is worth remembering that this convention wouldn't solve all of the repodata patching issues we have. We sometimes have to patch repodata that was generated from old versions of packages that were shipped years ago. We'd have to rebuild all of those old packages, which might be impossible or at least a ton of work.

[asmeurer]

You could extract the existing package, modify the metadata, and retar it to make a new package. The only downside to this approach is that it's space inefficient, but the plus is that it doesn't require support on the repodata server, and the metadata in the package will match the metadata in the repodata.

[beckermr]

You could extract the existing package, modify the metadata, and retar it to make a new package. The only downside to this approach is that it's space inefficient, but the plus is that it doesn't require support on the repodata server, and the metadata in the package will match the metadata in the repodata.

True. However, this process would be pretty slow. We iterate a lot on repodata patches and having to redo this each time we change a patch would be awful.

I do think maximizing build number in more cases would help with a lot of things. My only point is that it doesn't solve all issues.

[wolfv]

@asmeurer i think that would also heavily go against reproducibility as the SHA256 of the package will change and conda-lock won't work anymore etc.

[CJ-Wright]

I think a declarative interface that supplements the code based interface would be great. This would enable the bot to make patches as needed.

[asmeurer]

@wolfv I think you're misunderstanding. It would be a new package. The build number would be bumped.

[wolfv]

@asmeurer yeah I see your point but the patches that conda-forge currently does are sometimes affecting all python packages (as far as I understand) and the non-patched packages would basically not work anymore.