[feature] Allow "conan config install" to merge remotes

[andrey-zherikov]

What is your suggestion?

Currently conan config install overrides all existing remotes. I'd like to be able to tell conan that the remotes should be merged instead.
Our use case is the following:

• We have "base" repo where we store company wide conan configuration including shared remote. We use conan config install <repo> to setup basic configuration.
• Every team within a company can tune the configuration and add team-specific remote. If a team has additional configuration, it's configured by second invocation of conan config install pointing to a team-specific configuration repo. Having only "merge" strategy for remotes forces teams to repeat shared remote in their configurations which contradicts with DRY principle. The reason to have multiple remotes is permissioning - some teams don't want their packages to be used outside of their teams or they want to restrict what packages can be shared (i.e. published to shared remote).

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Thanks for your suggestion @andrey-zherikov

This is mostly a UI problem, the main current behavior is the correct one, because being able to remove remotes is more important than allowing to append/add remotes, as for many users removing remotes like conancenter in their production pipelines is critical.

So that leave us with what would be an interface to allow this that wouldn't be bad, would be easy to implement and understand by users and not breaking behavior.

[andrey-zherikov]

I don't know how conan config install is implemented but I have two ideas out of my head:

1. Add a parameter in configuration that has "replace" behavior by default.
2. Add a parameter to remotes.json:

   {
    "strategy": "merge",
    "remotes": [
     {
      "name": "conancenter",
      "url": "https://center.conan.io",
      "verify_ssl": true
     }
    ]
   }

[samuel-emrys]

This would be a nice feature to have. We have an increasingly large number of conan repositories, and providing an easy way for users to easily install their desired intersection of repositories (from multiple configuration repositories, used for different purposes) is becoming more difficult. I think an interface similar to:

$ conan config install https://git.someserver.com/config.git --type git # --strategy=replace by default, no interface breakage
$ conan config install https://git.someserver.com/config.git --type git --strategy merge

would be good

[memsharded]

Hi all,

We have considered this issue, even @AbrilRBS implemented something in #14659. We have reviewed it, and the problem we found is that there will be too many corner cases, that will inevitably clutter the ux and make something difficult to manage and understand:

• The remotes is an ordered list. There is not unique merge strategy, remotes can be appended, prepended or inserted. I am pretty sure that this will be required by different users
• As a unique strategy will not satisfy all users, it will be necessary to invent a custom way to specify the relative order of remotes. This will make the feature way more convoluted difficult to understand, and a source of bugs, issues and support requests.
• When things are updated/merged, the use case of removing existing repos is also a need, like removing the default conancenter repo is still relevant, and it will also need a separate syntax.
• Merging introduces a bunch of error checking and consistency problems: what happens when 2 remotes have different names, but same URL? And same URL but different ssl connection? And same name, but different URL, do they overwrite the previous one by name?

Regarding of a merge strategy for other files, like global.conf, this will be increasingly difficult, as these files are now jinja templates. That means in practice that it is almost impossible to do "round trip" loading-updating-saving.

Every team within a company can tune the configuration and add team-specific remote. If a team has additional configuration, it's configured by second invocation of conan config install pointing to a team-specific configuration repo. Having only "merge" strategy for remotes forces teams to repeat shared remote in their configurations which contradicts with DRY principle.

Taking everything into account, this seems the only sustainable way. Because it is not just a copy&paste, but in many cases it will be more elaborate, and order is critical. As creating a remotes.json file in the config-install repo that joins together in the desired order both the "company-wide" remotes and the "team" remotes, and this is a one-time operation done only by one person, potentially once every many months, and this operation can take only a few minutes, at this moment we think it doesn't justify the complexity and maintenance that this feature will introduce, for the value.

I am closing this as not planned, thanks very much all for the feedback!

[samuel-emrys]

Unfortunate, but thanks for exploring the idea. We'll look at leveraging #14659 as a command extension for our own purposes.