New "conf" configuration mechanism

[memsharded]

Changelog: Feature: Introducing a new [conf] section in profiles that allows a more systematic configuration management for recipes and helpers (build helpers, toolchains). Introducing a new conan_conf.txt cache configuration file that contains configuration definition with the same syntax as in profiles.
Docs: conan-io/docs#1993

Profiles can have:

[conf]
tools.microsoft.MSBuild:verbosity=Minimal
pkg_pattern*:tools.microsoft.MSBuild:verbosity=Quiet

Profiles confs:

• Can compose multiple profiles, as usual, latest one wins

A new file conan_conf.txt (name to be defined, can contain):

tools.microsoft.MSBuild:verbosity=Minimal
pkg_pattern*:tools.microsoft.MSBuild:verbosity=Quiet
core:required_conan_version=>=1.30

tools.* and user.* configuration from conan_conf.txt is composed with the profiles ones. The rest as core:required_conan_version are internal to Conan and will not travel with profiles, recipes cannot see them.

Recipes and helpers access them with:

verbosity= conanfile.conf["tools.microsoft.MSBuild"].verbosity  # None if not defined

The tools.microsoft.MSBuild:verbosity is actually applied in 2 different helpers the CMake one that had a msbuild-verbosity argument and the MSBuild() one.

Things that are configuration at the moment are removed from helpers. It is not necessary that users define something like the verbosity of MSBuild in each recipe, better remove them now, and we will focus one by one if some needs to be added in the helpers explicitly (to override/force the config, or to define a different default).

At the moment, no adding command line --config version, but it can be added later relatively easily, I had added it before, then dropped for simplicity at this moment.

[lasote]

Like it. Go ahead.

[danimtb]

I really like the syntax of the config in profiles. There are some missing tests like inheritance of profiles or implementing a merge with the general conan conf, but for the first iteration I think it a very good step forward! 👍

[uilianries]

Some points:

• I would consider all attributes and values as lower-case only. It avoids case-sensitive problems (package reference is a good example). e.g
  tools.microsoft.msbuild:vebosity = detailed

• I would use .ini or .toml format for that new file, including as extension to enforce code editor highlight.

• conan_conf.txt is temporary, so I would consider layout.toml or layout.ini, because we already have conan.conf as global file, so when someone says "configuration file", won't be confused by local or global file if they have different names.

• How do I know if I should use a configuration or an recipe option? For instance, there are projects which offers options to control compiler flags, but looking for this feature, they should be a configuration (which can affect or not the binary result).

• Here we are changing Conan code base, but we will need to increase it in the future, as we will receive more features requests. So, how can we extend it without creating a snowball? What alternative can we provide for users for extending their configurations globally? For instance:

  tools.makefile.gcc:verbosity = detailed

  We don't have such config right now, but let's say I want it, but for my company only. I could provide an external python file which contains such treatment. That python file could be something new, in a new folder, like ~/.conan/configs/enterprise.py, Or, a conan recipe which can used with python requires.

[SSE4]

I've just checked, the following worked for python 2.7:

[conf]
tools.microsoft.MSBuild:verbosity=Minimal
pkg_pattern*:tools.microsoft.MSBuild:verbosity=Quiet

import ConfigParser

class MyConfigParser(ConfigParser.RawConfigParser):
    import re
    OPTCRE = re.compile(
    r'(?P<option>[^=\s][^=]*)'          # allow only =
    r'\s*(?P<vi>[=])\s*'                # for option separator
    r'(?P<value>.*)$'
    )

config = MyConfigParser()
config.read("1.conf")

for section in config.sections():
    print("section", section)
    print(config.items(section))

('section', 'conf')
[('tools.microsoft.msbuild:verbosity', 'Minimal'), ('pkg_pattern*:tools.microsoft.msbuild:verbosity', 'Quiet')]

[SSE4]

I'd prefer this to be decoupled from the cache folder.
otherwise, we introduce a vicious circle here: config has a path to the cache folder, but the config path is relative to the cache folder. it's super confusing.

[memsharded]

This is not new, the current config (and absolutely everything from the cache) is already loaded this way, because assuming it is in the cache folder, you need the cache folder to locate it. It cannot be decoupled, but maybe I don't understand the suggestion.

[SSE4]

I'd say it's not always a valid argument "it was always like that" when designing a brand new thing, especially if the approach is already known to be problematic.
first, it's logically inconsistent, as the configuration file doesn't belong to cache idiomatically (we are not caching it, we caching binaries and sources).
IMO it belongs to the config directory (which might be the same or might not be the same as cache).
second, if we want conan to configure to use a different cache folder, we can't, as config file is within cache folder, so cache folder needs to be known before loading the config, which obviously, introduces a vicious circle.

[memsharded]

It is not an argument of "it was always like that". It is maybe a slightly incorrect naming. Lets rename the cache_folder to home_folder which is what it actually is. Conan codebase is referring to the Conan cache and the Conan home folder equivalently, because 97% of the home folder is devoted to the package cache. Basically, the Conan home folder contains the settings.yml, the remotes.json, the conan.db, the profiles plus the recipes and packages cache.

If we agree with that, I am very fine with renaming the Conan codebase from "cache" => "home" when it applies. But the logic implemented in this new_config_path() is correct and in the right place.

[jgsogo]

We need something like this.

This works great for built-in tools and classes (strings resemble the import).

Are there any other approaches considered? pros and cons? What is the answer to #8266 (comment) (classes that are imported from other python files or python_requires)? only via "user.*"? In the global config are there different sections for host and build contexts?

[jgsogo]

About this one:

The tools.microsoft.MSBuild:verbosity is actually applied in 2 different helpers the CMake one that had a msbuild-verbosity argument and the MSBuild() one.

I think it is confusing given the current syntax, if the syntax were pkg-name:verbosity=minimal then I would agree that it should apply to all build-helpers in pkg-name, but being the syntax explicit as tools.microsoft.MSBuild:verbosity=Quiet I would apply it just to the tools.microsoft.MSBuild helper.

[jgsogo]

py2/py3 and standard parsing... if it is possible to implement it using standard parsers only for Python3 I wonder if we should implement this experimental feature only using Python3 and forget about people using Python2. IMO we can receive enough feedback from people using Python3 and in the future (Conan 2.0) we don't need to change the implementation again.

[solvingj]

RE: @uilianries

• I would consider all attributes and values as lower-case only. It avoids case-sensitive problems (package reference is a good example). e.g
  tools.microsoft.msbuild:vebosity = detailed

Good point, I'm not sure which one I favor, since I think we're now supporting upper-case things in CLI like class name for generators at the CLI.

• I would use .ini or .toml format for that new file, including as extension to enforce code editor highlight.

Agree with either.

• conan_conf.txt is temporary, so I would consider layout.toml or layout.ini, because we already have conan.conf as global file, so when someone says "configuration file", won't be confused by local or global file if they have different names.

Definitely not layout since that's already a kind of keyword for workspaces feature.

• How do I know if I should use a configuration or an recipe option? For instance, there are projects which offers options to control compiler flags, but looking for this feature, they should be a configuration (which can affect or not the binary result).

Good question, going to think about this one today.

• Here we are changing Conan code base, but we will need to increase it in the future, as we will receive more features requests. So, how can we extend it without creating a snowball? What alternative can we provide for users for extending their configurations globally? For instance:
  tools.makefile.gcc:verbosity = detailed
  We don't have such config right now, but let's say I want it, but for my company only. I could provide an external python file which contains such treatment. That python file could be something new, in a new folder, like ~/.conan/configs/enterprise.py, Or, a conan recipe which can used with python requires.

I think this is also a really good point. I believe it relates to the fundamental challenge which has been discussed in the past. That is, built-in generators and toolchains versus custom ones. It's difficult to understand the slightly different behaviors between the two, both for users, and for us developing features. In particular, the built-in ones are less extensible for users. Typically, yes, if users want something slightly different from a built-in generator, they actually have to re-implement the generator themselves, just to change that one thing, and even then, they don't get all the same capability (like, custom generators from build-requires can't be passed at the command-line, you have to use the new, experimental, generator as a config strategy which then lacks versioning). I don't think we have a silver bullet solution for this problem yet, only a few ideas and each of them have drawbacks.

[solvingj]

Also, from discussion, I personally do not expect this to affect what the CMake build helper generates when using the Visual Studio Generator at all:
tools.microsoft.msbuild:vebosity = detailed

After hearing @memsharded reasoning, I can now see there's a logical argument for it, but I still find it unintuitive. Instead, I would expect this to be a separate config, and I can definitely imagine cases where I would want different default values for each specified in my global configurations:
tools.cmake.msbuild:vebosity = detailed

[madebr]

Will it be possible by a (build) requirement to override/set a configuration value?

[memsharded]

Hi @madebr

Will it be possible by a (build) requirement to override/set a configuration value?

At the moment it is not intended. The configuration so far is for changing things from the user side (profiles, command line, conan.conf):

• The "msbuild" verbosity level
• The "minimum_required_version" (from conan.conf)

That is, more or less to change things from some default values. If it is something that a build_require could transmit to its consumers it is not configuration, but definition. For example, I don't want a build_require to define or change the verbosity level of my build, this is something that comes from the end user. But a build_require could define some environment variables, for example, that it is not a matter of I can use those or other values, if you are using that build_requires, the build_requires will define those things into the consumers.

This is more or less the current way of thinking, but we probably need to learn more, other use cases, keep working on the toolchains and specially in the environment-variables management in Conan, to really know if this will still hold, need to adapt, or even another mechanism is necessary.

[madebr]

Thanks for the reply @memsharded
I was asking because me (and @jgsogo?) were thinking that this could be a way to pass information from profile/build_requirement to some recipe.
See #8290 and #8312 for what I mean:

• changing default gnu triplet
• changing MesonToolchain default variables

[memsharded]

Reviewed:

• Did one tools.microsoft:msbuild_verbosity that will define both the CMake and MSBuild helpers. The thing is that there will be other configuration, for example vcvars_command() that has internal configuration like preference = get_env("CONAN_VS_INSTALLATION_PREFERENCE", list()). Those things are used in multiple places, like CMake and Meson. So it makes sense to define something like tools.microsoft.visual:preference=xxxx. But we don't want to need to define both tools.cmake.cmake:visual_preference=xxxx and tools.meson.mesontoolchain:visual_preference=xxxx, Now both helpers need to capture this conf, extract the value, and pass it all the way down to the vcvars call, which now can receive this thing both as an argument and as a conf. Which one has precedence? because still the tools.microsoft.visual:preference=xxxx can, and many times, will be there. All of that involves more code, more testing, a lot more documentation... One configuration should be defined in one way, not multiple. So lets generalize if necessary, now tools.microsoft:msbuild_verbosity does not belong to the MSBuild build helper, but it is a general tools.microsoft configuration.

• Forced all module names and all confs keys to be lowercase

• Renamed conan_conf.txt => conan.cfg

[memsharded]

@madebr

For those things, changing default gnu triplet, changing MesonToolchain default variables, seems a good fit when applied in a profile. This seems exactly the kind of configuration that we want toolchain, build helpers, etc. to expose to users, so users can always move forward and escape limitations of those helpers by explicitly providing information, like the tripled.

Having them propagated from an upstream requirement is a very different story. In the same way that such build_requires cannot define the settings nor the options of the consumer, at the moment it is not evident if the configuration should propagate in this direction too. I would need to understand better what are the advantages of putting that information in the build_requires instead of explicitly and directly in the profile, which seems to me more straightforward, explicit and less problematic.

[madebr]

@madebr

Having them propagated from an upstream requirement is a very different story. In the same way that such build_requires cannot define the settings nor the options of the consumer, at the moment it is not evident if the configuration should propagate in this direction too. I would need to understand better what are the advantages of putting that information in the build_requires instead of explicitly and directly in the profile, which seems to me more straightforward, explicit and less problematic.

I was thinking about the meson cross files (https://mesonbuild.com/Cross-compilation.html + https://mesonbuild.com/Machine-files.html).
Some properties are compiler dependent, which might be packaged by conan.
It's useful to keep some variables in the package.

Using configuration info in build_requirements would also allow defining compiler tool chains in package_info.
When cross building, if this info is set for the build_profile and the host_profile, then this info can be used to automatically set CC/CXX (e.g. self.conf["compiler.cc"]/self.conf["compiler.cxx"]) CC_FOR_BUILD/CXX_FOR_BUILD (e.g. self.build_conf["compiler.cc"]/self.build_conf["compiler.cxx"]). (CC sets hosts compiler, CXX set build compiler: see https://mesonbuild.com/howtox.html#set-compiler).

I'm putting these things here, because it looks like conf will only allow thightly defined configuration variables, where-as cpp_info/user_info/env_info is about general libraries/strings/environment.

My request for always enabling overrides stems from meson machine entries.
Like CMAKE_SYSTEM_NAME, there is no standard for naming things.

My apologies if this is getting off topic.

[memsharded]

@madebr

Your example is relevant:

Using configuration info in build_requirements would also allow defining compiler tool chains in package_info.
When cross building, if this info is set for the build_profile and the host_profile, then this info can be used to automatically set CC/CXX (e.g. self.conf["compiler.cc"]/self.conf["compiler.cxx"]) CC_FOR_BUILD/CXX_FOR_BUILD (e.g. self.build_conf["compiler.cc"]/self.build_conf["compiler.cxx"]). (CC sets hosts compiler, CXX set build compiler: see https://mesonbuild.com/howtox.html#set-compiler).

This are explicitly env-vars in Meson, aren't they? If they are env-vars in Meson, then I don't see making configuration for them, why would be better to have configuration instead of directly the env-vars?

The idea is to be able to pass information that it is not structured/defined by the build systems, compilers or tools in general by env-vars, and avoid creating many new invented env-vars for things that are finally translated to a command line switch, or to certain logic inside a Conan tool or helper, because env-vars are dirtier for this purpose, less structured and very global (can be written in other recipes).

Again, it is just a preliminary idea, we need to keep discovering use cases, learning and evolving this.

My apologies if this is getting off topic.

It is fine, thanks very much for the feedback, the intent of this PR is to start rolling the feature to gather this feedback and use cases.

[memsharded]

Responding to

What is the answer to #8266 (comment) (classes that are imported from other python files or python_requires)? only via "user.*"?

Yes, if users provide their own helpers via python_requires they have 2 options:

• Creating their own custom configurations. Those should go to user.*.
• User python_requires could use existing configuration, and read, for example tools.microsoft:msbuild_verbosity and act accordingly. This might be a common use case for users that are creating custom variations of the Conan toolchains and helpers.

In the global config are there different sections for host and build contexts?

Not at the moment. The idea is that the global conan.cfg file contains only global information. For specific host and build, it makes sense that it goes to the profile:host and profile:build. Having the possibility of adding the host/build in the global conan.cfg would introduce another possibility of adding there core.* configurations, which would be an error. So the conan.cfg now would need at least 3 sections, a global one, a host one and a build one. I think it is better to keep it as simple as possible at the moment.