Build multiple images with different sdkconfigs (IDFGH-3735)

[galah92]

Using ESP-IDF v4.2.

My use case is an app with two different images - one is WiFi-based and the other one is BLE-based. I'd like to share components between the two.

I opted for the bare idf_build_process() as shown in idf_as_lib example, but not I now face the problem the incorporating two sdkconfig files, one for each image, where the first one has WiFi custom config and BLE disabled and the second one does the opposite.

By calling idf_build_process() twice I got the following:

CMake Error at C:/Git/esp-idf-v4.2/tools/cmake/kconfig.cmake:263 (add_custom_target):
  add_custom_target cannot create target "menuconfig" because another target
  with the same name already exists.  The existing target is a custom target
  created in source directory "C:/Git/MyProject".  See
  documentation for policy CMP0002 for more details.
Call Stack (most recent call first):
  C:/Git/esp-idf-v4.2/tools/cmake/build.cmake:442 (__kconfig_generate_config)
  CMakeLists.txt:213 (idf_build_process)

It looks like the second call is trying to create a target already created.
What can be done?

[projectgus]

Hi @galah92 ,

In general, we don't really supporting building more than one instance of IDF in a single invocation of CMake. There are quite a few global properties that won't play nicely together.

For the situation you describe, would recommend if possible that you have two project directories, one for the WiFi app and one for the BLE app. Make a third directory like extra_components to place the common components in and pass this common directory to both projects by setting the EXTRA_COMPONENT_DIRS variable in each top-level project CMakeLists.txt file.

If you do it this way then you can choose to use the "normal" IDF build system process or the more CMake-style process that uses idf_build_process().

You will have to run idf.py (or cmake --build) twice, once for each app. However you could make a small wrapper script to do it for both projects in one pass.

Does that work for your use case? If not, can you please give some more details about the requirements you have?

[galah92]

Hi @projectgus, thank you for your comment.

Let me elaborate regarding my situation:

• I have a repository with numerous applications for multiple MCUs, ESP32 is one of them. All the applications are built using CMake, with multiple common components shared between them.
• I can currently build and test all apps using cmake --build build_dir --target all && cmake --build build_dir --target test.
• The current CMake state of ESP-IDF doesn't fit into that - I can't easily integrate multiple ESP32 based projects.
• I don't want to clutter my build system with wrappers around CMake, which is idf.py in that case.

Regarding the ESP32-based apps:

• I've got two images intended to two different partitions in flash, were each image can reset to the other image as part of some business-logic. This means I have to enforce the same partition table for both, for example.
• I'd like to be able to build them both using single CMake command, because they both share a lot of code and I need an easy way to make sure I'm not "breaking" one while working on another.
• Flashing the images is done manually using esptool by the way, I don't need that kind of support from the build tool (CMake).

It seems like there are multiple things I need to work differently in ESP-IDF in order to support my case:

• I need a way to better control the build of the bootloader and partition table, which are shared between the images.
• I need a way to control sdkconfig parameters using CMake variables. I actually don't understand the whole usage of the sdkconfig. I think it should've been abandoned in the transition from make to CMake, as we can use CMake's cache variables and option variables for the same purpose.

[galah92]

Another bug I found when trying to work with the idf.py script:
Some of my components are fetched via CMake's FetchContent feature. Updating these components (updating the relevant git tag to fetch, for example) doesn't trigger a new build for the relevant parts when doing idf.py build. This is probably due to the usage of cmake -E external commands to build all the relevant targets (bootloader, partition table and actual image), and this is one of the many reasons why is it discourage.

[atanisoft]

@galah92 I worked around this in my build script here by bypassing idf.py and using the steps it executes for generating the sdkconfig and cmake bits. I have this in place to build two different configurations in parallel using GitHub Actions.

[galah92]

@atanisoft Thanks. I'm still trying to figure out a way to build multiple binaries using single CMake invocation though. Your script seems to build specific targets using different CMake configurations (different directories). Am I correct?

[atanisoft]

I'm still trying to figure out a way to build multiple binaries using single CMake invocation though.

With how the CMake scripts are configured I don't think this will be feasible anytime soon unfortunately.

Your script seems to build specific targets using different CMake configurations

The script is the equivalent of "idf.py build size" when the build directory and sdkconfig file does not exist. The GitHub Actions run in different containers so the output directory is "build" in both configurations that I'm currently using (sdkconfig.defaults and sdkconfig.defaults.pcb).

[galah92]

I'm still trying to figure out a way to build multiple binaries using single CMake invocation though.

With how the CMake scripts are configured I don't think this will be feasible anytime soon unfortunately.

Your script seems to build specific targets using different CMake configurations

The script is the equivalent of "idf.py build size" when the build directory and sdkconfig file does not exist. The GitHub Actions run in different containers so the output directory is "build" in both configurations that I'm currently using (sdkconfig.defaults and sdkconfig.defaults.pcb).

Now I get it, thanks. Unfortunately it looks like I'll have to patch things in order to have a better control over the compilation of the bootloader, partition table, and all the rest.

It just feels awkward that the way IDF wraps CMake actually disabled all the great features of it.

[projectgus]

Hi @galah92,

Thanks for explaining your use case and thanks @atanisoft for showing your build system approach.

I can confirm that at the moment there's no way for this to work all in one CMake invocation with the ESP-IDF build system. The closest thing that works now is probably to use ExternalProject to invoke the ESP-IDF build system as a separate CMake instance, once per ESP-IDF project to build. Which is obviously still hacky compared to what you're asking for.

We'll keep the request open to track this requirement and when we're next revisiting the build system architecture then we will look at it again and see if we can accommodate it. Unfortunately this isn't something I can give you an ETA for.

If you end up with any patches for ESP-IDF that maintain compatibility with the way it works now then please consider sending Pull Requests.

It just feels awkward that the way IDF wraps CMake actually disabled all the great features of it.

This is an understandable sentiment, especially given you are clearly quite familiar with CMake.

The reason for this is can be summed up as the classic "historical reasons". When we introduced the CMake build system we already had a lot of ESP-IDF users familiar with the existing GNU Make build system, and we needed a path forward that was familiar for them. The GNU Make build system we already had includes some complex things which do not fit neatly into "the CMake way" of what a build system is (for example running flashing or serial tools, as you noted above).

Could we have done better? Undoubtedly. Could we have switched absolutely everything over to "the CMake way" without causing a lot of anguish for our existing users? I think probably not.

We are interested in continuing to move closer to "the CMake way" where it's possible, though.

Some of my components are fetched via CMake's FetchContent feature. Updating these components (updating the relevant git tag to fetch, for example) doesn't trigger a new build for the relevant parts when doing idf.py build. This is probably due to the usage of cmake -E external commands to build all the relevant targets (bootloader, partition table and actual image), and this is one of the many reasons why is it discourage.

I don't fully understand the issue here, but this is something we can look at separately. Could you open a github issue with a few more details, please?

[KaiDroste]

Hi,
I think I have there a workaround for this problem.

One only thing wich I wish for the future is to select the right sdkconfig from the commandline for example : idf.py menuconfig -c sdkconfig.aio

I have written a skript in powershell wich renames the sdkconfig and runs then the build script. Here is my powershell script:

Write-Host -ForegroundColor Yellow "Compile for Analog board"
Rename-Item .\sdkconfig.aio .\sdkconfig
idf.py -B .\aio build 
Rename-Item .\sdkconfig .\sdkconfig.aio

One more thing what I noticed is that if I rename the sdkconfig file and do not have a seperated build folder for this file. I need to run the menuconfig that to have idf.py to rebuild the project.

[galah92]

@projectgus Thank you for the details response, I really appreciate it. First I'd like to the my words back - I understand most of the constraints you're facing with, especially regarding backward compatibility and "historical reasons". I'm glad that you're open towards the idea of moving closet to "the CMake way".

I think about it more and more these days, and I started realizing that there's probably no good reason to maintain a sdkconfig file at all. CMake can easily handle the job of parameterized builds. How open are you to transition to bare CMake in that direction? I started forking and playing with it, anyway.

I'd like to contribute as much as possible to the build system procedure, what changes are acceptable and what aren't, in your views?

@waterfox207 Thanks! I'm trying to to something like that in bare CMake, without using idf.py.

[projectgus]

Hi @waterfox207,

This might be a command line option we can support, the old GNU Make build system used to allow overriding this with an environment variable. I think it could be made into a cache variable in CMake to the same effect. Will look into it, I'm afraid I can't promise an ETA though - but your workaround looks good for now.

Hi @galah92,

I think about it more and more these days, and I started realizing that there's probably no good reason to maintain a sdkconfig file at all. CMake can easily handle the job of parameterized builds.

To check I understand: The proposed change is to store the sdkconfig settings in CMake cache variables of CMakeCache.txt for all of the current sdkconfig configuration values. Keeping the same KConfig files, menuconfig editor, etc?

I think we looked into this when we were choosing the build system to move to. It may be my lack of CMake understanding, but it boiled down to that CMakeCache.txt is conceptually a configuration file for the build rather than a configuration file for the project.

• As well as the cache variables, it contains a number of system-specific build configuration such as tool paths.
• It's not common for a project to commit the CMakeCache.txt file to source control. Committing the file and then sharing it can lead to other problems as the system-specific settings do not always translate to other users/computers/host OSes/etc.
• We can't move 100% to cache variables as cmake-gui can't give the same features that we have in menuconfig, so the Kconfig subsystem would have to be extended to read/write the CMakeCache.txt format (this is possible, it's just additional complexity to implement and maintain compared to the current format which is implemented out of the box.) Also I don't know if this format is guaranteed to stay stable over different CMake versions.

Let me know if I'm misunderstanding anything here.

I'd like to contribute as much as possible to the build system procedure, what changes are acceptable and what aren't, in your views?

Thanks, that's great to hear! We have a policy of not making breaking changes in minor versions (and we try to avoid them in major versions as well.)

So the hard requirement for a build system change is not to break anything which is documented in the current version, or to break any "public" APIs (these are harder to define in terms of CMake functions but basically anything which is in the docs or any example is probably "public".)

Aside from this, we're open to anything which is useful or which simplifies the design or the implementation. If you have an idea that's going to be a large change and need a lot of work then it's probably best to discuss it first and we can give you some feedback.

I should mention that we would consider making breaking changes in the next major release. The only difficulty is that we don't have any ETA for that yet - we won't plan a major version bump until we absolutely have to. But if you have other requests like this one that would be breaking changes in the build system, feel free to make them and we'll keep them in mind.

[galah92]

Hi @projectgus,

I'm raising this thread again since I started to port my build to idf_build_process() with esp-idf-v4.2-beta (this is the only release that allows customizing the partition table - #5053).

First I'd like to address your previous comments regarding sdkconfig-vs-cmake-variables:

• I think the whole point is to more to build-configuration instead of project-configuration, isn't it? It'll make the build system more flexible.
• I don't think a project should commit the CMakeCache.txt file, of course not. Rather I would've translated all KConfig to CMake variables with some default values. Then, as a project owner, I could either set(CONFIG_VAR ..) to override the default behavior, or just leave it as is.
• I don't use CMake-GUI, but I think it's capable of the same things menuconfig can.

Also, could you please elaborate regarding the build system changes which are planned for the next major release?

Now, coming to my point at last.. leaving off the requirement for different sdkconfig - I need a way to build multiple apps using the same build system. The incentive is to create a regular multi-project CMake structure, and mostly to allow sharing multiple components between different apps, where they compile once.
Currently, after including idf.cmake it looks like add_executable() creates the elf and only idf_build_executable() creates the actual image. Problem is, multiple calls to idf_build_executable() ends up creating only the image of the last call.

Is it possible to add a mechanism which will allow me to compile multiple application sharing multiple libraries, where everything compile only once? My list of applications grows large and, among others, I'd like to minimize the total compilation time and don't want to compile shared libraries multiple times.

Thank you

---

EDIT: I'd like to add - my ultimate goal is creating a unified build-system (relying on CMake) to support building multiple apps, sharing multiple common libraries, for different platforms. Most of my other platforms play well with normal CMake - I just specify as input the toolchain file and it all just works. IDF is the weird one here - I'm OK with having to call specific CMake functions for IDF, but the current situation makes it almost impossible to tinker with the build system. It's just not flexible enough.
Specifically, I'm currently struggling with idf_build_executable(). I don't quite understand why this call is actually needed:

• It makes the executable target depend on linker script generation, but it's that always true? If at some point all my executables will link to idf::esp32 for example, is it possible to "lower" the dependency to idf::esp32.
• It invokes elf2image - that could be added as a POST_BUILD step - thus making it possible to call idf_build_executable() multiple times.

[AshUK]

+1 would like to see support for multiple SDK configs.

In our case, we use a CI/CD platform to create and distribute our builds. When we build for release we disable the logging output, UART downloader and enable flash encryption. The first thing a developer has to do every time they work on a feature is turn off all the security features or risk disabling their local development board. Later we then have to run a bunch of checks on the CI to ensure that the developer didn't commit an insecure change. If however we used development config by default instead and just manually edit the sdkconfig during the CI production build. The image will always have a dirty version tag. We can get around this by having auto commits that prevent insecure builds but this seems like a hack and adds complexity to the build.

Another reason we would like to see multiple sdkconfig support which is similar to @galah92 is we have multiple builds (same hardware and firmware) but whereby the only difference is the Bluetooth advertising name, the device hostname and a GPIO configuration.

TLDR: multiple sdkconfigs would make our lives easier.

[igrr]

@AshUK would something like this work for you?

• Create sdkconfig.defaults — contains the settings you use for development (only those which are different from the default ones)
• Create sdkconfig.prod — contains the additional settings you need for production
• Add sdkconfig to .gitignore
• Developer use case: run idf.py build flash monitor as usual.
• CI use case:
  1. Verify "developer build" (artifacts are in build_dev directory)

     idf.py -B build_dev -DSDKCONFIG=build_dev/sdkconfig build
     

  2. Make production build (artifacts are in build_prod directory):

     idf.py -B build_prod -DSDKCONFIG=build_prod/sdkconfig -DSDKCONFIG_DEFAULTS="sdkconfig.defaults;sdkconfig.prod" build
     

Edit: forgot to address the 2nd part of your question. You can extend this to use more sdkconfig.defaults-like files. The list of sdkconfig.defaults files can also be set in your project CMakeLists.txt file using SDKCONFIG_DEFAULTS variable, so you can add some additional logic in project CMakeLists.txt file to determine which sdkconfig.defaults files need to be included, based e.g. on environment variables.

The only limitation is that IDF libraries will still be built multiple times, even if differences in sdkconfig are very small. However if you install ccache and set IDF_CCACHE_ENABLE=1 in the environment, then ccache will usually do quite a good job at reusing some of the built object files between different configuration.

[atanisoft]

@igrr regarding the option you posted above for "production" and having multiple sdkconfig.defaults files listed in the commandline. What is the expected behavior for resolving duplicate sdkconfig entries in the defaults files? Would it be based on the ordering of the defaults files in the commandline arguments?

[igrr]

When sdkconfig file doesn't exist, all sdkconfig.defaults-like files are concatenated to obtain the initial sdkconfig file. For each file listed in SDKCONFIG_DEFAULTS variable, the build system also adds <filename>.<IDF_TARGET> file, if it exists (see docs).

That is, in the example above, the files would be combined in this order, assuming IDF_TARGET=esp32:

• sdkconfig.defaults
• sdkconfig.prod
• sdkconfig.defaults.esp32
• sdkconfig.prod.esp32

The target-specific files are optional.

Options listed later in sdkconfig override the ones listed earlier. So you can even explicitly set some option to =n in your sdkconfig.defaults file and then change it to =y in sdkconfig.prod, if you like.

Edit: I'm sorry, I actually made a mistake in the above order when posting this. Here's the link to CMake code for this part. We'll update the docs to make the order clear.

esp-idf/tools/cmake/kconfig.cmake

Lines 161 to 173 in cf457d4

	if(sdkconfig_defaults)
	foreach(sdkconfig_default ${sdkconfig_defaults})
	list(APPEND defaults_arg --defaults "${sdkconfig_default}")
	endforeach()
	endif()
	if(sdkconfig_defaults)
	foreach(sdkconfig_default ${sdkconfig_defaults})
	if(EXISTS "${sdkconfig_default}.${idf_target}")
	list(APPEND defaults_arg --defaults "${sdkconfig_default}.${idf_target}")
	endif()
	endforeach()
	endif()

[AshUK]

@AshUK would something like this work for you?

 ```
 idf.py -B build_dev -DSDKCONFIG=build_dev/sdkconfig build
 ```

Edit: forgot to address the 2nd part of your question. You can extend this to use more sdkconfig.defaults-like files. The list of sdkconfig.defaults files can also be set in your project CMakeLists.txt file using SDKCONFIG_DEFAULTS variable, so you can add some additional logic in project CMakeLists.txt file to determine which sdkconfig.defaults files need to be included, based e.g. on environment variables.

The only limitation is that IDF libraries will still be built multiple times, even if differences in sdkconfig are very small. However if you install ccache and set IDF_CCACHE_ENABLE=1 in the environment, then ccache will usually do quite a good job at reusing some of the built object files between different configuration.

Thank you very much for this information. I can confirm this is exactly what we were looking, not sure how we missed it in the docs. I have already updated our CI tools to use this method of composable sdkconfigs and is working as expected. Brilliant as no more superfluous dirty tags on builds.

Not sure how much pain this cause when updating to a new IDF version as it will be difficult to see what has changed.

[igrr]

Not sure how much pain this cause when updating to a new IDF version as it will be difficult to see what has changed.

Do you mean the difficulty would be because sdkconfig files would be different between IDF versions?

I think this should be fine, because sdkconfig option names are backwards compatible. If we are renaming some sdkconfig option, for example because it is moved between components, we add it to sdkconfig.rename file in IDF which tells sdkconfig generation tools to automatically rename the legacy name to the new one. So if your sdkconfig.defaults file uses some option name and the name is changed in newer version of IDF, there shouldn't be an issue.

The other potential problem is related to relying on default option values provided by IDF. It is recommended to keep sdkconfig.defaults-like files minimal, only including the options there which you actually want to change from their default values. On the flip side, this means that if an IDF update changes the default value, this may affect your application. We try to mention any significant changes in default values of Kconfig options in release notes, and we plan to add an internal process to verify the changes in default values before a release. So while I can't say these changes will never cause problems, this is something we will try to make visible in release notes.

Thanks for your feedback on this approach. We will add an example to IDF to demonstrate this mechanism, to make it easier to find this solution for other users in the future.

[igrr]

By the way, here is a small improvement to the approach I suggested above. Instead of passing -DSDKCONFIG=build_prod/sdkconfig to idf.py, you can override sdkconfig file location once in your project CMakeLists.txt file:

cmake_minimum_required(VERSION 3.5)
set(SDKCONFIG "${CMAKE_BINARY_DIR}/sdkconfig")
include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(your_project)

This way the generated sdkconfig file will never be placed into the project root directory and will always stay in the build directory, making it possible for developers to have side-by-side builds for different products/configurations.

[atanisoft]

If we are renaming some sdkconfig option, for example because it is moved between components, we add it to sdkconfig.rename file in IDF which tells sdkconfig generation tools to automatically rename the legacy name to the new one.

The build log will also contain all renames as warnings near the top when generating the "final" sdkconfig / sdkconfig.h files.

On the flip side, this means that if an IDF update changes the default value, this may affect your application. We try to mention any significant changes in default values of Kconfig options in release notes, and we plan to add an internal process to verify the changes in default values before a release.

@AshUK With this note above I'd suggest adding a mechanism to your CI tool where it diffs the generated sdkconfig file from the current build to the previous and flag any options which are not expected to change.

[ghost]

I'm running into this specific issue too, but I can do just with the ability to use different per-board sdkconfig.defaults in Platform IO.

[igrr]

I'll close this issue now since the example of building different configs was added in 5730711.