-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CMake: From USCRPL/mbed-cmake to Mbed OS first-party CMake #13981
Comments
I haven't had a chance to poke around the official CMake scripts that much, but I do agree that building the linker script multiple times seems like the wrong way to do it since it doesn't change. It would be better to just generate it once and then attach it as part of a target's link flags (how mbed-cmake does it). Also, it definitely doesn't seem right if the entire source code is getting built once for each target. My understanding is that the mbed source code is supposed to get built as an object library, then the objects are supposed to get attached using TARGET_SOURCES to an interface library. But it sounds like maybe the source files themselves got added as the interface sources, rather than the object files. |
@ladislas thank you for raising this issue.Please take a look at the following comments: We cannot automatically identify a release based on the version of Mbed OS that you have provided. |
Thank you for raising this detailed GitHub issue. I am now notifying our internal issue triagers. |
@multiplemonomials it was since e7c0d93 that changed it from Would |
Yeah that commit is... not right, or at least it's not correct to the design we originally planned (I worked with Martin and Hugues to plan out how the object and interface libraries would be used). The interface libraries do need to be there, but there also need to be object libraries as well. The object libraries would be used to compile the sources, then the compiled objects from those libraries should get added via target_sources to the interface libraries. |
Thanks for the feedback, we will review and respond to all issues/questions. |
@ladislas can we convert example blinky to be able to reproduce the issue with spikes? I've checked your template repository, it does not reference mbedtools, so I might need an assistance to reproduce it there (how to).
I would say it should work as CMake libraries should be reusable. We are currently looking at testing and it might be the same use case.
We initially worked only on object libraries but we faced cmake issues with object libraries (we should be able to find PRs/commits) and converted to interface libraries. We exposed the main libraries mbed-os and mbed-baremetal as interface as well - it has some drawbacks as mentioned here, we should be able to address them. The branch I shared above was a concept we had previously.
Consider ld file being a template that is changed based on app needs (sections could be adjusted, an example is bootloader). Therefore we preprocess linker scripts. If you change an app, linker script must be regenerated - reusing the same template. We always wanted to have one .ld template and fill it in. |
@0xc0170 yes, I'll work on a PR! I'll also push to my template on a specific branch for testing. I'll also test your branch and let you know. And thanks for the explanation on the linker file. 👍 |
Okay, I've got time to take a look into what's going on here.
This is the same issue that we talked about at our meeting, and it seems that it still hasn't been fixed. Basically, the compiler flags that CMake is using to do all of its internal tests are extremely messed up, and you guys just papered over the issue with: set(CMAKE_C_COMPILER_WORKS TRUE)
set(CMAKE_CXX_COMPILER_WORKS TRUE) This is a really bad idea, because it means that all sorts of things will be subtly screwed up later on, since CMake cannot successfully run compilation or linking tests. |
@ladislas - COMMAND "arm-none-eabi-cpp" ${_linker_preprocess_definitions} -x assembler-with-cpp -E -Wp,-P
+ COMMAND ${CMAKE_CXX_COMPILER} ${_linker_preprocess_definitions} -x assembler-with-cpp -E -Wp,-P The existing code requires |
Update: I understand a little better why the current method was used (compiling all mbed sources for each target). The issue is that you have a set of base Mbed sources, mbed-core. You also have a number of extra modules, like mbed-rtos. However, these extra modules can add extra compile definitions (such as IMO, this is somewhat bad code design to have things tightly coupled like this, and it's hard to fix at the build system level. However, I also don't think it's a good solution to require rebuilding all mbed sources for each target. For now, I'm going to try at least making mbed-baremetal and mbed-rtos separate libraries to fix that dependency, but I'm not sure how many extra modules have this behavior. |
OK! I think I figured out a method that works to fix this. I realize that the solution I told you guys originally doesn't actually work given the constraints above. However, I figured out something that should work for what you need. For the main OS targets, I used this strategy:
For extra modules, I used this strategy:
Note that this process makes some assumptions:
Let me know what you think about this plan -- I'm hoping that it can make life easier for us users. But if it seems totally unworkable, let me know as I have another idea that might also be feasible. |
Thanks @multiplemonomials , we will review but most likely the next week (some of us are out of office in the next days) |
I run initial compiler checks to see what flags and files get into the app. I'll look at the structure the next week. Compiling K64F for GCC ARM shows startup is duplicated:
ARMClang:
@ladislas if you can share the multiple target example, or we shall create our own to test the fix? |
Oh whoops, there was an issue where the wrong linker script path was passed for GCC_ARM. I pushed a new commit to #13987 to fix it. I also found a variable name conflict with the profile scripts that was probably causing the arm compiler error, pushed a fix for that too. |
@0xc0170 yes, I'm working on a simple example, I'll share it later today. |
Alright, I've setup a working example of our project structure, available here:
Instructions are available here: If you want to compare the same template with USCRPL/mbed-cmake, you can follow the instructions in
|
OK @ladislas I tested your example, and my cmake-object-libs branch that I linked earlier is able to compile it correctly with no other changes and only 247 files compiled. |
@multiplemonomials that's great news, I've just tested it myself and I can confirm it compiles both the main src and the spike without any problems :) The number of files to compile (spike + lib + src) is also much lower than it was previously:
|
@0xc0170 @hugueskamba Have you had a chance to look at my cmake-object-libs branch? I think it's an important change to make but it might be too much of a project for me to do by myself, I was hoping you could take the ideas from it and make a PR that implements it. |
I was looking for the reference this week and could not find it in PRs, it's on a branch! I'll create a task for the next sprint to look at it.
got it now! |
@multiplemonomials Any chance you could rebase your branch on top of master? |
I stumbled upon this thread when I noticed that I cannot add compiler options only for my code without adding them to mbed-os. AFAIU, interface libraries are compiled with the options of the target, i.e. the compiler options apply to all linked interface libraries. Object libraries would solve this issue too. |
@0xc0170 where are we on that matter? :) |
The state stands as it was, it is on hold for now (#13981 (comment)). We most likely can resume the work in Q3. I am happy to help meanwhile if there are any contributions for the feature branch. I'll rebase the feature branch tomorrow to bring it up to date after unittests changes. |
@boraozgen Note that my patch I submitted for #14274 will provide a workaround for that issue, and you may even just be able to drop mbed_create_distro.cmake into your project and go without changing Mbed source files in the latest version. |
We hope to get #15126 into master and close this issue. |
We closed this issue because it has been inactive for quite some time and we believe it to be low priority. If you think that the priority should be higher, then please reopen with your justification for increasing the priority. |
#15126 has just been merged, we'll start moving to vanilla m bed cmake next week. I'll keep you posted here. Maybe it's good to reopen until the transition is over. |
The discussion can be ongoing here. If there are any other issues, you can report via a new bug report. |
We are currently trying to move from USCRPL/mbed-cmake to Mbed's "vanilla" CMake. The process is not as straight forward as I hoped it would be, compared to setting up USCRPL/mbed-cmake in the first place. I'll report the issues here and open new ones if needed:
Conclusion on 2021/12/31: After spending 48h hours on this, I was hoping the transition would be easier. I'm a bit disappointed by the fact that compilation and linking are so long. The modularity CMake should bring is not there. This is mostly due by the use of CMake target naming could be improved. Right now, I'm not sure what to do. Should I keep fighting the tools or keep our current setup using USCRPL/mbed-cmake? Our product will be on the market in one month, and as much as I would like to help improve mbed, I'm not sure we have the time and resources right now. Again, a clear roadmap about what to expect in the coming month would really help us. |
Wait, so create_distro() didn't work to reduce the compile time for building multiple targets? Is it because there are more source files than the corresponding mbed-cmake project? Or is it that you had to make multiple distros for things with different functionality? |
It does reduce compared to not using it, but not using it would be shooting yourself in the foot so I'm not comparing to that. I'm comparing to USCRPL/mbed-cmake. We only have one "distro" with everything we need. Compilation time is longer because there are more files, files we were able to exclude with .mbedignore before. What worries me is the final compilation and link time it takes to build the final executables. With USCRPL/mbed-cmake, mbed was compiling, our libs/drivers were compiling, then all the executable targets were compiling and then linking and it a was all one after the other, with memap output, without any wait time in between. Now it just hangs for several seconds or tens of seconds and I don't know why. Also I've noticed that they are like hundreds of flags passed for compiling each file. I don't know if it has an influence but the output is really hard to parse and read because of that. |
Hmm, it's possible the additional compilation time is because mbed-cmake compiles all of mbed-os into a .a STATIC library, while the official build system uses an OBJECTlibrary, so the linker has to read hundreds of individual .o files every time it links something. The reason for this is that STATIC libraries are not usable with Arm Compiler since Arm Compiler lacks the Also yeah, I was bothered too by the sheer number of #define flags that Mbed OS uses when I made mbed-cmake, which is why I had the Python script dump them all into a .h file which then gets force-included to everything with the |
Thanks @multiplemonomials for the ideas! i've tried the STATIC library changing this line mbed-os/tools/cmake/create_distro.cmake Line 35 in 149c5b1
- add_library(${NAME} OBJECT)
+ add_library(${NAME} STATIC) But that produces the following error: [1137/1166] Linking CXX executable spikes/mbed_blinky/spike_mbed_blinky.elf
FAILED: spikes/mbed_blinky/spike_mbed_blinky.elf
: && /usr/local/bin/arm-none-eabi-g++ -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fmessage-length=0 -fno-exceptions -ffunction-sections -fdata-sections -funsigned-char -fomit-frame-pointer -g3 -mthumb -mfpu=fpv5-sp-d16 -mfloat-abi=softfp -mcpu=cortex-m7 -Wl,--start-group -lstdc++ -lsupc++ -lm -lc -lgcc -lnosys -Wl,--end-group -specs=nosys.specs -Wl,--cref -Wl,-Map=/Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf.map -Wl,--gc-sections -Wl,--wrap,main -Wl,--wrap,_malloc_r -Wl,--wrap,_free_r -Wl,--wrap,_realloc_r -Wl,--wrap,_memalign_r -Wl,--wrap,_calloc_r -Wl,--wrap,exit -Wl,--wrap,atexit -Wl,-n -T /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/extern/mbed-os/targets/TARGET_STM/TARGET_STM32F7/TARGET_STM32F769xI/mbed-stm32f769xi.link_script.ld spikes/mbed_blinky/CMakeFiles/spike_mbed_blinky.dir/main.cpp.obj -o spikes/mbed_blinky/spike_mbed_blinky.elf libs/Utils/libUtils.a libs/LogKit/libLogKit.a libs/HelloWorld/libHelloWorld.a libs/CriticalSection/libCriticalSection.a libmbed-os-static.a libs/ContainerKit/libContainerKit.a libs/LogKit/libLogKit.a libs/ContainerKit/libContainerKit.a libs/Utils/libUtils.a libs/HelloWorld/libHelloWorld.a libs/CriticalSection/libCriticalSection.a libmbed-os-static.a && cd /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky && arm-none-eabi-objcopy -O binary /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.bin && /usr/local/Cellar/cmake/3.22.1/bin/cmake -E echo "-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.bin" && arm-none-eabi-objcopy -O ihex /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.hex && /usr/local/Cellar/cmake/3.22.1/bin/cmake -E echo "-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.hex" && cd /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky && /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 /Users/ladislas/dev/leka/LekaOS/extern/mbed-os/tools/memap.py -t GCC_ARM /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/mbed_blinky/spike_mbed_blinky.elf.map
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/softfp/libc.a(lib_a-fflush.o): in function `__sflush_r':
fflush.c:(.text.__sflush_r+0x94): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/softfp/libc.a(lib_a-malloc.o): in function `free':
malloc.c:(.text.free+0x6): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: /usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/lib/thumb/v7e-m+fp/softfp/libc.a(lib_a-fclose.o): in function `_fclose_r':
fclose.c:(.text._fclose_r+0x50): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: fclose.c:(.text._fclose_r+0x5e): undefined reference to `__wrap__free_r'
/usr/local/Cellar/arm-gcc-bin@10/10.3-2021.10/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld: fclose.c:(.text._fclose_r+0xbe): undefined reference to `__wrap__free_r'
collect2: error: ld returned 1 exit status Any ideas on how to fix?
Can this be made "toolchain" dependent? If we use Arm Compiler, we call If so, I can make a PR. |
@multiplemonomials I've tried fixing it by adding the I'm far from an expert with CMake and compiler settings so I might do something very wrong! Do not hesitate to state the obvious things I might have missed. :) |
Hmm, that is the right compiler option but you want to use it only when linking other things to the distro. What I did in mbed-cmake was create a wrapper target for mbed OS. For example, if you had target_link_libraries(mbed_for_my_app_wrapped INTERFACE
-Wl,--whole-archive
mbed_for_my_app
-Wl,--no-whole-archive) Then, executables will link to I will say, I'm not totally sure how this caused a crash though! I've never seen anything like that. |
@multiplemonomials so the wrapper worked! Thanks a lot! It hangs on the first executable but then the rest is quite okay. But the output doesn't look good, I've got this: [1163/1166] Linking CXX executable spikes/lk_file_manager/spike_lk_file_manager.elf
-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/lk_file_manager/spike_lk_file_manager.bin
-- built: /Users/ladislas/dev/leka/LekaOS/_build/LEKA_V1_2_DEV/spikes/lk_file_manager/spike_lk_file_manager.hex
Unknown object name found in GCC map file: libmbed-os-static-raw.a(startup_stm32f769xx.S.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(irq_cm4f.S.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_delay.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_delay.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(rtx_evr.c.obj)
[...]
Unknown object name found in GCC map file: libmbed-os-static-raw.a(us_ticker.c.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(QSPIFBlockDevice.cpp.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(BLEInstanceBaseImpl.cpp.obj)
Unknown object name found in GCC map file: libmbed-os-static-raw.a(GattServerImpl.cpp.obj)
| Module | .text | .data | .bss |
|-----------------|---------------|-------------|-------------|
| [fill] | 116(+116) | 8(+8) | 24(+24) |
| [lib]/CoreHTS.a | 1520(+1520) | 0(+0) | 0(+0) |
| [lib]/CoreI2C.a | 52(+52) | 0(+0) | 0(+0) |
| [lib]/Utils.a | 140(+140) | 0(+0) | 0(+0) |
| [lib]/c.a | 25296(+25296) | 2472(+2472) | 58(+58) |
| [lib]/gcc.a | 3116(+3116) | 0(+0) | 0(+0) |
| [lib]/misc | 188(+188) | 4(+4) | 28(+28) |
| [lib]/nosys.a | 32(+32) | 0(+0) | 0(+0) |
| [misc] | 29708(+29708) | 444(+444) | 8114(+8114) |
| main.cpp.obj | 88(+88) | 0(+0) | 0(+0) |
| Subtotals | 60256(+60256) | 2928(+2928) | 8224(+8224) |
Total Static RAM memory (data + bss): 11152(+11152) bytes
Total Flash memory (text + data): 63184(+63184) bytes for hundreds of lines for each executable. Also everything is in |
Yeah, that happens, I get that issue with mbed-cmake too. I've been meaning to dive into memap.py and see why it can't handle objects inside static libraries. |
Yes, I figured it was a memap.py issue as well. is there a way to know what cmake is doing when it hangs on the first executable? The first one is random, it changes every time I recompile, but it always hangs. |
Does it hang before it starts compiling any source files, or during the link operation? I normally see a moderate (5-20 sec) hang before it compiles any source files (only the first time a new binary dir is used) due to the CMake dependency scanner running. |
I wish the things were different and the migration would be much smoother. @ladislas if it is causing that much trouble, I would stay with the setup you have working and continue using it. Thanks for sharing the feedback. |
Description of defect
Intro
As mentioned in #13974, I'm in the process of moving our current project from USCRPL/mbed-cmake to mbed os first-party CMake.
I thought it might be interesting for others to share our progress, our questions, feedback and issues we might be facing. I think that github is better than the forum for this as it is easier to share code, other repos and reference issues and PR
Our project is quite complex, with a main
src
directory containing our product firmware (main program). We have different drivers and libraries that are being used by the firmware.We also have a lot of
spikes
: standalone projects (with their ownmain.cpp
) to test components, features, libs, technical solutions, etc.The following repository is a simpler example of our big project, and I'll use it to test all the features.
How it works now
USCRPL/mbed-cmake allows us to have the spikes and the main program live in the same repository and compile together without any issues. We then use openocd to flash the
.bin
we want to use in our product.USCRPL/mbed-cmake first creates a
mbed-os-static STATIC
library (https://github.com/USCRPL/mbed-cmake/blob/c0b0f7d4080bba179b9390e877eb0c1e7467f6d0/cmake/MbedOS-GCCArm.cmake#L35-L37) which is then wrapped into anmbed-os INTERFACE
that defined the linker script and link options (https://github.com/USCRPL/mbed-cmake/blob/c0b0f7d4080bba179b9390e877eb0c1e7467f6d0/cmake/MbedOS-GCCArm.cmake#L50-L58). It's thismbed-os INTERFACE
that our main program, spikes drivers and libs link with.From a user perspective, mbed-os-static is compiled once for the main program and for all the spikes, drivers and libs. Adding a new spike with just a
main.cpp
file, will only increment the whole compilation steps number by one.First try with Mbed OS first-party CMake
On the other hand, mbed os first-party cmake "recompiles" all the needed sources for each target depending on it. Adding a new spike linking with
mbed-os
will throw in the compilation a few hundred more "steps". Looking at the output, one can clearly see that mbed os or target files are compiled multiple times for the libs, the spikes and the main program.Issues
And then it stops as the way it handles the linker script is not made for multiple
add_executable
targets.Another issue was that adding another
executable
and callingmbed_set_mbed_target_linker_script
resulted in the following error about.compile_time_defs.txt
:CMake Error: Files to be generated by multiple different commands: "/Users/ladislas/dev/ladislas/mbed-cmake-template/build/compile_time_defs.txt"
The issue comes from here:
mbed-os/tools/cmake/toolchain.cmake
Lines 22 to 23 in 33a7e66
Note that the two files generated are identical. If they can never be different, it would be better to juste generate one and use it for all
executable
.Questions
Please note I'm neither a CMake expert nor a
.ld
expert.From my understanding: for the same mbed target (DISCO_F769NI in our case), the linker script is always the same. If we change target, we must change the linker script. In real life projects this happens seldom as your board/target is usually the same for a long period of time. But even if we need to change it, a new run of
mbed-tools configure
will provide everything needed.Currently
mbed-os
,mbed-core
,mbed-rtos
are defined asINTERFACE
libraries, meaning they are not compiled on their own (as static or object libraries would be), but become part of the target (other library or executable) that links with it.So calling
mbed_set_mbed_target_linker_script
for eachexecutable
will also callmbed_set_toolchain_options
multiple times.My first question is:
And secondly:
I'd be happy to help in this case.
Thanks for reading till here and also thanks for bringing CMake to mbed-os! :)
(cc @0xc0170 @hugueskamba @rajkan01 @multiplemonomials @ProExpertProg)
Target(s) affected by this defect ?
DISCO_F769NI
Toolchain(s) (name and version) displaying this defect ?
What version of Mbed-os are you using (tag or sha) ?
33a7e66
What version(s) of tools are you using. List all that apply (E.g. mbed-cli)
How is this defect reproduced ?
to do
The text was updated successfully, but these errors were encountered: