mbed-cli build path exceeds Windows command line argument sting length

[dlfryar-zz]

Description

Building mbed-cloud-client-example and possibly other projects that have lots of source files overruns the Windows command line argument sting length of 32k. Targets like Nordic where the peripheral feature set is broken down to separate directories (e.g. mbed-os/targets/TARGET_NORDIC/TARGET_NRF5x/TARGET_SDK_14_2/drivers_nrf/*) exacerbate this problem and make it more noticeable on smaller source projects.

maximum-length-of-command-line-string

Error Output

Source files to be passed to compiler are truncated on the command line in Windows.

[DEBUG] Return: 1
[DEBUG] Output: arm-none-eabi-g++: error: CreateProcess: No such file or directory
  File "C:\mbed-cloud-client-example\mbed-os\tools\toolchains\__init__.py", line 974, in compile_queue
    raise ToolException(err)
ToolException: arm-none-eabi-g++: error: CreateProcess: No such file or directory

[mbed] ERROR: "c:\python27\python.exe" returned error code 1.
[mbed] ERROR: Command "c:\mbed-cloud-client-example\mbed-os\tools\make.py -t GCC_ARM -m nRF52840_DK --source . --build .\BUILD\NRF52840_DK\GCC_ARM -v" in "C:\mbed-cloud-client-example"

Issue request type

[ ] Question
[ ] Enhancement
[X] Bug

[dlfryar-zz]

cc: @bridadan @theotherjimmy @surajdagar @bentcooke

[surajdagar]

After adding below in .mbedignore It compiled but break the UART.

source/platform/linux/
source/platform/freertos/
_K64F_FreeRTOS_mbedtls/
mbed-os/features/lorawan/
mbed-os/features/nanostack/mbed-mesh-api/
mbed-os/features/nanostack/sal-stack-nanostack/source/
mbed-os/features/nanostack/sal-stack-nanostack/docs/
mbed-os/features/nanostack/sal-stack-nanostack/doxygen/
mbed-os/features/nanostack/targets/
mbed-os/features/nanostack/nanostack-interface/
easy-connect/atmel-rf-driver/
easy-connect/mcr20a-rf-driver/
easy-connect/stm-spirit1-rf-driver/

[bridadan]

@marcuschangarm Flatenning parts of the nordic SDK port would help mediate this problem quite a bit. Specifically the mbed-os/targets/TARGET_NORDIC/TARGET_NRF5x/TARGET_SDK_14_2/drivers_nrf/* folders as mentioned by @dlfryar. What do you think?

[ciarmcom]

ARM Internal Ref: MBOTRIAGE-426

[0xc0170]

There are at least 2 issues reported and currently opened. Thus this should be closed as duplicate and add the initial report to the opened one.

[0xc0170]

Duplicate of:

#6335
#6861

[0xc0170]

@marcuschangarm Flatenning parts of the nordic SDK port would help mediate this problem quite a bit. Specifically the mbed-os/targets/TARGET_NORDIC/TARGET_NRF5x/TARGET_SDK_14_2/drivers_nrf/* folders as mentioned by @dlfryar. What do you think?

Flatenning just postpones the real issue. From experience, we get new features, targets hal folders - all these will extend the command length and will exceed the limit eventually.

[bridadan]

Agreed that it postpones the problem, but seems like the real solution is to fundamentally change how our code is built, which will take some planning

[dlfryar-zz]

@0xc0170 the other two linked are exporter and/or makefile type issues not core compilation using Windows command interpreter.

[dlfryar-zz]

cc: @bulislaw

[0xc0170]

#6861 - this is not exporter but build - all of them however are about the same - if it is exporter or a build - it is related to the command line length - one fix should resolve all of them.

I reopened #6861 (this one is duplicate, same report) - to keep 2 issues - exporters and cli building problem

[theotherjimmy]

one fix should resolve all of them.

I strongly disagree with this statement: exporters have there own limitations that I can't fix.

[theotherjimmy]

Repopening as this is not a duplicate. @0xc0170 Do not assume that mbed test == mbed compile #6861 is a bug where an absolute path is used.

[cmonr]

Some more info.

After running

git clone https://github.com/ARMmbed/mbed-cloud-client-example
cd mbed-cloud-client-example
mbed deploy
mbed compile -t GCC_ARM -m NRF52840_DK -v

the command is reliably unable to compile the first file.

The output:

C:\Users\root\tmp\mbed-cloud-client-example>mbed compile -t GCC_ARM -m NRF52840_DK -v
[mbed] Working path "C:\Users\root\tmp\mbed-cloud-client-example" (program)
[mbed] WARNING: Could not find mbed program in current path "C:\Users\root\tmp\mbed-cloud-client-example".
       You can fix this by calling "mbed new ." in the root of your program.
---
[mbed] Exec "c:\python27\python.exe -u C:\Users\root\tmp\mbed-cloud-client-example\mbed-os\tools\make.py -t GCC_ARM -m NRF52840_DK --source . --build .\BUILD\NRF52840_DK\GCC_ARM -v" in "C:\Users\root\tmp\mbed-cloud-client-example"
Building project mbed-cloud-client-example (NRF52840_DK, GCC_ARM)
Scan: .
Scan: mbed
Scan: env
Scan: FEATURE_CRYPTOCELL310
Scan: FEATURE_BLE
Using regions bootloader, application in this build.
  Region bootloader: size 0x22000, offset 0x0
  Region application: size 0xde000, offset 0x22000
Macros: -DDEVICE_SPI=1 -DCMSIS_VECTAB_VIRTUAL -D__MBED__=1 -D__FPU_PRESENT=1 -DDEVICE_PORTINOUT=1 -DTARGET_RTOS_M4_M7 -DDEVICE_RTC=1 -DDEVICE_SERIAL_ASYNCH=1 -D__CMSIS_RTOS -D__CORTEX_M4 -DDEVICE_ITM=1 -DTARGET_NRF5x -DTARGET_FF_ARDUINO -DNRF52840_XXAA -DTOOLCHAIN_GCC -DTARGET_NRF52840_DK -DTARGET_CORTEX_M -DTARGET_LIKE_CORTEX_M4 -DDEVICE_TRNG=1 -DTARGET_M4 -DTARGET_UVISOR_UNSUPPORTED -DFEATURE_BLE=1 -DTARGET_SOFTDEVICE_S140_FULL -DDEVICE_SPI_ASYNCH=1 -DDEVICE_LPTICKER=1 -DDEVICE_SERIAL=1 -DMBEDTLS_CONFIG_HW_SUPPORT -DMBED_TICKLESS -DMBED_BUILD_TIMESTAMP=1533916639.77 -DTARGET_CORTEX -DDEVICE_I2C=1 -DDEVICE_PORTOUT=1 -DTARGET_SOFTDEVICE_COMMON -DTARGET_LIKE_MBED -DFEATURE_CRYPTOCELL310=1 -DFEATURE_COMMON_PAL=1 -DDEVICE_PORTIN=1 -DTARGET_RELEASE -DTARGET_SDK_14_2 -DDEVICE_STCLK_OFF_DURING_SLEEP=1 -DTARGET_NORDIC -DTARGET_NRF52 -DDEVICE_USTICKER=1 -DCONFIG_GPIO_AS_PINRESET -DBOARD_PCA10056 -D__MBED_CMSIS_RTOS_CM -DDEVICE_SLEEP=1 -DTOOLCHAIN_GCC_ARM -DSWI_DISABLE0 -DNRF52_ERRATA_20 -DDEVICE_I2C_ASYNCH=1 -DCMSIS_VECTAB_VIRTUAL_HEADER_FILE="cmsis_nvic.h" -DDEVICE_INTERRUPTIN=1 -DTARGET_MCU_NRF52840 -DDEVICE_ANALOGIN=1 -DDEVICE_PWMOUT=1 -DTARGET_NRF52840 -DDEVICE_FLASH=1 -DARM_MATH_CM4 -DDEVICE_SERIAL_FC=1
Compile [  0.1%]: NanostackRfPhyAtmel.cpp
Compile: arm-none-eabi-g++ -std=gnu++98 -fno-rtti -Wvla -c -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -fmessage-length=0 -fno-exceptions -fno-builtin -ffunction-sections -fdata-sections -funsigned-char -MMD -fno-delete-null-pointer-checks -fomit-frame-pointer -Os -g1 -D_RTE_ -mcpu=cortex-m4 -mthumb -mfpu=fpv4-sp-d16 -mfloat-abi=softfp -DBOOTLOADER_ADDR=0x0 -DBOOTLOADER_SIZE=0x22000 -DAPPLICATION_ADDR=0x22000 -DAPPLICATION_SIZE=0xde000 -DDEVICE_SPI=1 -DCMSIS_VECTAB_VIRTUAL -D__MBED__=1 -D__FPU_PRESENT=1 -DDEVICE_PORTINOUT=1 -DTARGET_RTOS_M4_M7 -DDEVICE_RTC=1 -DDEVICE_SERIAL_ASYNCH=1 -D__CMSIS_RTOS -D__CORTEX_M4 -DDEVICE_ITM=1 -DTARGET_NRF5x -DTARGET_FF_ARDUINO -DNRF52840_XXAA -DTOOLCHAIN_GCC -DTARGET_NRF52840_DK -DTARGET_CORTEX_M -DTARGET_LIKE_CORTEX_M4 -DDEVICE_TRNG=1 -DTARGET_M4 -DTARGET_UVISOR_UNSUPPORTED -DFEATURE_BLE=1 -DTARGET_SOFTDEVICE_S140_FULL -DDEVICE_SPI_ASYNCH=1 -DDEVICE_LPTICKER=1 -DDEVICE_SERIAL=1 -DMBEDTLS_CONFIG_HW_SUPPORT -DMBED_TICKLESS -DMBED_BUILD_TIMESTAMP=1533916639.77 -DTARGET_CORTEX -DDEVICE_I2C=1 -DDEVICE_PORTOUT=1 -DTARGET_SOFTDEVICE_COMMON -DTARGET_LIKE_MBED -DFEATURE_CRYPTOCELL310=1 -DFEATURE_COMMON_PAL=1 -DDEVICE_PORTIN=1 -DTARGET_RELEASE -DTARGET_SDK_14_2 -DDEVICE_STCLK_OFF_DURING_SLEEP=1 -DTARGET_NORDIC -DTARGET_NRF52 -DDEVICE_USTICKER=1 -DCONFIG_GPIO_AS_PINRESET -DBOARD_PCA10056 -D__MBED_CMSIS_RTOS_CM -DDEVICE_SLEEP=1 -DTOOLCHAIN_GCC_ARM -DSWI_DISABLE0 -DNRF52_ERRATA_20 -DDEVICE_I2C_ASYNCH=1 -DCMSIS_VECTAB_VIRTUAL_HEADER_FILE="cmsis_nvic.h" -DDEVICE_INTERRUPTIN=1 -DTARGET_MCU_NRF52840 -DDEVICE_ANALOGIN=1 -DDEVICE_PWMOUT=1 -DTARGET_NRF52840 -DDEVICE_FLASH=1 -DARM_MATH_CM4 -DDEVICE_SERIAL_FC=1 @.\BUILD\NRF52840_DK\GCC_ARM\.includes_caee3eb4543e3b31a0d18d93d4078d69.txt -include .\BUILD\NRF52840_DK\GCC_ARM\mbed_config.h -MD -MF BUILD\NRF52840_DK\GCC_ARM\easy-connect\atmel-rf-driver\source\NanostackRfPhyAtmel.d -o BUILD\NRF52840_DK\GCC_ARM\easy-connect\atmel-rf-driver\source\NanostackRfPhyAtmel.o .\easy-connect\atmel-rf-driver\source\NanostackRfPhyAtmel.cpp
[DEBUG] Return: 1
[DEBUG] Output: arm-none-eabi-g++: error: CreateProcess: No such file or directory
Traceback (most recent call last):
  File "C:\Users\root\tmp\mbed-cloud-client-example\mbed-os\tools\make.py", line 290, in <module>
    ignore=options.ignore)
  File "C:\Users\root\tmp\mbed-cloud-client-example\mbed-os\tools\build_api.py", line 586, in build_project
    objects = toolchain.compile_sources(resources, resources.inc_dirs)
  File "C:\Users\root\tmp\mbed-cloud-client-example\mbed-os\tools\toolchains\__init__.py", line 914, in compile_sources
    return self.compile_queue(queue, objects)
  File "C:\Users\root\tmp\mbed-cloud-client-example\mbed-os\tools\toolchains\__init__.py", line 977, in compile_queue
    raise ToolException(err)
ToolException: arm-none-eabi-g++: error: CreateProcess: No such file or directory

[mbed] ERROR: "c:\python27\python.exe" returned error.
       Code: 1
       Path: "C:\Users\root\tmp\mbed-cloud-client-example"
       Command: "c:\python27\python.exe -u C:\Users\root\tmp\mbed-cloud-client-example\mbed-os\tools\make.py -t GCC_ARM -m NRF52840_DK --source . --build .\BUILD\NRF52840_DK\GCC_ARM -v"
       Tip: You could retry the last command with "-v" flag for verbose output
---

Attached is the .includes file that is autogenerated so that it can be piped into GCC. Note that because it's above 32kb (the Windows CreateProcess limit), includes get trunkcated causing the compile to fail.

includes_caee3eb4543e3b31a0d18d93d4078d69.txt

Some sort of directory squashing needs to happen, since there are a whole bunch of long directories still showing up in the include file.

[adbridge]

Internal Jira reference: https://jira.arm.com/browse/IOTCORE-529

[ciarmcom]

We closed this issue because it has been inactive for quite some time and we believe it to be low priority. If you think that the priority should be higher, then please reopen with your justification for increasing the priority.