cmake compiler cache failures in CI

[nashif]

see https://app.shippable.com/github/zephyrproject-rtos/zephyr/runs/22294/4/console

-- Performing Test toolchain_is_ok - Success
CMake Error at /home/buildslave/.cache/zephyr/ToolchainCapabilityDatabase.cmake:76:
  Parse error.  Function missing ending ")".  End of file reached.
Call Stack (most recent call first):
  ../../../cmake/extensions.cmake:638 (include)
  ../../../cmake/extensions.cmake:1020 (zephyr_check_compiler_flag)
  ../../../cmake/extensions.cmake:996 (target_cc_option_fallback)
  ../../../cmake/extensions.cmake:106 (target_cc_option)
  ../../../CMakeLists.txt:212 (zephyr_cc_option)

[SebastianBoe]

Will get this fixed. Any information / half-baked-theories would be useful ...

[SebastianBoe]

Possible causes:

• The process that writes to the cache is killed mid-line
• Some cache entries are not parsable as set(a b) code.
• The process that reads the cache does so while another process is mid-line in a write

[nashif]

https://app.shippable.com/github/zephyrproject-rtos/zephyr/runs/22778/5/console

[SebastianBoe]

I see from those logs that there is a new symptom (or a new root cause). CMake runs the command

  file(
    APPEND
    ${ZEPHYR_TOOLCHAIN_CAPABILITY_CACHE}
    "set(${key} ${inner_check})\n"
    )

and fails with the error:

CMake Error at ../../../../cmake/extensions.cmake:671 (file):
  file failed to open for writing (No such file or directory):

    /home/buildslave/.cache/zephyr/ToolchainCapabilityDatabase.cmake
Call Stack (most recent call first):
  ../../../../cmake/extensions.cmake:1020 (zephyr_check_compiler_flag)
  ../../../../cmake/extensions.cmake:996 (target_cc_option_fallback)
  ../../../../cmake/extensions.cmake:106 (target_cc_option)
  ../../../../CMakeLists.txt:210 (zephyr_cc_option)

I cannot yet explain how this can happen. But I can say that it is not a permission issue, as I have tested what happens when you have insufficient permissions, and the error message is different (Permission denied). Note that file(APPEND should create the file when it doesn't exist, so there is no user-error here.

Will continue investigating ...

[SebastianBoe]

Have added more debug information in PR #11027 . Will investigate when it triggers with the same symptoms again (AFAICT it has not triggered for a week now).