cmake: Re-write the capability database to avoid file appends #10481
Conversation
CI sometimes fails with a temporarily corrupted toolchain capabilitiy database file. Although not proven, there is evidence that CMake's file(APPEND does not work atomically when there are concurrent writes and reads of a certain size. To avoid file appending, we re-write the key-value database implementation to store keys in filenames and values in individual single-byte files. This is (most likely) fixes zephyrproject-rtos#9992. NB: Users that have been overriding the database file location with the CMake variable ZEPHYR_TOOLCHAIN_CAPABILITY_CACHE must now specify a directory with the variable ZEPHYR_TOOLCHAIN_CAPABILITY_CACHE_DIR. Signed-off-by: Sebastian Bøe <[email protected]>
|
recheck |
Codecov Report
@@ Coverage Diff @@
## master #10481 +/- ##
=========================================
Coverage ? 53.23%
=========================================
Files ? 210
Lines ? 25800
Branches ? 5677
=========================================
Hits ? 13735
Misses ? 9755
Partials ? 2310Continue to review full report at Codecov.
|
|
@nashif : This is ready for review. |
| file( | ||
| WRITE | ||
| ${key_path} | ||
| ${inner_check} |
There was a problem hiding this comment.
Doesn't that just reduce the window of the race? I mean it's much better but you could still have two cmake instances trying to create the same key at the same time, no?
There was a problem hiding this comment.
I believe this is thread-safe.
AFAIK two threads writing the same byte to the same position of the same file is thread-safe.
But I could be wrong.
There was a problem hiding this comment.
OK so there is a hopefully harmless race but a race nevertheless. PR #15128 just submitted removes the "hopefully".
CI sometimes fails with a temporarily corrupted toolchain capabilitiy
database file. Although not proven, there is evidence that CMake's
file(APPEND does not work atomically when there are concurrent writes
and reads of a certain size.
To avoid file appending, we re-write the key-value database
implementation to store keys in filenames and values in individual
single-byte files.
This (most likely) fixes #9992.
NB: Users that have been overriding the database file location with
the CMake variable ZEPHYR_TOOLCHAIN_CAPABILITY_CACHE must now specify
a directory with the variable ZEPHYR_TOOLCHAIN_CAPABILITY_CACHE_DIR.
Signed-off-by: Sebastian Bøe [email protected]