[BOLT] llvm-bolt crashes during Python 3.13.1 build with addDebugFilenameToUnit segmentation fault

[ms178]

Compiling python 3.13.1 with --enable-bolt on CachyOS with Clang-20git (7226b39) and the patch below yields the following crash.

Analysis:

llvm-bolt crashes with a segmentation fault during the optimization phase. The crash occurs in the llvm::bolt::BinaryContext::addDebugFilenameToUnit function, suggesting an issue related to handling debug information.

BOLT-INFO: SCTC: patched 53 tail calls (53 forward) tail calls (0 backward) from a total of 53 while removing 11 double jumps and removing 59 basic blocks totalling 280 bytes of code. CTCs total execution count is 6035 and the number of times CTCs are taken is 5854
BOLT-INFO: FOP optimized 54 redundant load(s) and 0 unused store(s)
BOLT-INFO: Frequency of redundant loads is 139 and frequency of unused stores is 0
BOLT-INFO: Frequency of loads changed to use a register is 139 and frequency of loads changed to use an immediate is 0
BOLT-INFO: FOP deleted 38 load(s) (dyn count: 139) and 0 store(s)
BOLT-INFO: FRAME ANALYSIS: 5171 function(s) were not optimized.
BOLT-INFO: FRAME ANALYSIS: 703 function(s) (67.5% dyn cov) could not have its frame indices restored.
BOLT-INFO: Shrink wrapping moved 108 spills inserting load/stores and 3 spills inserting push/pops
BOLT-INFO: Shrink wrapping reduced 53877 store executions (0.4% total instructions executed, 4.1% store instructions)
BOLT-INFO: Shrink wrapping failed at reducing 0 store executions (0.0% total instructions executed, 0.0% store instructions)
BOLT-INFO: Allocation combiner: 97 empty spaces coalesced (dyn count: 70232).
#0 0x000064bed9145205 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) Signals.cpp:0:0
#1 0x000064bed91455fc SignalHandler(int) Signals.cpp:0:0
#2 0x000074ab0c9fc4b0 (/usr/lib/libc.so.6+0x4b4b0)
#3 0x000064bed9693b51 llvm::bolt::BinaryContext::addDebugFilenameToUnit(unsigned int, unsigned int, unsigned int) (/home/marcus/llvm20/bin/llvm-bolt+0x3c93b51)
#4 0x000064bed96a5251 (anonymous namespace)::BinaryEmitter::emitFunctionBody(llvm::bolt::BinaryFunction&, llvm::bolt::FunctionFragment&, bool) (.llvm.10412579348155975206) BinaryEmitter.cpp:0:0
#5 0x000064bed96a5abc (anonymous namespace)::BinaryEmitter::emitFunction(llvm::bolt::BinaryFunction&, llvm::bolt::FunctionFragment&) BinaryEmitter.cpp:0:0
#6 0x000064bed96a2f24 llvm::bolt::emitBinaryContext(llvm::MCStreamer&, llvm::bolt::BinaryContext&, llvm::StringRef) (/home/marcus/llvm20/bin/llvm-bolt+0x3ca2f24)
#7 0x000064bed91d7813 llvm::bolt::RewriteInstance::run() (/home/marcus/llvm20/bin/llvm-bolt+0x37d7813)
#8 0x000064bed853d234 main (/home/marcus/llvm20/bin/llvm-bolt+0x2b3d234)
#9 0x000074ab0c9db127 (/usr/lib/libc.so.6+0x2a127)
#10 0x000074ab0c9db1e4 __libc_start_main (/usr/lib/libc.so.6+0x2a1e4)
#11 0x000064bed8530241 _start (/home/marcus/llvm20/bin/llvm-bolt+0x2b30241)
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
0.      Program arguments: /home/marcus/llvm20/bin/llvm-bolt libpython3.13.so.1.0.prebolt -o libpython3.13.so.1.0.bolt -data=libpython3.13.so.1.0.fdata -update-debug-sections -reorder-blocks=ext-tsp -reorder-functions=cdsort -split-functions -icf=1 -inline-all -split-eh -reorder-functions-use-hot-size -peepholes=none -jump-tables=aggressive -inline-ap -indirect-call-promotion=all -dyno-stats -use-gnu-stack -frame-opt=hot -infer-stale-profile=1 -split-strategy=cdsplit --skip-funcs=sre_ucs1_match/1,_PyEval_EvalFrameDefault.localalias/1

bolt.patch.txt

Flags used in /etc/makepkg.conf:

export CC=clang
export CXX=clang++
export CC_LD=lld
export CXX_LD=lld
export AR=llvm-ar
export NM=llvm-nm
export STRIP=llvm-strip
export OBJCOPY=llvm-objcopy
export OBJDUMP=llvm-objdump
export READELF=llvm-readelf
export RANLIB=llvm-ranlib
export HOSTCC=clang
export HOSTCXX=clang++
export HOSTAR=llvm-ar
export CPPFLAGS="-D_FORTIFY_SOURCE=0"
export CFLAGS="-O3 -march=native -mtune=native -fno-semantic-interposition -fcf-protection=none -mharden-sls=none -flto -fwhole-program-vtables -fdata-sections -ffunction-sections -w"
export CXXFLAGS="${CFLAGS} -Wp,-U_GLIBCXX_ASSERTIONS"
export LDFLAGS="-Wl,--lto-CGO3 -Wl,--gc-sections -Wl,--icf=all -Wl,--lto-O3,-O3,-Bsymbolic-functions,--as-needed -fcf-protection=none -mharden-sls=none -flto -fwhole-program-vtables -fuse-ld=lld -Wl,-zmax-page-size=0x200000 -Wl,--push-state -Wl,-whole-archive -lmimalloc -Wl,--pop-state -lpthread -lstdc++ -lm -ldl -Wl,-z,now -Wl,-z,relro -Wl,-z,pack-relative-relocs -Wl,--hash-style=gnu -Wl,--undefined-version"
export CCLDFLAGS="$LDFLAGS"
export CXXLDFLAGS="$LDFLAGS"
export ASFLAGS="-D__AVX__=1 -D__AVX2__=1 -D__FMA__=1"

BOLT_APPLY_FLAGS as found in the bolt.patch above:

BOLT_APPLY_FLAGS=" -update-debug-sections -reorder-blocks=ext-tsp -reorder-functions=cdsort -split-functions -icf=1 -inline-all -split-eh -reorder-functions-use-hot-size -peepholes=none -jump-tables=aggressive -inline-ap -indirect-call-promotion=all -dyno-stats -use-gnu-stack -frame-opt=hot -infer-stale-profile=1 -split-strategy=cdsplit --skip-funcs=sre_ucs1_match/1,_PyEval_EvalFrameDefault.localalias/1 "

[ms178]

Here is the related Python upstream bug report: python/cpython#124948

[liusy58]

Could you please provide a reproducer step by step?

[ms178]

@liusy58 Sure.

1. As mentioned in that discussion, you can grab a docker image with CachyOS and the llvm-bolt 19.1: [BOLT] [3.12] Python 3.12.7 --enable bolt option not working python/cpython#124948 (comment)

2. Grab the bolt-enabled PKGBUILD for python 3.13.1 and related files from my repo that comes with the required bolt.patch to use sampling instead of instrumenting. Also noticed my sligthly altered BOLT flags and the exclusion of some functions that are known to be problematic.

3. Assuming that you've downloaded these python files from my repo to your Downloads/python folder, just open a terminal and make the LLVM/Clang toolchain available in your path, then go into the python folder and then build the package with makepkg -si --cleanbuild --skipchecksums --skippgpcheck

I haven't tried it with Clang-19.1, ideally you should use the version of Clang-20git that I used which comes with BOLT support (7226b39). I've used a custom compiled Clang-20git from that revision from these scripts: https://github.com/ms178/archpkgbuilds/tree/main/toolchain-experimental/llvm-bolt-scripts-master

Just make these scripts executable with chmod +x *.sh and start the build with ./full_workflow.sh; but as I use a local PGO-Profile for the stage 1 compiler, you'd need to edit that stage1 script first to remove the PGO-related flags. My CPU is a 14700K (Intel Raptor Lake).

P.S: As BOLT is supported upstream via instrumentation with python 3.13.1, you can test both instrumenting BOLT with the upstream solution and sampling with the added patch. You just need to edit the PKGBUILD and hide the application of the patch with using #.

[ms178]

Also please notice these changes on the Python side that are supposed to adress some of the BOLT issues:

astral-sh/python-build-standalone#463

I haven't tested these yet as they are not included in an upstream release.

This merge request for BOLT is also relevant: #120267

[liusy58]

@ms178 Ok, Let me try to reproduce. Thank you~