-
-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add bolt #22204
base: main
Are you sure you want to change the base?
add bolt #22204
Changes from all commits
8886325
e9097a6
b888381
4d9a06b
c7c344f
dd2dbeb
12d9023
46b34e2
350e74c
aedb4ee
61cc902
f79e231
dca8d45
d39eba5
19aebfc
828a41b
c605794
e51d279
53b59f6
cb2f1c4
f8d0234
0e96961
11472ee
eebba24
99ba39a
6bb8bdc
a54de91
bd1a6f1
004cdc9
4f17e53
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
@echo on | ||
|
||
:: until https://github.com/llvm/llvm-project/pull/97130 lands, | ||
:: follow https://github.com/conda-forge/llvmdev-feedstock/blob/main/recipe/build.sh, | ||
:: with a reduced set of targets, no tests, plus LLVM_ENABLE_PROJECTS="bolt" | ||
|
||
mkdir build | ||
cd build | ||
|
||
cmake -G "Ninja" ^ | ||
-DCMAKE_BUILD_TYPE="Release" ^ | ||
-DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX% ^ | ||
-DCMAKE_INSTALL_PREFIX:PATH=%LIBRARY_PREFIX% ^ | ||
-DCMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL ^ | ||
-DLLVM_USE_INTEL_JITEVENTS=ON ^ | ||
-DLLVM_ENABLE_DUMP=ON ^ | ||
-DLLVM_ENABLE_LIBXML2=FORCE_ON ^ | ||
-DLLVM_ENABLE_PROJECTS="bolt" ^ | ||
-DLLVM_ENABLE_RTTI=ON ^ | ||
-DLLVM_ENABLE_ZLIB=FORCE_ON ^ | ||
-DLLVM_ENABLE_ZSTD=FORCE_ON ^ | ||
-DLLVM_INCLUDE_BENCHMARKS=OFF ^ | ||
-DLLVM_INCLUDE_DOCS=OFF ^ | ||
-DLLVM_INCLUDE_EXAMPLES=OFF ^ | ||
-DLLVM_INCLUDE_TESTS=ON ^ | ||
-DLLVM_INCLUDE_UTILS=ON ^ | ||
-DLLVM_INSTALL_UTILS=ON ^ | ||
-DLLVM_TARGETS_TO_BUILD="X86;AArch64" ^ | ||
-DLLVM_USE_SYMLINKS=OFF ^ | ||
-DLLVM_UTILS_INSTALL_DIR=libexec\llvm ^ | ||
-DLLVM_BUILD_LLVM_C_DYLIB=ON ^ | ||
-DCMAKE_POLICY_DEFAULT_CMP0111=NEW ^ | ||
%SRC_DIR%/llvm | ||
if %ERRORLEVEL% neq 0 exit 1 | ||
|
||
cmake --build . | ||
if %ERRORLEVEL% neq 0 exit 1 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,68 @@ | ||
#!/bin/bash | ||
set -ex | ||
|
||
# until https://github.com/llvm/llvm-project/pull/97130 lands, | ||
# follow https://github.com/conda-forge/llvmdev-feedstock/blob/main/recipe/build.sh, | ||
# with a reduced set of targets, no tests, plus LLVM_ENABLE_PROJECTS="bolt" | ||
|
||
# Make osx work like linux. | ||
sed -i.bak "s/NOT APPLE AND ARG_SONAME/ARG_SONAME/g" llvm/cmake/modules/AddLLVM.cmake | ||
sed -i.bak "s/NOT APPLE AND NOT ARG_SONAME/NOT ARG_SONAME/g" llvm/cmake/modules/AddLLVM.cmake | ||
|
||
mkdir build | ||
cd build | ||
|
||
if [[ "$target_platform" == "linux-64" ]]; then | ||
CMAKE_ARGS="${CMAKE_ARGS} -DLLVM_USE_INTEL_JITEVENTS=ON" | ||
elif [[ "$target_platform" == osx-* ]]; then | ||
# only supported on osx, see | ||
# https://github.com/llvm/llvm-project/blob/llvmorg-16.0.6/llvm/tools/llvm-shlib/CMakeLists.txt#L82-L85 | ||
# currently off though, because it doesn't build yet | ||
# CMAKE_ARGS="${CMAKE_ARGS} -DLLVM_BUILD_LLVM_C_DYLIB=ON" | ||
true | ||
fi | ||
|
||
if [[ "$CC_FOR_BUILD" != "" && "$CC_FOR_BUILD" != "$CC" ]]; then | ||
NATIVE_FLAGS="-DCMAKE_C_COMPILER=$CC_FOR_BUILD;-DCMAKE_CXX_COMPILER=$CXX_FOR_BUILD;-DCMAKE_C_FLAGS=-O2;-DCMAKE_CXX_FLAGS=-O2" | ||
NATIVE_FLAGS="${NATIVE_FLAGS};-DCMAKE_EXE_LINKER_FLAGS=-Wl,-rpath,${BUILD_PREFIX}/lib;-DCMAKE_MODULE_LINKER_FLAGS=;-DCMAKE_SHARED_LINKER_FLAGS=" | ||
NATIVE_FLAGS="${NATIVE_FLAGS};-DCMAKE_STATIC_LINKER_FLAGS=;-DLLVM_INCLUDE_BENCHMARKS=OFF" | ||
NATIVE_FLAGS="${NATIVE_FLAGS};-DLLVM_ENABLE_ZSTD=OFF;-DLLVM_ENABLE_LIBXML2=OFF;-DLLVM_ENABLE_ZLIB=OFF;" | ||
CMAKE_ARGS="${CMAKE_ARGS} -DCROSS_TOOLCHAIN_FLAGS_NATIVE=${NATIVE_FLAGS}" | ||
fi | ||
|
||
# disable -fno-plt due to https://bugs.llvm.org/show_bug.cgi?id=51863 due to some GCC bug | ||
if [[ "$target_platform" == "linux-ppc64le" ]]; then | ||
CFLAGS="$(echo $CFLAGS | sed 's/-fno-plt //g')" | ||
CXXFLAGS="$(echo $CXXFLAGS | sed 's/-fno-plt //g')" | ||
fi | ||
|
||
cmake -G Ninja \ | ||
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \ | ||
-DCMAKE_BUILD_TYPE=Release \ | ||
-DCMAKE_LIBRARY_PATH="${PREFIX}" \ | ||
-DLLVM_ENABLE_BACKTRACES=ON \ | ||
-DLLVM_ENABLE_DUMP=ON \ | ||
-DLLVM_ENABLE_LIBEDIT=OFF \ | ||
-DLLVM_ENABLE_LIBXML2=FORCE_ON \ | ||
-DLLVM_ENABLE_PROJECTS="bolt" \ | ||
-DLLVM_ENABLE_RTTI=ON \ | ||
-DLLVM_ENABLE_TERMINFO=OFF \ | ||
-DLLVM_ENABLE_ZLIB=FORCE_ON \ | ||
-DLLVM_ENABLE_ZSTD=FORCE_ON \ | ||
-DLLVM_DEFAULT_TARGET_TRIPLE=${CONDA_TOOLCHAIN_HOST} \ | ||
-DLLVM_HOST_TRIPLE=${CONDA_TOOLCHAIN_HOST} \ | ||
-DLLVM_INCLUDE_BENCHMARKS=OFF \ | ||
-DLLVM_INCLUDE_DOCS=OFF \ | ||
-DLLVM_INCLUDE_EXAMPLES=OFF \ | ||
-DLLVM_INCLUDE_GO_TESTS=OFF \ | ||
-DLLVM_INCLUDE_TESTS=ON \ | ||
-DLLVM_INCLUDE_UTILS=ON \ | ||
-DLLVM_INSTALL_UTILS=ON \ | ||
-DLLVM_TARGETS_TO_BUILD="X86;AArch64" \ | ||
-DLLVM_UTILS_INSTALL_DIR=libexec/llvm \ | ||
-DLLVM_BUILD_LLVM_DYLIB=yes \ | ||
-DLLVM_LINK_LLVM_DYLIB=yes \ | ||
${CMAKE_ARGS} \ | ||
../llvm | ||
|
||
ninja -j${CPU_COUNT} |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
@echo on | ||
|
||
:: temporary prefix to be able to install files more granularly | ||
mkdir temp_prefix | ||
|
||
if "%PKG_NAME%" == "libbolt-devel" ( | ||
cmake --install ./build --prefix=./temp_prefix | ||
if %ERRORLEVEL% neq 0 exit 1 | ||
REM only bolt libraries | ||
move .\temp_prefix\lib\LLVMBOLT*.lib %LIBRARY_LIB% | ||
REM copy CMake metadata | ||
move .\temp_prefix\lib\cmake\llvm %LIBRARY_LIB%\cmake\llvm | ||
REM unclear which headers belong to bolt, but if some are there, install | ||
move .\temp_prefix\include %LIBRARY_INC% | ||
) else ( | ||
REM bolt: everything else | ||
cmake --install .\build --prefix=%LIBRARY_PREFIX% | ||
if %ERRORLEVEL% neq 0 exit 1 | ||
) | ||
|
||
rmdir /s /q temp_prefix |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
#!/bin/bash | ||
set -ex | ||
|
||
# temporary prefix to be able to install files more granularly | ||
mkdir temp_prefix | ||
|
||
if [[ "${PKG_NAME}" == "libbolt-devel" ]]; then | ||
cmake --install ./build --prefix=./temp_prefix | ||
mv ./temp_prefix/lib/libLLVMBOLT* $PREFIX/lib | ||
# only on linux-64 | ||
mv ./temp_prefix/lib/libbolt* $PREFIX/lib || true | ||
# move CMake metadata | ||
mv ./temp_prefix/lib/cmake/llvm/* $PREFIX/lib/cmake/llvm/ | ||
# unclear which headers belong to bolt, but if some are there, install | ||
mv ./temp_prefix/include/* $PREFIX/include/ | ||
else | ||
# bolt: install everything else | ||
cmake --install ./build --prefix=$PREFIX | ||
fi | ||
|
||
rm -rf temp_prefix |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,143 @@ | ||
{% set version = "17.0.6" %} | ||
{% set major_ver = version.split(".")[0] %} | ||
|
||
# as of LLVM 17, we expect an "rc" suffix for the release candidates | ||
{% set extra = "rc" if version.split(".")[-1] not in "0123456789" else "" %} | ||
{% set extra = "git" if version.split(".")[-1] == "dev0" else extra %} | ||
|
||
package: | ||
name: bolt-split | ||
version: {{ version }} | ||
|
||
source: | ||
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-{{ version.replace(".rc", "-rc") }}/llvm-project-{{ version.replace(".rc", "rc") }}.src.tar.xz | ||
sha256: 58a8818c60e6627064f312dbf46c02d9949956558340938b71cf731ad8bc0813 | ||
# matching https://github.com/conda-forge/llvmdev-feedstock/tree/main/recipe/patches, | ||
# since we rebuild llvmdev here (until https://github.com/llvm/llvm-project/pull/97130) | ||
patches: | ||
- patches/0001-pass-through-QEMU_LD_PREFIX-SDKROOT.patch | ||
# backports patch for issues with LLJIT, see | ||
# https://github.com/llvm/llvm-project/commit/122ebe3b500190b1f408e2e6db753853e297ba28 | ||
- patches/0002-ORC-Use-EPC-bootstrap-symbols-to-communicate-eh-fram.patch | ||
|
||
build: | ||
number: 0 | ||
merge_build_host: false | ||
|
||
requirements: | ||
build: | ||
- {{ stdlib('c') }} | ||
- {{ compiler('cxx') }} | ||
- cmake | ||
- ninja | ||
- libcxx {{ cxx_compiler_version }} # [osx] | ||
host: | ||
- libcxx {{ cxx_compiler_version }} # [osx] | ||
# needs aarch/ppc/arm (re)build of conda-forge/backtrace-feedstock | ||
- backtrace # [unix and x86] | ||
- libxml2 | ||
- zlib | ||
- zstd | ||
|
||
outputs: | ||
# Contains bolt support libraries | ||
- name: libbolt-devel | ||
script: install_bolt.sh # [unix] | ||
script: install_bolt.bat # [win] | ||
build: | ||
activate_in_script: true | ||
always_include_files: | ||
# Must overwrite CMake targets from llvmdev | ||
{% set prefix = "Library/" if win else "" %} | ||
- {{ prefix }}lib/cmake/llvm/LLVM-Config.cmake | ||
- {{ prefix }}lib/cmake/llvm/LLVMConfig.cmake | ||
- {{ prefix }}lib/cmake/llvm/LLVMConfigVersion.cmake | ||
- {{ prefix }}lib/cmake/llvm/LLVMExports-release.cmake | ||
- {{ prefix }}lib/cmake/llvm/LLVMExports.cmake | ||
|
||
requirements: | ||
build: | ||
- {{ stdlib('c') }} | ||
- {{ compiler('cxx') }} | ||
- cmake | ||
- ninja | ||
- libcxx {{ cxx_compiler_version }} # [osx] | ||
host: | ||
# ensure we don't pick up stuff already packaged in llvmdev | ||
- llvmdev {{ version }} | ||
run: | ||
# necessary due to shared CMake metadata | ||
- llvmdev {{ version }} | ||
run_constrained: | ||
- llvm {{ version }} | ||
- llvmdev {{ version }} | ||
- clang {{ version }} | ||
- clang-tools {{ version }} | ||
test: | ||
commands: | ||
{% for each_lib in [ | ||
"Core", "Passes", "Profile", "Rewrite", "RuntimeLibs", | ||
"TargetAArch64", "TargetX86", "Utils" | ||
] %} | ||
- test -f $PREFIX/lib/libLLVMBOLT{{ each_lib }}.a # [unix] | ||
- if not exist %LIBRARY_LIB%\LLVMBOLT{{ each_lib }}.lib exit 1 # [win] | ||
{% endfor %} | ||
# only on linux-64 | ||
- test -f $PREFIX/lib/libbolt_rt_hugify.a # [linux64] | ||
- test -f $PREFIX/lib/libbolt_rt_instr.a # [linux64] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also check that the cmake files are here There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Only way to do that currently is to clobber
from llvmdev, unfortunately. But I guess better than nothing. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. bolt doesn't have different cmake files? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nope, it's built as part of llvm. I guess we could ask for separate CMake files in the "standalone" upstream PR... There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this change your opinion somehow, or are you fine with the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @isuruf, could you let me know how you'd like to proceed with the CMake metadata here? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Let's make There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks. I'm planning to merge this once CI passes (well, on unix at least; the rest I'll fix on the feedstock). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please don't. We require reviews from everyone including core team. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Then please don't drip-feed change requests. If there's been two rounds of review and no more open comments are left, it's not unreasonable to think that things are wrapped up.
Comment on lines
+86
to
+87
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These two should be in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure (between the two open threads) what you want me to do now. Here you say
But in the other you say
I understood that we'll move
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 3 is preferred. 1 if not. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. OK, I found the bolt headers: https://github.com/llvm/llvm-project/tree/main/bolt/include/bolt; no idea why they're not getting installed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also, I guess we can restrict this to linux until anything changes upstream w.r.t. llvm/llvm-project#72205 |
||
|
||
{% for each_file in [ | ||
"-Config", "Config", "ConfigVersion", "Exports", "Exports-release" | ||
] %} | ||
# CMake metadata | ||
- test -f $PREFIX/lib/cmake/llvm/LLVM{{ each_file }}.cmake # [unix] | ||
- if not exist %LIBRARY_LIB%/cmake/llvm/LLVM{{ each_file }}.cmake exit 1 # [win] | ||
{% endfor %} | ||
|
||
# Contains bolt | ||
- name: bolt | ||
script: install_bolt.sh # [unix] | ||
script: install_bolt.bat # [win] | ||
build: | ||
activate_in_script: true | ||
requirements: | ||
build: | ||
- {{ stdlib('c') }} | ||
- {{ compiler('cxx') }} | ||
- cmake | ||
- ninja | ||
- libcxx {{ cxx_compiler_version }} # [osx] | ||
host: | ||
- libcxx {{ cxx_compiler_version }} # [osx] | ||
# ensure we don't pick up stuff already packaged in llvmdev/libbolt-devel | ||
- llvmdev {{ version }} | ||
- {{ pin_subpackage("libbolt-devel", exact=True) }} | ||
- zlib | ||
- zstd | ||
run: | ||
- {{ pin_subpackage("libbolt-devel", exact=True) }} | ||
run_constrained: | ||
- llvm {{ version }} | ||
- llvmdev {{ version }} | ||
- clang {{ version }} | ||
- clang-tools {{ version }} | ||
test: | ||
commands: | ||
- llvm-bolt -version | ||
- llvm-boltdiff -version | ||
- llvm-bolt-heatmap -version | ||
- merge-fdata -version | ||
- perf2bolt -version | ||
|
||
about: | ||
home: https://github.com/llvm/llvm-project/tree/main/bolt | ||
license: Apache-2.0 WITH LLVM-exception | ||
license_file: bolt/LICENSE.TXT | ||
license_family: Apache | ||
summary: BOLT is a post-link optimizer developed to speed up large applications. | ||
dev_url: https://github.com/llvm/llvm-project/ | ||
|
||
extra: | ||
recipe-maintainers: | ||
- h-vetinari | ||
feedstock-name: bolt |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
From f2ee4893b93c254d97fd51fe6f6ee605934cf6a9 Mon Sep 17 00:00:00 2001 | ||
From: Isuru Fernando <[email protected]> | ||
Date: Tue, 4 Aug 2020 21:06:30 -0500 | ||
Subject: [PATCH 1/2] pass through QEMU_LD_PREFIX & SDKROOT | ||
|
||
--- | ||
llvm/utils/lit/lit/TestingConfig.py | 2 ++ | ||
1 file changed, 2 insertions(+) | ||
|
||
diff --git a/llvm/utils/lit/lit/TestingConfig.py b/llvm/utils/lit/lit/TestingConfig.py | ||
index 76fd66502009..3009e921b621 100644 | ||
--- a/llvm/utils/lit/lit/TestingConfig.py | ||
+++ b/llvm/utils/lit/lit/TestingConfig.py | ||
@@ -25,6 +25,8 @@ class TestingConfig(object): | ||
"LD_LIBRARY_PATH", | ||
"SYSTEMROOT", | ||
"TERM", | ||
+ "QEMU_LD_PREFIX", | ||
+ "SDKROOT", | ||
"CLANG", | ||
"LLDB", | ||
"LD_PRELOAD", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What about include files?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had been wondering about that too, but there's nothing I can find. The only occurrences of
include/bolt
in the logs (example) are during the build, not during installation. I've now done a regex-search (include/.*bolt
) as well, and it only finds false positivesThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there's anything that goes beyond what's in
llvmdev
, it's now being installed (4f17e53)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If there are no include files, then we might as well drop
libbolt-devel
as users can't use those libraries (not easily anyways).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't know sufficiently well how bolt works, but it's conceivable to me that bolt the binary needs those libraries to actually transform a binary (perhaps embedding some symbols from the support libraries along the way).
It seemed natural to split off the libraries, but if you want, I can just stuff everything into
bolt
itself.