Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add bolt #22204

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open

add bolt #22204

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
8886325
add bolt
h-vetinari Mar 4, 2023
e9097a6
lint
h-vetinari Mar 4, 2023
b888381
rename to llvm-bolt
h-vetinari Mar 4, 2023
4d9a06b
patch in minimal standalone changes for bolt CMakeLists
h-vetinari Mar 4, 2023
c7c344f
simplify patch
h-vetinari Mar 4, 2023
dd2dbeb
include necessary CMake module
h-vetinari Mar 4, 2023
12d9023
point CMAKE_MODULE_PATH to llvm's cmake/Modules
h-vetinari Mar 4, 2023
46b34e2
patch in module search path like libcxx
h-vetinari Mar 4, 2023
350e74c
try lld approach
h-vetinari Mar 4, 2023
aedb4ee
bump to 16.0.1
h-vetinari Apr 15, 2023
61cc902
use matching compiler on osx
h-vetinari Apr 15, 2023
f79e231
support libs
h-vetinari Apr 15, 2023
dca8d45
bump to 16.0.3
h-vetinari May 18, 2023
d39eba5
Merge remote-tracking branch 'up/main' into bolt
h-vetinari Jul 10, 2024
19aebfc
bump to 17.0.6
h-vetinari Jul 10, 2024
828a41b
rebase patches
h-vetinari Jul 10, 2024
c605794
add stdlib
h-vetinari Jul 10, 2024
e51d279
Merge remote-tracking branch 'upstream/main' into bolt
h-vetinari Jul 24, 2024
53b59f6
build bolt together with llvmdev
h-vetinari Jul 24, 2024
cb2f1c4
fix clobber
h-vetinari Jul 24, 2024
f8d0234
-devel
h-vetinari Jul 24, 2024
0e96961
package CMake metadata for libbolt-devel
h-vetinari Jul 24, 2024
11472ee
add test for overlooked binary
h-vetinari Jul 24, 2024
eebba24
more -devel
h-vetinari Jul 24, 2024
99ba39a
copy CMake metadata
h-vetinari Jul 25, 2024
6bb8bdc
Properly copy into existing folder
h-vetinari Jul 25, 2024
a54de91
remove superfluous prefix
h-vetinari Jul 25, 2024
bd1a6f1
make libbolt-devel depend on llvmdev
h-vetinari Jul 26, 2024
004cdc9
remove extra target; build with MSVC on windows
h-vetinari Jul 26, 2024
4f17e53
install bolt includes, if any exist
h-vetinari Jul 26, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions recipes/bolt/bld.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
@echo on

:: until https://github.com/llvm/llvm-project/pull/97130 lands,
:: follow https://github.com/conda-forge/llvmdev-feedstock/blob/main/recipe/build.sh,
:: with a reduced set of targets, no tests, plus LLVM_ENABLE_PROJECTS="bolt"

mkdir build
cd build

cmake -G "Ninja" ^
-DCMAKE_BUILD_TYPE="Release" ^
-DCMAKE_PREFIX_PATH=%LIBRARY_PREFIX% ^
-DCMAKE_INSTALL_PREFIX:PATH=%LIBRARY_PREFIX% ^
-DCMAKE_MSVC_RUNTIME_LIBRARY=MultiThreadedDLL ^
-DLLVM_USE_INTEL_JITEVENTS=ON ^
-DLLVM_ENABLE_DUMP=ON ^
-DLLVM_ENABLE_LIBXML2=FORCE_ON ^
-DLLVM_ENABLE_PROJECTS="bolt" ^
-DLLVM_ENABLE_RTTI=ON ^
-DLLVM_ENABLE_ZLIB=FORCE_ON ^
-DLLVM_ENABLE_ZSTD=FORCE_ON ^
-DLLVM_INCLUDE_BENCHMARKS=OFF ^
-DLLVM_INCLUDE_DOCS=OFF ^
-DLLVM_INCLUDE_EXAMPLES=OFF ^
-DLLVM_INCLUDE_TESTS=ON ^
-DLLVM_INCLUDE_UTILS=ON ^
-DLLVM_INSTALL_UTILS=ON ^
-DLLVM_TARGETS_TO_BUILD="X86;AArch64" ^
-DLLVM_USE_SYMLINKS=OFF ^
-DLLVM_UTILS_INSTALL_DIR=libexec\llvm ^
-DLLVM_BUILD_LLVM_C_DYLIB=ON ^
-DCMAKE_POLICY_DEFAULT_CMP0111=NEW ^
%SRC_DIR%/llvm
if %ERRORLEVEL% neq 0 exit 1

cmake --build .
if %ERRORLEVEL% neq 0 exit 1
68 changes: 68 additions & 0 deletions recipes/bolt/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
#!/bin/bash
set -ex

# until https://github.com/llvm/llvm-project/pull/97130 lands,
# follow https://github.com/conda-forge/llvmdev-feedstock/blob/main/recipe/build.sh,
# with a reduced set of targets, no tests, plus LLVM_ENABLE_PROJECTS="bolt"

# Make osx work like linux.
sed -i.bak "s/NOT APPLE AND ARG_SONAME/ARG_SONAME/g" llvm/cmake/modules/AddLLVM.cmake
sed -i.bak "s/NOT APPLE AND NOT ARG_SONAME/NOT ARG_SONAME/g" llvm/cmake/modules/AddLLVM.cmake

mkdir build
cd build

if [[ "$target_platform" == "linux-64" ]]; then
CMAKE_ARGS="${CMAKE_ARGS} -DLLVM_USE_INTEL_JITEVENTS=ON"
elif [[ "$target_platform" == osx-* ]]; then
# only supported on osx, see
# https://github.com/llvm/llvm-project/blob/llvmorg-16.0.6/llvm/tools/llvm-shlib/CMakeLists.txt#L82-L85
# currently off though, because it doesn't build yet
# CMAKE_ARGS="${CMAKE_ARGS} -DLLVM_BUILD_LLVM_C_DYLIB=ON"
true
fi

if [[ "$CC_FOR_BUILD" != "" && "$CC_FOR_BUILD" != "$CC" ]]; then
NATIVE_FLAGS="-DCMAKE_C_COMPILER=$CC_FOR_BUILD;-DCMAKE_CXX_COMPILER=$CXX_FOR_BUILD;-DCMAKE_C_FLAGS=-O2;-DCMAKE_CXX_FLAGS=-O2"
NATIVE_FLAGS="${NATIVE_FLAGS};-DCMAKE_EXE_LINKER_FLAGS=-Wl,-rpath,${BUILD_PREFIX}/lib;-DCMAKE_MODULE_LINKER_FLAGS=;-DCMAKE_SHARED_LINKER_FLAGS="
NATIVE_FLAGS="${NATIVE_FLAGS};-DCMAKE_STATIC_LINKER_FLAGS=;-DLLVM_INCLUDE_BENCHMARKS=OFF"
NATIVE_FLAGS="${NATIVE_FLAGS};-DLLVM_ENABLE_ZSTD=OFF;-DLLVM_ENABLE_LIBXML2=OFF;-DLLVM_ENABLE_ZLIB=OFF;"
CMAKE_ARGS="${CMAKE_ARGS} -DCROSS_TOOLCHAIN_FLAGS_NATIVE=${NATIVE_FLAGS}"
fi

# disable -fno-plt due to https://bugs.llvm.org/show_bug.cgi?id=51863 due to some GCC bug
if [[ "$target_platform" == "linux-ppc64le" ]]; then
CFLAGS="$(echo $CFLAGS | sed 's/-fno-plt //g')"
CXXFLAGS="$(echo $CXXFLAGS | sed 's/-fno-plt //g')"
fi

cmake -G Ninja \
-DCMAKE_INSTALL_PREFIX="${PREFIX}" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_LIBRARY_PATH="${PREFIX}" \
-DLLVM_ENABLE_BACKTRACES=ON \
-DLLVM_ENABLE_DUMP=ON \
-DLLVM_ENABLE_LIBEDIT=OFF \
-DLLVM_ENABLE_LIBXML2=FORCE_ON \
-DLLVM_ENABLE_PROJECTS="bolt" \
-DLLVM_ENABLE_RTTI=ON \
-DLLVM_ENABLE_TERMINFO=OFF \
-DLLVM_ENABLE_ZLIB=FORCE_ON \
-DLLVM_ENABLE_ZSTD=FORCE_ON \
-DLLVM_DEFAULT_TARGET_TRIPLE=${CONDA_TOOLCHAIN_HOST} \
-DLLVM_HOST_TRIPLE=${CONDA_TOOLCHAIN_HOST} \
-DLLVM_INCLUDE_BENCHMARKS=OFF \
-DLLVM_INCLUDE_DOCS=OFF \
-DLLVM_INCLUDE_EXAMPLES=OFF \
-DLLVM_INCLUDE_GO_TESTS=OFF \
-DLLVM_INCLUDE_TESTS=ON \
-DLLVM_INCLUDE_UTILS=ON \
-DLLVM_INSTALL_UTILS=ON \
-DLLVM_TARGETS_TO_BUILD="X86;AArch64" \
-DLLVM_UTILS_INSTALL_DIR=libexec/llvm \
-DLLVM_BUILD_LLVM_DYLIB=yes \
-DLLVM_LINK_LLVM_DYLIB=yes \
${CMAKE_ARGS} \
../llvm

ninja -j${CPU_COUNT}
21 changes: 21 additions & 0 deletions recipes/bolt/install_bolt.bat
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
@echo on

:: temporary prefix to be able to install files more granularly
mkdir temp_prefix

if "%PKG_NAME%" == "libbolt-devel" (
cmake --install ./build --prefix=./temp_prefix
if %ERRORLEVEL% neq 0 exit 1
REM only bolt libraries
move .\temp_prefix\lib\LLVMBOLT*.lib %LIBRARY_LIB%
REM copy CMake metadata
move .\temp_prefix\lib\cmake\llvm %LIBRARY_LIB%\cmake\llvm
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about include files?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had been wondering about that too, but there's nothing I can find. The only occurrences of include/bolt in the logs (example) are during the build, not during installation. I've now done a regex-search (include/.*bolt) as well, and it only finds false positives

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there's anything that goes beyond what's in llvmdev, it's now being installed (4f17e53)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are no include files, then we might as well drop libbolt-devel as users can't use those libraries (not easily anyways).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't know sufficiently well how bolt works, but it's conceivable to me that bolt the binary needs those libraries to actually transform a binary (perhaps embedding some symbols from the support libraries along the way).

It seemed natural to split off the libraries, but if you want, I can just stuff everything into bolt itself.

REM unclear which headers belong to bolt, but if some are there, install
move .\temp_prefix\include %LIBRARY_INC%
) else (
REM bolt: everything else
cmake --install .\build --prefix=%LIBRARY_PREFIX%
if %ERRORLEVEL% neq 0 exit 1
)

rmdir /s /q temp_prefix
21 changes: 21 additions & 0 deletions recipes/bolt/install_bolt.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
#!/bin/bash
set -ex

# temporary prefix to be able to install files more granularly
mkdir temp_prefix

if [[ "${PKG_NAME}" == "libbolt-devel" ]]; then
cmake --install ./build --prefix=./temp_prefix
mv ./temp_prefix/lib/libLLVMBOLT* $PREFIX/lib
# only on linux-64
mv ./temp_prefix/lib/libbolt* $PREFIX/lib || true
# move CMake metadata
mv ./temp_prefix/lib/cmake/llvm/* $PREFIX/lib/cmake/llvm/
# unclear which headers belong to bolt, but if some are there, install
mv ./temp_prefix/include/* $PREFIX/include/
else
# bolt: install everything else
cmake --install ./build --prefix=$PREFIX
fi

rm -rf temp_prefix
143 changes: 143 additions & 0 deletions recipes/bolt/meta.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,143 @@
{% set version = "17.0.6" %}
{% set major_ver = version.split(".")[0] %}

# as of LLVM 17, we expect an "rc" suffix for the release candidates
{% set extra = "rc" if version.split(".")[-1] not in "0123456789" else "" %}
{% set extra = "git" if version.split(".")[-1] == "dev0" else extra %}

package:
name: bolt-split
version: {{ version }}

source:
url: https://github.com/llvm/llvm-project/releases/download/llvmorg-{{ version.replace(".rc", "-rc") }}/llvm-project-{{ version.replace(".rc", "rc") }}.src.tar.xz
sha256: 58a8818c60e6627064f312dbf46c02d9949956558340938b71cf731ad8bc0813
# matching https://github.com/conda-forge/llvmdev-feedstock/tree/main/recipe/patches,
# since we rebuild llvmdev here (until https://github.com/llvm/llvm-project/pull/97130)
patches:
- patches/0001-pass-through-QEMU_LD_PREFIX-SDKROOT.patch
# backports patch for issues with LLJIT, see
# https://github.com/llvm/llvm-project/commit/122ebe3b500190b1f408e2e6db753853e297ba28
- patches/0002-ORC-Use-EPC-bootstrap-symbols-to-communicate-eh-fram.patch

build:
number: 0
merge_build_host: false

requirements:
build:
- {{ stdlib('c') }}
- {{ compiler('cxx') }}
- cmake
- ninja
- libcxx {{ cxx_compiler_version }} # [osx]
host:
- libcxx {{ cxx_compiler_version }} # [osx]
# needs aarch/ppc/arm (re)build of conda-forge/backtrace-feedstock
- backtrace # [unix and x86]
- libxml2
- zlib
- zstd

outputs:
# Contains bolt support libraries
- name: libbolt-devel
script: install_bolt.sh # [unix]
script: install_bolt.bat # [win]
build:
activate_in_script: true
always_include_files:
# Must overwrite CMake targets from llvmdev
{% set prefix = "Library/" if win else "" %}
- {{ prefix }}lib/cmake/llvm/LLVM-Config.cmake
- {{ prefix }}lib/cmake/llvm/LLVMConfig.cmake
- {{ prefix }}lib/cmake/llvm/LLVMConfigVersion.cmake
- {{ prefix }}lib/cmake/llvm/LLVMExports-release.cmake
- {{ prefix }}lib/cmake/llvm/LLVMExports.cmake

requirements:
build:
- {{ stdlib('c') }}
- {{ compiler('cxx') }}
- cmake
- ninja
- libcxx {{ cxx_compiler_version }} # [osx]
host:
# ensure we don't pick up stuff already packaged in llvmdev
- llvmdev {{ version }}
run:
# necessary due to shared CMake metadata
- llvmdev {{ version }}
run_constrained:
- llvm {{ version }}
- llvmdev {{ version }}
- clang {{ version }}
- clang-tools {{ version }}
test:
commands:
{% for each_lib in [
"Core", "Passes", "Profile", "Rewrite", "RuntimeLibs",
"TargetAArch64", "TargetX86", "Utils"
] %}
- test -f $PREFIX/lib/libLLVMBOLT{{ each_lib }}.a # [unix]
- if not exist %LIBRARY_LIB%\LLVMBOLT{{ each_lib }}.lib exit 1 # [win]
{% endfor %}
# only on linux-64
- test -f $PREFIX/lib/libbolt_rt_hugify.a # [linux64]
- test -f $PREFIX/lib/libbolt_rt_instr.a # [linux64]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also check that the cmake files are here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only way to do that currently is to clobber

lib/cmake/llvm/LLVM-Config.cmake
lib/cmake/llvm/LLVMConfig.cmake
lib/cmake/llvm/LLVMConfigVersion.cmake
lib/cmake/llvm/LLVMExports-release.cmake
lib/cmake/llvm/LLVMExports.cmake

from llvmdev, unfortunately. But I guess better than nothing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bolt doesn't have different cmake files?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope, it's built as part of llvm. I guess we could ask for separate CMake files in the "standalone" upstream PR...

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change your opinion somehow, or are you fine with the always_include_files: (0e96961)?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@isuruf, could you let me know how you'd like to proceed with the CMake metadata here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make libbolt-devel depend on llvmdev

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I'm planning to merge this once CI passes (well, on unix at least; the rest I'll fix on the feedstock).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't. We require reviews from everyone including core team.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then please don't drip-feed change requests. If there's been two rounds of review and no more open comments are left, it's not unreasonable to think that things are wrapped up.

Comment on lines +86 to +87
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two should be in bolt as they are used at runtime of bolt. The others are only needed if bolt is used as a library.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure (between the two open threads) what you want me to do now. Here you say

The others are only needed if bolt is used as a library.

But in the other you say

If there are no include files, then we might as well drop libbolt-devel as users can't use those libraries

I understood that we'll move libbolt_rt_{hugify,instr} to bolt. But for the rest, do we want to:

  • ship them as libbolt-devel
  • ship them as part of bolt
  • not ship them at all until a use-case comes up

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 is preferred. 1 if not.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I found the bolt headers: https://github.com/llvm/llvm-project/tree/main/bolt/include/bolt; no idea why they're not getting installed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, I guess we can restrict this to linux until anything changes upstream w.r.t. llvm/llvm-project#72205


{% for each_file in [
"-Config", "Config", "ConfigVersion", "Exports", "Exports-release"
] %}
# CMake metadata
- test -f $PREFIX/lib/cmake/llvm/LLVM{{ each_file }}.cmake # [unix]
- if not exist %LIBRARY_LIB%/cmake/llvm/LLVM{{ each_file }}.cmake exit 1 # [win]
{% endfor %}

# Contains bolt
- name: bolt
script: install_bolt.sh # [unix]
script: install_bolt.bat # [win]
build:
activate_in_script: true
requirements:
build:
- {{ stdlib('c') }}
- {{ compiler('cxx') }}
- cmake
- ninja
- libcxx {{ cxx_compiler_version }} # [osx]
host:
- libcxx {{ cxx_compiler_version }} # [osx]
# ensure we don't pick up stuff already packaged in llvmdev/libbolt-devel
- llvmdev {{ version }}
- {{ pin_subpackage("libbolt-devel", exact=True) }}
- zlib
- zstd
run:
- {{ pin_subpackage("libbolt-devel", exact=True) }}
run_constrained:
- llvm {{ version }}
- llvmdev {{ version }}
- clang {{ version }}
- clang-tools {{ version }}
test:
commands:
- llvm-bolt -version
- llvm-boltdiff -version
- llvm-bolt-heatmap -version
- merge-fdata -version
- perf2bolt -version

about:
home: https://github.com/llvm/llvm-project/tree/main/bolt
license: Apache-2.0 WITH LLVM-exception
license_file: bolt/LICENSE.TXT
license_family: Apache
summary: BOLT is a post-link optimizer developed to speed up large applications.
dev_url: https://github.com/llvm/llvm-project/

extra:
recipe-maintainers:
- h-vetinari
feedstock-name: bolt
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
From f2ee4893b93c254d97fd51fe6f6ee605934cf6a9 Mon Sep 17 00:00:00 2001
From: Isuru Fernando <[email protected]>
Date: Tue, 4 Aug 2020 21:06:30 -0500
Subject: [PATCH 1/2] pass through QEMU_LD_PREFIX & SDKROOT

---
llvm/utils/lit/lit/TestingConfig.py | 2 ++
1 file changed, 2 insertions(+)

diff --git a/llvm/utils/lit/lit/TestingConfig.py b/llvm/utils/lit/lit/TestingConfig.py
index 76fd66502009..3009e921b621 100644
--- a/llvm/utils/lit/lit/TestingConfig.py
+++ b/llvm/utils/lit/lit/TestingConfig.py
@@ -25,6 +25,8 @@ class TestingConfig(object):
"LD_LIBRARY_PATH",
"SYSTEMROOT",
"TERM",
+ "QEMU_LD_PREFIX",
+ "SDKROOT",
"CLANG",
"LLDB",
"LD_PRELOAD",
Loading
Loading