Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LLVM] Add LLVM_full, LLVM_full_assert 19 recipes #9204

Merged
merged 25 commits into from
Oct 12, 2024

Conversation

Zentrik
Copy link
Contributor

@Zentrik Zentrik commented Aug 5, 2024

No description provided.

L/LLVM/common.jl Outdated
@@ -20,6 +20,7 @@ const llvm_tags = Dict(
v"16.0.6" => "499f87882a4ba1837ec12a280478cf4cb0d2753d", # julia-16.0.6-2
v"17.0.6" => "0007e48608221f440dce2ea0d3e4f561fc10d3c6", # julia-17.0.6-5
v"18.1.7" => "ed30d043a240d06bb6e010a41086e75713156f4f", # julia-18.1.7-2
v"19.0.0" => "6c8f4c12da7b3ac3d45e9fc9f68e2da7f559cf76", # julia-release/19.x
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reminder: we need to create the git tag on the commit we will eventually use here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Zentrik
Copy link
Contributor Author

Zentrik commented Aug 6, 2024

Not sure why the assert build can't find bundled.

@giordano
Copy link
Member

giordano commented Aug 6, 2024

You need to either follow symlinks https://docs.binarybuilder.org/stable/reference/#BinaryBuilderBase.DirectorySource or not use symlinks

@Zentrik
Copy link
Contributor Author

Zentrik commented Aug 31, 2024

You need to either follow symlinks docs.binarybuilder.org/stable/reference#BinaryBuilderBase.DirectorySource or not use symlinks

I don't understand what changed between the llvm 18 build that requires changing anything.

@giordano
Copy link
Member

https://github.com/Zentrik/Yggdrasil/blob/4a3cf630b50ce9c2d4c06770ebc1f7b3678fbe9e/L/LLVM/LLVM_full_assert%4019/bundled isn't a symlink, but a file whose content is the path of the symlink.

@Zentrik
Copy link
Contributor Author

Zentrik commented Aug 31, 2024

Looks like it's building nicely with some commits reverted, the reverts are:

  • Zentrik/llvm-project@482578d - Fixes following issue on window
    ninja: job failed: /opt/bin/x86_64-w64-mingw32-libgfortran5-cxx11/x86_64-w64-mingw32-clang++ --target=x86_64-w64-mingw32 --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -DGTEST_HAS_RTTI=0 -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/workspace/build/tools/mlir/lib/ExecutionEngine -I/workspace/srcdir/llvm-project/mlir/lib/ExecutionEngine -I/workspace/build/include -I/workspace/srcdir/llvm-project/llvm/include -I/workspace/srcdir/llvm-project/mlir/include -I/workspace/build/tools/mlir/include -remap -D__USING_SJLJ_EXCEPTIONS__ -D__CRT__NO_INLINE -pthread -DMLIR_CAPI_ENABLE_WINDOWS_DLL_DECLSPEC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -ffunction-sections -fdata-sections -Wundef -Werror=mismatched-tags -O3 -DNDEBUG -fvisibility=hidden -fvisibility-inlines-hidden  -fno-exceptions -funwind-tables -fno-rtti -std=c++17 -MD -MT tools/mlir/lib/ExecutionEngine/CMakeFiles/obj.mlir_arm_sme_abi_stubs.dir/ArmSMEStubs.cpp.obj -MF tools/mlir/lib/ExecutionEngine/CMakeFiles/obj.mlir_arm_sme_abi_stubs.dir/ArmSMEStubs.cpp.obj.d -o tools/mlir/lib/ExecutionEngine/CMakeFiles/obj.mlir_arm_sme_abi_stubs.dir/ArmSMEStubs.cpp.obj -c /workspace/srcdir/llvm-project/mlir/lib/ExecutionEngine/ArmSMEStubs.cpp
    clang++: warning: argument unused during compilation: '-remap' [-Wunused-command-line-argument]
    /workspace/srcdir/llvm-project/mlir/lib/ExecutionEngine/ArmSMEStubs.cpp:37:35: error: dllimport cannot be applied to non-inline function definition
       37 | bool MLIR_ARMSMEABISTUBS_EXPORTED __aarch64_sme_accessible() {
          |                                   ^
    /workspace/srcdir/llvm-project/mlir/lib/ExecutionEngine/ArmSMEStubs.cpp:47:40: error: dllimport cannot be applied to non-inline function definition
       47 | sme_state MLIR_ARMSMEABISTUBS_EXPORTED __arm_sme_state() {
          |                                        ^
    /workspace/srcdir/llvm-project/mlir/lib/ExecutionEngine/ArmSMEStubs.cpp:52:35: error: dllimport cannot be applied to non-inline function definition
       52 | void MLIR_ARMSMEABISTUBS_EXPORTED __arm_tpidr2_restore() {
          |                                   ^
    /workspace/srcdir/llvm-project/mlir/lib/ExecutionEngine/ArmSMEStubs.cpp:56:35: error: dllimport cannot be applied to non-inline function definition
       56 | void MLIR_ARMSMEABISTUBS_EXPORTED __arm_tpidr2_save() {
          |                                   ^
    /workspace/srcdir/llvm-project/mlir/lib/ExecutionEngine/ArmSMEStubs.cpp:60:35: error: dllimport cannot be applied to non-inline function definition
       60 | void MLIR_ARMSMEABISTUBS_EXPORTED __arm_za_disable() {
          |                                   ^
    5 errors generated.
    
    Also, Zentrik/llvm-project@5d9a2e2 to avoid conflict. Filed Building ArmSMEStubs.cpp fails on mingw with LLVM_BUILD_LLVM_DYLIB llvm/llvm-project#106899. Workaround at 38b06f4.
  • Zentrik/llvm-project@f84efd8, fixes MLIRExecutionEngineShared doesn't build with mingw due to undefined symbols llvm/llvm-project#106859.
  • Zentrik/llvm-project@3074f16, fixes the following on windows.
    ninja: job failed: /opt/bin/x86_64-w64-mingw32-libgfortran5-cxx11/x86_64-w64-mingw32-clang++ --target=x86_64-w64-mingw32 --sysroot=/opt/x86_64-w64-mingw32/x86_64-w64-mingw32/sys-root/ -D_FILE_OFFSET_BITS=64 -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -I/workspace/build/projects/compiler-rt/lib/fuzzer -I/workspace/srcdir/llvm-project/compiler-rt/lib/fuzzer -I/workspace/build/include -I/workspace/srcdir/llvm-project/llvm/include -I/workspace/srcdir/llvm-project/compiler-rt/lib/fuzzer/../../include -remap -D__USING_SJLJ_EXCEPTIONS__ -D__CRT__NO_INLINE -pthread -DMLIR_CAPI_ENABLE_WINDOWS_DLL_DECLSPEC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wimplicit-fallthrough -Wcovered-switch-default -Wno-noexcept-type -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wsuggest-override -Wstring-conversion -ffunction-sections -fdata-sections -Wall -O3 -DNDEBUG -m64 -fno-builtin -fno-exceptions -fomit-frame-pointer -funwind-tables -fno-stack-protector -fno-sanitize=safe-stack -fvisibility=hidden -fno-lto -O3 -gline-tables-only -fms-extensions -fno-omit-frame-pointer -std=c++17 -MD -MT projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerUtilWindows.cpp.obj -MF projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerUtilWindows.cpp.obj.d -o projects/compiler-rt/lib/fuzzer/CMakeFiles/RTfuzzer.x86_64.dir/FuzzerUtilWindows.cpp.obj -c /workspace/srcdir/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtilWindows.cpp
    clang-16: warning: argument unused during compilation: '-remap' [-Wunused-command-line-argument]
    /workspace/srcdir/llvm-project/compiler-rt/lib/fuzzer/FuzzerUtilWindows.cpp:252:30: error: cannot initialize a parameter of type 'HANDLE' (aka 'void *') with an rvalue of type 'native_handle_type' (aka 'unsigned long long')
            (void)ThreadNameProc(thread.native_handle(), buf.c_str());
                                 ^~~~~~~~~~~~~~~~~~~~~~
    1 error generated.
    
    Also, Zentrik/llvm-project@78d7ef1 to avoid conflict. Filed libFuzzer does not build on mingw with pthreads llvm/llvm-project#106871. Fix up for backporting at Backport 7c4cffd9d8be424e9e9542be9aec3b5a6f69073e llvm/llvm-project#106903.
  • Zentrik/llvm-project@e5a7644 fixed issue on windows where building just errored with no message. Also, Zentrik/llvm-project@08d65dd to avoid conflict. https://github.com/JuliaPackaging/Yggdrasil//commit/90a02367481c9f9ced335067b8612aeb7251e266 fixes this properly.

Also, the problems on non windows seemed to get fixed by these reverts. I would guess the darwin issues were fixed by Zentrik/llvm-project@e5a7644.

I guess issues need to be filed and smaller fixes can probably be found.

L/LLVM/common.jl Outdated Show resolved Hide resolved
@maleadt
Copy link
Contributor

maleadt commented Sep 19, 2024

LLVM 19.1 has been released: https://discourse.llvm.org/t/llvm-19-1-0-released/81285

@giordano
Copy link
Member

Yes, yesterday I rebased https://github.com/JuliaLang/llvm-project/commits/julia-release/19.x/ on top of 19.1.0.

@Zentrik I see you opened a few tickets upstream about all the issues you faced. What do you suggest to do? Wait a few days if there's any movement there? It has been a few weeks already though.... Should we revert the offending commits in our fork for the time being? A proper resolution upstream would definitely be a better outcome though, diverging too much from upstream is problematic.

@Zentrik
Copy link
Contributor Author

Zentrik commented Sep 19, 2024

I'll make some prs upstream and to our fork. Just haven't had the time.

L/LLVM/common.jl Outdated Show resolved Hide resolved
@Zentrik
Copy link
Contributor Author

Zentrik commented Sep 25, 2024

Build for LLVM_full_assert on i686-linux-gnu-cxx03 https://buildkite.com/julialang/yggdrasil/builds/13546#01922635-1e02-4a32-bdef-f3341a6d31a6/6-2596

[23:22:29] /workspace/srcdir/llvm-project/llvm/include/llvm/CodeGenTypes/MachineValueType.h:43:10: fatal error: llvm/CodeGen/GenVT.inc: No such file or directory
[23:22:29]    43 | #include "llvm/CodeGen/GenVT.inc"
[23:22:29]       |          ^~~~~~~~~~~~~~~~~~~~~~~~

A restart will probably fix it, llvm/llvm-project#107687.

@giordano
Copy link
Member

We still need to create the tag on the fork repo, but it's all green here! @vchuravy @gbaraldi would you be able to give a quick look?

@giordano giordano requested a review from maleadt September 26, 2024 21:04
@eschnett
Copy link
Contributor

eschnett commented Oct 2, 2024

LLVM 19.1.1 has been released https://discourse.llvm.org/t/llvm-19-1-1-released/82321

L/LLVM/common.jl Outdated Show resolved Hide resolved
@Zentrik
Copy link
Contributor Author

Zentrik commented Oct 8, 2024

Did this fail to build or is llvm taking too long to clone?

@giordano
Copy link
Member

giordano commented Oct 8, 2024

I cancelled them because libgit2 is insufferably slow, I fetched the repos manually with CLI git on the build machine with

for agent in {0..15}; do cd /julia/agent-cache/yggy-amdci7.${agent}/yggdrasil/downloads/clones/llvm-project.git-5a9787eb535c2edc5dea030cc221c1d60f38c9f42344f410e425ea2139e233aa&&git fetch; done

to make this go (I put the command here so that I remember next time)

@Zentrik
Copy link
Contributor Author

Zentrik commented Oct 8, 2024

I suspect llvm/llvm-project#110382 is an issue in LLVM 19 as well so probably good to wait a couple days to see if it can get fixed.

L/LLVM/LLVM_full@19/build_tarballs.jl Outdated Show resolved Hide resolved
L/LLVM/LLVM_full_assert@19/build_tarballs.jl Outdated Show resolved Hide resolved
@giordano
Copy link
Member

giordano commented Oct 9, 2024

@Zentrik the ittapi patch looks broken

@Zentrik
Copy link
Contributor Author

Zentrik commented Oct 9, 2024

yh JuliaLang/llvm-project#36 breaks the git cloning as ${PROJECT_BINARY_DIR} exists probably

L/LLVM/common.jl Outdated Show resolved Hide resolved
Co-authored-by: Mosè Giordano <[email protected]>
@giordano
Copy link
Member

@Zentrik I still need to create the tag, but apart from that, is this now ready to go or are there other things we should wait for?

@Zentrik
Copy link
Contributor Author

Zentrik commented Oct 12, 2024

@Zentrik I still need to create the tag, but apart from that, is this now ready to go or are there other things we should wait for?

I think this is ready to go.

@giordano giordano merged commit fab2bef into JuliaPackaging:master Oct 12, 2024
31 checks passed
@giordano
Copy link
Member

Thank you so much for all your work!

avik-pal pushed a commit to avik-pal/Yggdrasil that referenced this pull request Oct 25, 2024
* [LLVM] Add LLVM_full, LLVM_full_assert 19 recipes

* Update patch

* Test fixes

* Test another fix

* Try less targeted fix

* Remove assert bundled

* Add assert bundled

* Adapt to llvm/llvm-project#89628

* Workaround llvm/llvm-project#106899

* Include less patches to llvm

* Fix `DMLIR_SRC_SHARDER_TABLEGEN`

* Just set `LLVM_NATIVE_TOOL_DIR` instead

* Trigger rebuilds

* Build off of Julia Fork

* Fix tag

* Set correct version

* Set correct version

* Update L/LLVM/common.jl

* Update L/LLVM/common.jl

* [LLVM] Upgrade to v19.1.1

* Bump LLVM version

* Apply suggestions from code review

* Bump LLVM

Co-authored-by: Mosè Giordano <[email protected]>

* rebuild

* rebuild

---------

Co-authored-by: Zentrik <[email protected]>
Co-authored-by: Mosè Giordano <[email protected]>
Co-authored-by: Mosè Giordano <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants