[boost-modular-build-helper] Bugfix/boost arm64 ios #18124

StefAviator · 2021-05-26T09:40:43Z

Building boost for arm64 ios created x86_64 .a libraries. This is fixed by using a different modular builder, using gcc 12.0.0 and adding compiler options to the user-config.jam file.

Building boost for arm64-ios was creating x86_64 .a output. Fixed by conditionally using a different modular builder, using gcc 12.0.0 and adding the appropriate compiler options to user-config.jam

ghost · 2021-05-26T09:40:55Z

All CLA requirements met.

StefAviator · 2021-05-26T09:42:48Z

ports/boost-modular-build-helper/boost-modular-build.cmake

@@ -99,7 +99,7 @@ function(boost_modular_build)
        vcpkg_install_cmake()
    endfunction()

-    if(VCPKG_CMAKE_SYSTEM_NAME AND NOT VCPKG_CMAKE_SYSTEM_NAME STREQUAL "WindowsStore")
+    if(VCPKG_CMAKE_SYSTEM_NAME AND NOT VCPKG_CMAKE_SYSTEM_NAME STREQUAL "WindowsStore" AND NOT VCPKG_TARGET_ARCHITECTURE STREQUAL "arm64")


To compile arm64 for ios, we need to skip this builder and use the one below which allows the use of the user-config.jam file

This may need to limit to ios?

Ok agreed, I will check for VCPKG_TARGET_ARCHITECTURE.

@StefAviator, thank you for this PR! The code path below seems to be tailored specifically for Windows, e.g., it doesn't seem to copy over any .dylib files as opposed to the CMakeLists.txt file. The unix_build function goes through the CMakeLists.txt file which does seem to use the user-config.jam file:

add_custom_target(boost ALL COMMAND "${B2_EXE}" --user-config=${CMAKE_CURRENT_BINARY_DIR}/user-config.jam

Were there any errors using the the unix build?

Thanks for the feedback, I will try out your suggestions.

StefAviator · 2021-05-26T09:43:46Z

ports/boost-modular-build-helper/user-config.jam

 else
 {
-    using gcc : 5.4.1 : @CMAKE_CXX_COMPILER@
+    using gcc : 12.0.0 : @CMAKE_CXX_COMPILER@


Updating to the latest gcc.

StefAviator · 2021-05-26T09:43:50Z

ports/boost-modular-build-helper/user-config.jam

+        # MINGW here causes b2 to not run cygpath
+        <flavor>mingw
+        <compileflags>"-std=c++14 -stdlib=libc++ -DNDEBUG -arch arm64 -fembed-bitcode -Wno-unused-local-typedef -Wno-nullability-completeness -mmacosx-version-min=10.12 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk"
+        ;        


This compilerflag is needed to compile arm64 for iOS

There's a way to specify the iOS target via the CMakeLists.txt file, which could remove the need to change the user-config.jam. Specifically the line that specifies the target-os option:

if(APPLE) list(APPEND B2_OPTIONS target-os=darwin toolset=clang) elseif(WIN32) list(APPEND B2_OPTIONS target-os=windows toolset=gcc) elseif(ANDROID)

For iOS, that would be target-os=iphone docs

It would also be nice to make this work for both arm64 iOS and macOS builds, i.e. the arm64-ios and arm64-osx triplets:

set(VCPKG_TARGET_ARCHITECTURE arm64) set(VCPKG_CMAKE_SYSTEM_NAME iOS)

and

set(VCPKG_TARGET_ARCHITECTURE arm64) set(VCPKG_CMAKE_SYSTEM_NAME Darwin)

Boosts's b2 doesn't natively support arm64, so the extra -arch arm64 can be added to the CMakeLists.txt config like so:

if(APPLE AND VCPKG_TARGET_ARCHITECTURE STREQUAL "arm64") set(CFLAGS "${CFLAGS} <compileflags>-arch <compileflags>arm64") set(CXXFLAGS "${CXXFLAGS} <compileflags>-arch <compileflags>arm64") set(LDFLAGS "${LDFLAGS} <linkflags>-arch <linkflags>arm64") endif()

I have tried adding these compilerflags to CMakeLists.txt but it keeps outputting x86-64 library files so unfortunately it does not work. I tried this change using both unix_build and the second builder. If you know how to get this to work to cross-compile on an Apple PC/laptop with Intel chip to arm64 by adjusting the CMakeLists file, please share.

PhoebeHui · 2021-05-27T09:49:39Z

@StefAviator, thanks for the PR! Since the triplet doesn't cover in CI testing, could you test it locally?

PhoebeHui

Could you also update the port version to '10' in https://github.com/microsoft/vcpkg/blob/master/ports/boost-modular-build-helper/vcpkg.json#L4? Submit the changes and execute './vcpkg x-add-version boost-modular-build-helper' to update the baseline version, then submit the changes again.

PhoebeHui · 2021-05-27T09:50:18Z

ports/boost-modular-build-helper/boost-modular-build.cmake

@@ -99,7 +99,7 @@ function(boost_modular_build)
        vcpkg_install_cmake()
    endfunction()

-    if(VCPKG_CMAKE_SYSTEM_NAME AND NOT VCPKG_CMAKE_SYSTEM_NAME STREQUAL "WindowsStore")
+    if(VCPKG_CMAKE_SYSTEM_NAME AND NOT VCPKG_CMAKE_SYSTEM_NAME STREQUAL "WindowsStore" AND NOT VCPKG_TARGET_ARCHITECTURE STREQUAL "arm64")


This may need to limit to ios?

StefAviator · 2021-05-28T08:19:26Z

@StefAviator, thanks for the PR! Since the triplet doesn't cover in CI testing, could you test it locally?

On my MacBook Pro, building boost components with commands such as
./vcpkg install boost-exception:arm64-ios
now outputs .a files which show 'arm64' when checked with 'lipo -info'. Previously it would show as 'x86-64'. Is that enough for testing purposes? If I need to attach any log files let me know.

I don't have a device with M1 chipset currently but would appreciate if someone can confirm the output. I have asked someone in the original issue.

vejmartin · 2021-05-29T08:33:56Z

ports/boost-modular-build-helper/boost-modular-build.cmake

@@ -99,7 +99,7 @@ function(boost_modular_build)
        vcpkg_install_cmake()
    endfunction()

-    if(VCPKG_CMAKE_SYSTEM_NAME AND NOT VCPKG_CMAKE_SYSTEM_NAME STREQUAL "WindowsStore")
+    if(VCPKG_CMAKE_SYSTEM_NAME AND NOT VCPKG_CMAKE_SYSTEM_NAME STREQUAL "WindowsStore" AND NOT VCPKG_TARGET_ARCHITECTURE STREQUAL "arm64")


@StefAviator, thank you for this PR! The code path below seems to be tailored specifically for Windows, e.g., it doesn't seem to copy over any .dylib files as opposed to the CMakeLists.txt file. The unix_build function goes through the CMakeLists.txt file which does seem to use the user-config.jam file:

add_custom_target(boost ALL COMMAND "${B2_EXE}" --user-config=${CMAKE_CURRENT_BINARY_DIR}/user-config.jam

Were there any errors using the the unix build?

vejmartin · 2021-05-29T08:47:09Z

ports/boost-modular-build-helper/user-config.jam

+        # MINGW here causes b2 to not run cygpath
+        <flavor>mingw
+        <compileflags>"-std=c++14 -stdlib=libc++ -DNDEBUG -arch arm64 -fembed-bitcode -Wno-unused-local-typedef -Wno-nullability-completeness -mmacosx-version-min=10.12 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.1.sdk"
+        ;        


There's a way to specify the iOS target via the CMakeLists.txt file, which could remove the need to change the user-config.jam. Specifically the line that specifies the target-os option:

if(APPLE) list(APPEND B2_OPTIONS target-os=darwin toolset=clang) elseif(WIN32) list(APPEND B2_OPTIONS target-os=windows toolset=gcc) elseif(ANDROID)

For iOS, that would be target-os=iphone docs

It would also be nice to make this work for both arm64 iOS and macOS builds, i.e. the arm64-ios and arm64-osx triplets:

set(VCPKG_TARGET_ARCHITECTURE arm64) set(VCPKG_CMAKE_SYSTEM_NAME iOS)

and

set(VCPKG_TARGET_ARCHITECTURE arm64) set(VCPKG_CMAKE_SYSTEM_NAME Darwin)

Boosts's b2 doesn't natively support arm64, so the extra -arch arm64 can be added to the CMakeLists.txt config like so:

if(APPLE AND VCPKG_TARGET_ARCHITECTURE STREQUAL "arm64") set(CFLAGS "${CFLAGS} <compileflags>-arch <compileflags>arm64") set(CXXFLAGS "${CXXFLAGS} <compileflags>-arch <compileflags>arm64") set(LDFLAGS "${LDFLAGS} <linkflags>-arch <linkflags>arm64") endif()

StefAviator · 2021-05-31T12:41:07Z

I have added a check for either 'iOS' or 'Darwin' system (the latter per @vejmartin's suggestion) to use this build fix. Now it also supports cross-compiling for arm64 osx. I was not able to get it to work using adjustments to the CMakeLists.txt file that @vejmartin suggested so it is necessary to keep using the user-config.jam file update. The functionality can be confirmedted as such:

./vcpkg install boost-exception:arm64-ios boost-atomic:arm64-osx
-- (build)
~/vcpkg $ lipo -info installed/arm64-ios/lib/libboost_exception.a 
Non-fat file: installed/arm64-ios/lib/libboost_exception.a is architecture: arm64
~/vcpkg $ lipo -info installed/arm64-osx/lib/libboost_atomic.a   
Non-fat file: installed/arm64-osx/lib/libboost_atomic.a is architecture: arm64

We have had feedback that using the current implementation (vcpkg master) works for compiling on an arm64 system (M1 chip macbook) to create arm64 boost library files. So this fix is limited to fixing cross-compiling from intel Apple PC/laptop (x86-64) to arm64 boost libraries. As such it might be interesting to check if the host Apple system is arm64, in which case this fix of using user-config.jam file can be bypassed. I would appreciate some feedback if this is deemed necessary and how this can be done. Thanks.

StefAviator · 2021-05-31T13:29:54Z

ports/boost-modular-build-helper/user-config.jam

+        @LDFLAGS@
+        # MINGW here causes b2 to not run cygpath
+        <flavor>mingw
+        <compileflags>"-std=c++14 -stdlib=libc++ -DNDEBUG -arch arm64 -fembed-bitcode -Wno-unused-local-typedef -Wno-nullability-completeness -mmacosx-version-min=10.12 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk"


Here we are using the default MacOS sdk on the system. "Darwin" above is used in the triplet for unix / osx / MacOS.

StefAviator · 2021-05-31T13:35:02Z

Could you also update the port version to '10' in https://github.com/microsoft/vcpkg/blob/master/ports/boost-modular-build-helper/vcpkg.json#L4? Submit the changes and execute './vcpkg x-add-version boost-modular-build-helper' to update the baseline version, then submit the changes again.

@PhoebeHui Ok done.

vejmartin · 2021-05-31T14:46:53Z

@StefAviator, I think the reason why the options weren't being picked up by the user-config might have been that the CMakeLists.txt file sets the toolset to clang:

list(APPEND B2_OPTIONS target-os=darwin toolset=clang)

while, the user-config doesn't define a clang toolset (only gcc):

using gcc : 5.4.1 : @CMAKE_CXX_COMPILER@

By conditionally converting the gcc toolset to clang, the options from CMakeLists.txt are picked up again:

if "@CMAKE_CXX_COMPILER_ID@" in "Clang" "AppleClang" 
{
    compiler_name = "clang" ;
    compiler_version = "@CMAKE_CXX_COMPILER_VERSION@" ;
}
else
{
    compiler_name = "gcc" ;
    compiler_version = "5.4.1" ;
}
using $(compiler_name) : $(compiler_version) : @CMAKE_CXX_COMPILER@

Please also note that the user might specify a custom SDK root via the SDKROOT environment variable.
Here's a working example with CMakeLists.txt for arm64 iOS/macOS: vejmartin@18d39b9

StefAviator · 2021-06-10T11:45:55Z

Here's a working example with CMakeLists.txt for arm64 iOS/macOS: vejmartin@18d39b9

@vejmartin Thanks for the code. Unfortunately it doesn't work on my system. I have both gcc and clang installed. If I use your code as written, it still uses the gcc compiler and outputs x86_64 libraries, the original problem. If I remove the 'if' statement and force it to use clang in boost-modular-build.cmake, it ends with error. I can see in the logs (attached) that it starts off using clang, but then proceeds with gcc anyway. Possibly B2 uses gcc, but that part is unclear.

build-arm64-ios-rel-out.log

So far my implementation using the user-config file is the only solution that allows cross-compiling from x86 system to output boost arm64 libs on my system. If you have any suggestions feel free to share.

vejmartin · 2021-06-10T22:52:38Z

@StefAviator, thank you for looking into and testing my example.

Thank you also for the build log. I'm assuming build-arm64-ios-rel-out.log is from using boost-modular-build.cmake on macOS, as the CMakeLists.txt method produces install-arm64-ios-rel-out.log instead.

Unfortunately it doesn't work on my system.

Could you please attach the install-arm64-ios-rel-out.log log when using the CMakeLists.txt method or any other errors?

If we could work out what's causing the error when using the CMakeLists.txt method, we could ensure there won't be two separate files/methods for building arm64 and x86_64 on macOS.

As boost-modular-build.cmake isn't currently used to build on x64-osx*, it doesn't support dynamic libraries. You can verify that by building boost-system:arm64-osx-dynamic (please note that boost-exception always produces a static lib). This should produce the following file: install/arm64-osx-dynamic/lib/libboost_system.dylib.

I also found the following comment useful for debugging: boostorg/build#716 (comment):

run b2 with -d+2 flag to make logs useful. A log with --debug-building may later be required to debug too.

You could add these two flags to the build command in CMakeLists.txt (add_custom_target) to get more info in the logs.

vejmartin · 2021-06-10T23:00:13Z

These issues also seem to be related, #16630, #11132

PhoebeHui · 2021-07-07T02:24:15Z

@StefAviator @vejmartin, is work still being done for this PR?

StefAviator · 2021-07-08T11:15:00Z

@PhoebeHui Yes this is still ongoing. Unfortunately I was not able to get @vejmartin 's suggestions to work. That is to say, while I would prefer to do it his way, it still only outputs x86_64 libraries instead of arm64. I will do one more update this week to address an issue and then let david-antiteum test it (he has arm64 chip mac) and then it can be included as i believe there is currently no other working version that outputs iOS arm64.

vejmartin · 2021-07-13T06:37:56Z

@StefAviator, a fix for the toolchain options was recently merged, #18529, which also fixes arm64-osx/ios builds for me.

StefAviator · 2021-07-19T13:07:56Z

@vejmartin @PhoebeHui This is indeed fixed by #18529 for arm64-osx/ios builds, confirmed as well on my system, they produce arm64 libraries. We can close this PR as it's not needed anymore.

Thanks vejmartin for your assistance.

PhoebeHui · 2021-07-20T00:47:21Z

@vejmartin @StefAviator, thanks for conforming this!

Stefan Segers added 2 commits May 26, 2021 11:32

bugfix/boost_arm64_ios

ef00875

Building boost for arm64-ios was creating x86_64 .a output. Fixed by conditionally using a different modular builder, using gcc 12.0.0 and adding the appropriate compiler options to user-config.jam

Merge branch 'master' of https://github.com/microsoft/vcpkg

b9a8813

StefAviator commented May 26, 2021

View reviewed changes

Stefan Segers added 2 commits May 26, 2021 17:01

using iPhoneOS.sdk

94f9511

adding iOS condition

21c8181

JonLiu1993 assigned PhoebeHui May 27, 2021

PhoebeHui changed the title ~~Bugfix/boost arm64 ios~~ [boost-modular-build-helper] Bugfix/boost arm64 ios May 27, 2021

PhoebeHui added category:community-triplet A PR or issue related to community triplets not officially validated by the vcpkg team. category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist labels May 27, 2021

PhoebeHui requested changes May 27, 2021

View reviewed changes

PhoebeHui added the requires:author-response label May 27, 2021

vejmartin suggested changes May 29, 2021

View reviewed changes

Restrict change to system iOS and added OSX fix

2186fa2

StefAviator commented May 31, 2021

View reviewed changes

StefAviator added 2 commits May 31, 2021 15:32

update port version to 10

fd750d5

update port version part 2

778c474

PhoebeHui closed this Jul 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[boost-modular-build-helper] Bugfix/boost arm64 ios #18124

[boost-modular-build-helper] Bugfix/boost arm64 ios #18124

StefAviator commented May 26, 2021

ghost commented May 26, 2021 •

edited by ghost

Loading

StefAviator May 26, 2021

PhoebeHui May 27, 2021

StefAviator May 28, 2021

vejmartin May 29, 2021

wintergt May 31, 2021

StefAviator May 26, 2021

StefAviator May 26, 2021

vejmartin May 29, 2021

StefAviator May 31, 2021 •

edited

Loading

PhoebeHui commented May 27, 2021

PhoebeHui left a comment •

edited

Loading

PhoebeHui May 27, 2021

StefAviator commented May 28, 2021

vejmartin May 29, 2021

vejmartin May 29, 2021

StefAviator commented May 31, 2021

StefAviator May 31, 2021

StefAviator commented May 31, 2021 •

edited

Loading

vejmartin commented May 31, 2021

StefAviator commented Jun 10, 2021

vejmartin commented Jun 10, 2021

vejmartin commented Jun 10, 2021

PhoebeHui commented Jul 7, 2021

StefAviator commented Jul 8, 2021

vejmartin commented Jul 13, 2021

StefAviator commented Jul 19, 2021 •

edited

Loading

PhoebeHui commented Jul 20, 2021

[boost-modular-build-helper] Bugfix/boost arm64 ios #18124

[boost-modular-build-helper] Bugfix/boost arm64 ios #18124

Conversation

StefAviator commented May 26, 2021

ghost commented May 26, 2021 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StefAviator May 31, 2021 • edited Loading

Choose a reason for hiding this comment

PhoebeHui commented May 27, 2021

PhoebeHui left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StefAviator commented May 28, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

StefAviator commented May 31, 2021

Choose a reason for hiding this comment

StefAviator commented May 31, 2021 • edited Loading

vejmartin commented May 31, 2021

StefAviator commented Jun 10, 2021

vejmartin commented Jun 10, 2021

vejmartin commented Jun 10, 2021

PhoebeHui commented Jul 7, 2021

StefAviator commented Jul 8, 2021

vejmartin commented Jul 13, 2021

StefAviator commented Jul 19, 2021 • edited Loading

PhoebeHui commented Jul 20, 2021

ghost commented May 26, 2021 •

edited by ghost

Loading

StefAviator May 31, 2021 •

edited

Loading

PhoebeHui left a comment •

edited

Loading

StefAviator commented May 31, 2021 •

edited

Loading

StefAviator commented Jul 19, 2021 •

edited

Loading