Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] cross-compilation issues with 18.0.0rc0: inclusion of <nmmintrin.h> resp. mis-detection of grpc_cpp_plugin #44448

Closed
h-vetinari opened this issue Oct 16, 2024 · 37 comments

Comments

@h-vetinari
Copy link
Contributor

Describe the bug, including details regarding any error messages, version, and platform.

I'm testing 18.0.0rc0 in conda-forge, and getting a compilation error on osx-arm64 (which for us is cross-compiled from osx-64). It seems something is going wrong in the following code

# if defined(ARROW_HAVE_AVX2) || defined(ARROW_HAVE_AVX512) || \
defined(ARROW_HAVE_RUNTIME_AVX2) || defined(ARROW_HAVE_RUNTIME_AVX512)
# include <immintrin.h>
# elif defined(ARROW_HAVE_SSE4_2) || defined(ARROW_HAVE_RUNTIME_SSE4_2)
# include <nmmintrin.h>
# endif

in the sense that this tries to include <nmmintrin.h> in cross-compilation (probably the HAVE_RUNTIME_X is based on the architecture of the agent?), despite the target architecture having no support for that.

[124/491] Building CXX object src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o
FAILED: src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o 
$BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang++ -DARROW_HAVE_RUNTIME_SSE4_2 -DARROW_WITH_TIMING_TESTS -DBOOST_ALL_DYN_LINK -DGFLAGS_IS_A_DLL=0 -DGLOG_USE_GFLAGS -DGLOG_USE_GLOG_EXPORT -DURI_STATIC_BUILD -I$SRC_DIR/cpp/build/src -I$SRC_DIR/cpp/src -I$SRC_DIR/cpp/src/generated -fno-aligned-new -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-18.0.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -D_LIBCPP_DISABLE_AVAILABILITY -Qunused-arguments -fcolor-diagnostics  -Wall -Wno-unknown-warning-option -Wno-pass-failed -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-18.0.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -D_LIBCPP_DISABLE_AVAILABILITY -O3 -DNDEBUG -O2  -std=c++17 -isysroot /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.0.sdk -mmacosx-version-min=11.0 -fPIC -MD -MT src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o -MF src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o.d -o src/arrow/CMakeFiles/arrow_util.dir/util/utf8.cc.o -c $SRC_DIR/cpp/src/arrow/util/utf8.cc
In file included from $SRC_DIR/cpp/src/arrow/util/utf8.cc:28:
In file included from $SRC_DIR/cpp/src/arrow/util/utf8_internal.h:33:
In file included from $SRC_DIR/cpp/src/arrow/util/simd.h:38:
$BUILD_PREFIX/lib/clang/18/include/nmmintrin.h:14:2: error: "This header is only meant to be used on x86 and x64 architecture"
   14 | #error "This header is only meant to be used on x86 and x64 architecture"
      |  ^

Last relevant-looking change in that area is 87d6477 CC @zanmato1984 @pitrou

Component(s)

C++, Packaging

@kou
Copy link
Member

kou commented Oct 17, 2024

Could you share the actual CMake options used for the cross-compilation?

@h-vetinari
Copy link
Contributor Author

h-vetinari commented Oct 17, 2024

Sure. This is the CMake call as extracted (and formatted for legibility) from the logs:

cmake -GNinja \
    -DARROW_ACERO=ON \
    -DARROW_AZURE=ON \
    -DARROW_BOOST_USE_SHARED=ON \
    -DARROW_BUILD_BENCHMARKS=OFF \
    -DARROW_BUILD_STATIC=OFF \
    -DARROW_BUILD_TESTS=OFF \
    -DARROW_BUILD_UTILITIES=OFF \
    -DARROW_COMPUTE=ON \
    -DARROW_CSV=ON \
    -DARROW_CXXFLAGS='-ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -fmessage-length=0 -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/apache-arrow-18.0.0 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -D_LIBCPP_DISABLE_AVAILABILITY' \
    -DARROW_DATASET=ON \
    -DARROW_DEPENDENCY_SOURCE=SYSTEM \
    -DARROW_FILESYSTEM=ON \
    -DARROW_FLIGHT=ON \
    -DARROW_FLIGHT_REQUIRE_TLSCREDENTIALSOPTIONS=ON \
    -DARROW_FLIGHT_SQL=ON \
    -DARROW_GANDIVA=ON \
    -DARROW_GANDIVA_PC_CXX_FLAGS=-D_LIBCPP_DISABLE_AVAILABILITY \
    -DARROW_GCS=ON \
    -DARROW_GDB_INSTALL_DIR=replace_this_section_with_absolute_slashed_path_to_CONDA_PREFIX/lib \
    -DARROW_HDFS=ON \
    -DARROW_JEMALLOC=ON \
    -DARROW_JSON=ON \
    -DARROW_MIMALLOC=ON \
    -DARROW_ORC=ON \
    -DARROW_PACKAGE_PREFIX=$PREFIX \
    -DARROW_PARQUET=ON \
    -DARROW_S3=ON \
    -DARROW_SIMD_LEVEL=NONE \
    -DARROW_SUBSTRAIT=ON \
    -DARROW_USE_GLOG=ON \
    -DARROW_USE_LD_GOLD=ON \
    -DARROW_WITH_BROTLI=ON \
    -DARROW_WITH_BZ2=ON \
    -DARROW_WITH_LZ4=ON \
    -DARROW_WITH_NLOHMANN_JSON=ON \
    -DARROW_WITH_OPENTELEMETRY=OFF \
    -DARROW_WITH_SNAPPY=ON \
    -DARROW_WITH_UCX=OFF \
    -DARROW_WITH_ZLIB=ON \
    -DARROW_WITH_ZSTD=ON \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_BUILD_TYPE=release \
    -DCMAKE_CXX_STANDARD=17 \
    -DCMAKE_INSTALL_LIBDIR=lib \
    -DCMAKE_INSTALL_PREFIX=$PREFIX \
    -DLLVM_TOOLS_BINARY_DIR=$PREFIX/bin \
    -DMAKE=$BUILD_PREFIX/bin/make \
    -DPARQUET_REQUIRE_ENCRYPTION=ON \
    -DPython3_EXECUTABLE= \
    -DARROW_CUDA=OFF \
    -DProtobuf_PROTOC_EXECUTABLE=$BUILD_PREFIX/bin/protoc \
    -DCLANG_EXECUTABLE=$BUILD_PREFIX/bin/arm64-apple-darwin20.0.0-clang \
    -DLLVM_LINK_EXECUTABLE=$BUILD_PREFIX/bin/llvm-link \
    -DARROW_JEMALLOC_LG_PAGE=16 \
    ..

@zanmato1984
Copy link
Collaborator

Hi @h-vetinari , thanks for letting us know. I wonder if this is a regression? (That is, the same build used to pass before?) Thanks.

@h-vetinari
Copy link
Contributor Author

Yes, definitely a regression. Conda-forge has been cross-compiling osx-arm64 from osx-64 since arrow v0.17, and never ran into this (to my knowledge; some of it was before my time).

@zanmato1984
Copy link
Collaborator

zanmato1984 commented Oct 17, 2024

I think here is a possible temporary workaround: adding an extra CMake option -DARROW_RUNTIME_SIMD_LEVEL=NONE, if you want to proceed with anything you originally intended. (Also, please let me know if it works with your current problem.)

But I can't promise this would be the final solution, which I'll have to talk to community about and come up with. We'll keep you posted of course.

Thanks.

@zanmato1984
Copy link
Collaborator

zanmato1984 commented Oct 17, 2024

OK. By looking at the log more carefully I seem to get to the root cause.

From the log, there is a preset CMAKE_ARGS:

2024-10-17T00:53:32.7864890Z +CMAKE_ARGS= ... -DCMAKE_SYSTEM_PROCESSOR=arm64 ...

This is well expected for a cross-compiling config.

Then from the output of Arrow CMake:

2024-10-17T00:53:45.7948650Z -- System processor: x86_64

This is printed by:

message(STATUS "System processor: ${CMAKE_SYSTEM_PROCESSOR}")

For some unknown reason the preset -DCMAKE_SYSTEM_PROCESSOR=arm64 didn't take effect. And it's not surprising that the subsequent SIMD options are set for x86 platform and eventually lead to the erroring header.

UPDATE: The above has been confirmed to be the root cause but the following guess I original posted is NOT the cure.

With a little googling there are discussions seemed to be related:

I had a similar issue. The answer is that when CMAKE_SYSTEM_NAME is not set, CMake overwrites CMAKE_SYSTEM_PROCESSOR (source code). This is arguably a bug but regardless is how CMake behaves.

I can locally reproduce the questioning CMake behavior on my M1 Mac CMake version 3.25.2:

cmake -DCMAKE_SYSTEM_PROCESSOR=x86_64 ..
-- Building using CMake version: 3.25.2
...
-- System processor: arm64
cmake -DCMAKE_SYSTEM_NAME=Darwin -DCMAKE_SYSTEM_PROCESSOR=x86_64 ..
-- Building using CMake version: 3.25.2
...
-- System processor: x86_64

Worth a try and hope this solves your issue, @h-vetinari . Thanks.

@h-vetinari
Copy link
Contributor Author

h-vetinari commented Oct 23, 2024

So while we solved the CMake bits for not picking up nmmintrin.h, we still have broken cross-compilation currently. This should IMO be considered a blocker, as conda-forge won't be able to release arrow 18 in this state. If you change the milestone on an issue I raise for an rc build without a comment, I don't even get a notification...

I noted on the conda-forge side:

One thing that looks weird to me compared to other such CMake errors is:

+ cmake --build . --config Release
ninja: error: 'src/arrow/flight/gRPC::grpc_cpp_plugin', needed by 'src/arrow/flight/Flight.pb.cc', missing and no known rule to make it
               ^^^^^^^^^^^^^^^^^
               ?????????????????

AFAIU this should be detected/used as gRPC::grpc_cpp_plugin. Perhaps something is going wrong in https://github.com/apache/arrow/blob/main/cpp/cmake_modules/FindgRPCAlt.cmake ...?

We have correct CMake metadata for grpc (we're testing this as part of the feedstock), and so I suspect there's something going wrong with the custom processing in FindgRPCAlt. I've tried to read that file (also cpp/cmake_modules/ThirdpartyToolchain.cmake) to see where things might go wrong, and I've tried disabling its use, but to no avail.

@kou
Copy link
Member

kou commented Oct 23, 2024

Could you share the build log URL that includes the error message?

@kou
Copy link
Member

kou commented Oct 23, 2024

Does https://github.com/conda-forge/grpc-cpp-feedstock provide gRPC::grpc_cpp_plugin target?

@h-vetinari
Copy link
Contributor Author

Could you share the build log URL that includes the error message?

https://dev.azure.com/conda-forge/feedstock-builds/_build/results?buildId=1060564&view=results

@h-vetinari
Copy link
Contributor Author

Does https://github.com/conda-forge/grpc-cpp-feedstock provide gRPC::grpc_cpp_plugin target?

The plugin definitely gets installed; we haven't modified the CMake installation targets, so it's going to be whatever grpc puts in there by default (so I assume gRPC::grpc_cpp_plugin to be there, but haven't checked that).

@kou
Copy link
Member

kou commented Oct 23, 2024

It seems that https://anaconda.org/conda-forge/libgrpc/1.67.0/download/osx-64/libgrpc-1.67.0-h0fbbd33_0.conda provides gRPC CMake package and gRPCPlugin CMake package. And gRPCPlugin CMake package provides gRPC::grpc_cpp_plugin.

@kou
Copy link
Member

kou commented Oct 23, 2024

Apache Arrow C++ uses only gRPC CMake package. So we don't have gRPC::grpc_cpp_plugin.

@h-vetinari
Copy link
Contributor Author

Yeah, that makes sense. Though again, we're using the default CMake install of upstream grpc, that split does not come from conda-forge. So in that sense, it would be good for arrow to be(come) compatible with it.

@kou
Copy link
Member

kou commented Oct 23, 2024

grpc/grpc@831d2a6
It seems that gRPC changed CMake package since 1.52.0 that was released on 2023-02-04.

@h-vetinari
Copy link
Contributor Author

h-vetinari commented Oct 23, 2024

I did a quick check on our package content, and also found that this was introduced in 1.52... :)

We've been using grpc newer than that in conda-forge for a long time already, so it seems to have worked by accident somehow (or perhaps there were some compatibility targets in the old CMake module that eventually got removed).

@kou
Copy link
Member

kou commented Oct 23, 2024

https://packages.ubuntu.com/search?keywords=libgrpc
Ubuntu 24.04 ships gRPC 1.51.1. So our CI doesn't use system gRPC 1.52.0 or later yet.

@h-vetinari
Copy link
Contributor Author

h-vetinari commented Oct 23, 2024

Ubuntu 24.04 ships gRPC 1.51.1.

Wow, that was already 14 months old when 24.04 got released 😱

Edit: Actually, it seems ubuntu stopped updating libgrc. Even in oracular it's still on 1.51

@kou
Copy link
Member

kou commented Oct 23, 2024

Ah, wait.
It seems that gRPC CMake package includes gRPCPluginTargets.cmake automatically EXCEPT CROSS_COMPILING:

$ cat libgrpc-1.67.0-h0fbbd33_0/lib/cmake/grpc/gRPCConfig.cmake
# Module path
list(APPEND CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR}/modules)

# Depend packages
if(NOT ZLIB_FOUND)
  find_package(ZLIB)
endif()
include(CMakeFindDependencyMacro)
find_dependency(Protobuf CONFIG)
if(NOT OPENSSL_FOUND)
  find_package(OpenSSL)
endif()
if(NOT c-ares_FOUND)
  find_package(c-ares)
endif()
if(NOT TARGET absl::strings)
  find_package(absl CONFIG)
endif()
if(NOT re2_FOUND)
  find_package(re2)
endif()


# Targets
include(${CMAKE_CURRENT_LIST_DIR}/gRPCTargets.cmake)
if(NOT CMAKE_CROSSCOMPILING)
  include(${CMAKE_CURRENT_LIST_DIR}/gRPCPluginTargets.cmake)
endif()

@kou
Copy link
Member

kou commented Oct 23, 2024

It seems that https://anaconda.org/conda-forge/libgrpc/1.67.0/download/osx-64/libgrpc-1.67.0-h0fbbd33_0.conda provides gRPC CMake package and gRPCPlugin CMake package. And gRPCPlugin CMake package provides gRPC::grpc_cpp_plugin.

I was wrong. There is only gRPC CMake package. It just has gRPCTargets.cmake and gRPCPluginTargets.cmake.

@kou
Copy link
Member

kou commented Oct 23, 2024

Does conda really need to use cross-compiling?
In general, executables (grpc_cpp_plugin in this case) on host isn't usable in cross-compiling.

@h-vetinari
Copy link
Contributor Author

h-vetinari commented Oct 23, 2024

It seems that gRPC CMake package includes gRPCPluginTargets.cmake automatically EXCEPT CROSS_COMPILING:

Yeah, that makes sense... It's not possible to use the plugins for the target architecture. But in our case we actually supply both, i.e. there's a libgrpc in the build environment (where we can execute the plugins), and then one in the host environment (where we don't need the plugins but need to match the target architecture).

Does conda really need to use cross-compiling?

Yes, and this is impossible to change. We don't have native runners for linux-aarch64, linux-ppc64le or osx-arm64, but they are important platforms we support, so the only way to do it is to cross-compile (as mentioned above, we've been doing this since arrow v0.17).

It would be easy on our end to provide some CMake variable, for example -DGRPC_PLUGIN=$BUILD_PREFIX/bin/grpc_cpp_plugin (and this is how it works for many other such situations). It's also where things like Protobuf_PROTOC_EXECUTABLE come from; i.e. it can be explicitly overridden in cross-compilation scenarios.

@kou
Copy link
Member

kou commented Oct 23, 2024

Hmm. Why can we cross-compile 17.0.0...? We don't have any changes around it...

Anyway, could you try GH-44448?

@kou kou changed the title [18.0.0rc0] BUG: cross-compiling osx-arm64 fails due to inclusion of <nmmintrin.h> [C++] cross-compiling osx-arm64 fails due to inclusion of <nmmintrin.h> with 18.0.0rc0 Oct 23, 2024
@h-vetinari
Copy link
Contributor Author

Hmm. Why can we cross-compile 17.0.0...? We don't have any changes around it...

I think our CMake got confused and didn't actually enter into the NOT CMAKE_CROSSCOMPILING, so the target got found. Now that we fixed the CMake confusion (which became necessary due to the failure at the top of the ticket), the target suddenly was gone...

@h-vetinari
Copy link
Contributor Author

h-vetinari commented Oct 23, 2024

Anyway, could you try GH-44507?

Thanks a lot! Started a new job

Update: it works! 🥳

@h-vetinari h-vetinari changed the title [C++] cross-compiling osx-arm64 fails due to inclusion of <nmmintrin.h> with 18.0.0rc0 [C++] cross-compilation issues with 18.0.0rc0: inclusion of <nmmintrin.h> resp. mis-detection of grpc_cpp_plugin Oct 23, 2024
kou added a commit that referenced this issue Oct 23, 2024
…cross-compiling (#44507)

### Rationale for this change

We can't use `find_package(gRPC)` and `gRPC::grpc_cpp_plugin` for cross-compiling because it's for host. We need `grpc_cpp_plugin` for target in cross-compiling.

### What changes are included in this PR?

Add `ARROW_GRPC_CPP_PLUGIN` CMake option that overwrites `gRPC::grpc_cpp_plugin` path found by `find_package(gRPC)`.

### Are these changes tested?

Yes.

conda-forge/arrow-cpp-feedstock#1432

### Are there any user-facing changes?

Yes.
* GitHub Issue: #44448

Authored-by: Sutou Kouhei <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@kou
Copy link
Member

kou commented Oct 23, 2024

Issue resolved by pull request 44507
#44507

@kou kou closed this as completed Oct 23, 2024
@kou
Copy link
Member

kou commented Oct 23, 2024

I don't think that this is a blocker of 18.0.0 RC0 because:

  • <nmmintrin.h> is cross-compiling problem by conda side
  • Cross-compiling didn't work with old releases (It worked accidentally.)

Please patch #44507 for 18.0.0 if we use RC0 for 18.0.0. If we need RC1, we may add #44507 to 18.0.0.

@h-vetinari
Copy link
Contributor Author

I don't think that this is a blocker of 18.0.0 RC0 because:

  • [...] Cross-compiling didn't work with old releases (It worked accidentally.)

Accident or not, it worked for years. When a major distribution channel ends up being broken, that should be the definition of a release blocker, even though we'll manage to backport the patch (again, thanks for that). It's not about the effort to backport, it's about what role conda-forge has. With several million downloads, the arrow feedstock should IMO be important enough to be a release consideration (that's originally why the conda-forge CI was integrated into this repo...)

@kou
Copy link
Member

kou commented Oct 23, 2024

It seems that we need to define support level or something like other projects.
For example, problems in the support level A environments must be blockers, problems in the support level B environments may be blockers, the support level C is best effort and so on. The support level A environments' CI must be prepared.

We may not be able to support all environments because we have limited maintenance resources.

How about discuss this on the mailing list? https://arrow.apache.org/community/#mailing-lists
Could you open a discussion thread?

@raulcd
Copy link
Member

raulcd commented Oct 24, 2024

If there would be no patch for this, I would consider the issue a release blocker. The release is blocked until we have a patch. Mainly because breaking conda-forge being such a big distribution channel is a release blocker for me.

In my opinion on how to proceed now that we have a patch and due to the current effort on creating a new Release Candidate with the current limited maintenance resources we have, I feel it's reasonable to port the patch on conda and release RC0.

@pitrou
Copy link
Member

pitrou commented Oct 24, 2024

As a side note, it may be nice to have at least one CI build to exercise cross-compiling (perhaps as a Crossbow build?). That needs someone to set up, though.

@raulcd
Copy link
Member

raulcd commented Oct 24, 2024

it may be nice to have at least one CI build to exercise cross-compiling

I've opened an issue to track that #44531

@h-vetinari
Copy link
Contributor Author

As a side note, it may be nice to have at least one CI build to exercise cross-compiling (perhaps as a Crossbow build?).

The in-repo conda(-forge) CI does cross compilation already. Not sure what's the state of that though, presumably unmaintained?

@pitrou
Copy link
Member

pitrou commented Oct 24, 2024

Yes, it's unmaintained because it comes with the entire cognitive overhead of conda-forge that none of us is familiar with (except you ;-)).

@h-vetinari
Copy link
Contributor Author

People like Uwe and Keith would equally know how to do it, and I would mentor anyone who invests time on the feedstock. 😉

The offer to maintain it myself also still stands, but I cannot do that out of my free time. Perhaps the situation may change/improve now that QuantStack has jumped into the fray (I'm already collaborating with them). 🙃

@pitrou
Copy link
Member

pitrou commented Oct 25, 2024

Perhaps the situation may change/improve now that QuantStack has jumped into the fray (I'm already collaborating with them).

Yes, hopefully. We might get to work together :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants