-
Notifications
You must be signed in to change notification settings - Fork 6.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[arrow] Linker errors with arrow[parquet] #22552
Comments
Is port |
Confirmed, this issue exists in the latest master branch with the following code: #include "arrow/api.h"
#include "parquet/metadata.h"
int main(){
auto pool = arrow::default_memory_pool();
parquet::FileMetaData* foo;
foo->num_columns();
return 0;
} find_package(Arrow CONFIG REQUIRED)
find_package(Parquet CONFIG REQUIRED)
include_directories(${ARROW_INCLUDE_DIR})
add_executable(test main.cxx)
target_link_libraries(test PRIVATE ${PARQUET_LIBRARIES} ${ARROW_LIBRARIES})
|
I'm still seeing this issue with the new 7.0 Arrow port and port-version 1. Its unusable in it's current state. I tried linking in Thrift directly using vcpkg, but there's no information how to do that anywhere online that I can find and nothing in the port provides a helpful hint. @LilyWangLL @ianmcook (sorry for the tags, but wanted to get this some visibility) |
@diehard2 @JackBoosY PR #23423 should have fixed the issue |
@diehard2 Can you double confirm that? |
Hi @JackBoosY @Thomas1664 thanks for the follow-up! Same errors as before. I can't figure anyway to make this useable. New docker setup showing the issues is below. Similar to the one above, but its using almalinux since CentOS 8 went to a stream. Any help here would be greatly appreciated. (sorry, I can't get the code block to render correctly)
|
@JackBoosY @Thomas1664 here's a docker build that shows I can successfully use a manually built version of the libraries (granted, they're dynamic). Hope this helps
|
Still facing link errors with vcpkg installed arrow. Error disappears when I use a self built arrow. |
I was still able to reproduce this issue with 9.0.0, it is caused by |
I tried #22552 (comment) (that is based on Apache Arrow 7.0.0). The following dependencies are missed from link command line:
It seems that $ grep INTERFACE_LINK_LIBRARIES vcpkg_installed/x64-linux/share/arrow/ArrowTargets.cmake
INTERFACE_LINK_LIBRARIES "OpenSSL::Crypto;OpenSSL::SSL;BZip2::BZip2;Snappy::snappy;ZLIB::ZLIB;zstd::libzstd_static;re2::re2;Threads::Threads;rt;\$<LINK_ONLY:>" I'm not sure that this is a problem in Apache Arrow's CMake configuration or patches in vcpkg. |
The following patch for Apache Arrow may fix the diff --git a/cpp/src/parquet/CMakeLists.txt b/cpp/src/parquet/CMakeLists.txt
index 542fa5bc08..88f270b6b1 100644
--- a/cpp/src/parquet/CMakeLists.txt
+++ b/cpp/src/parquet/CMakeLists.txt
@@ -221,6 +221,8 @@ if(NOT PARQUET_MINIMAL_DEPENDENCY)
set(PARQUET_STATIC_LINK_LIBS ${PARQUET_STATIC_LINK_LIBS} thrift::thrift)
endif(NOT PARQUET_MINIMAL_DEPENDENCY)
+set(PARQUET_STATIC_INTERFACE_INSTALL_LIBS ${PARQUET_STATIC_LINK_LIBS})
+
if(CXX_LINKER_SUPPORTS_VERSION_SCRIPT)
set(PARQUET_SHARED_LINK_FLAGS
"-Wl,--version-script=${CMAKE_CURRENT_SOURCE_DIR}/symbols.map")
@@ -244,7 +246,9 @@ add_arrow_lib(parquet
SHARED_PRIVATE_LINK_LIBS
${PARQUET_SHARED_PRIVATE_LINK_LIBS}
STATIC_LINK_LIBS
- ${PARQUET_STATIC_LINK_LIBS})
+ ${PARQUET_STATIC_LINK_LIBS}
+ STATIC_INSTALL_INTERFACE_LIBS
+ ${PARQUET_STATIC_INSTALL_INTERFACE_LIBS})
if(WIN32 AND NOT (ARROW_TEST_LINKAGE STREQUAL "static"))
add_library(parquet_test_support STATIC |
@kou Can you please provide the related official patch / PR link? |
@kou with your patch and 9.0.0 it looks like only thirf::thrift is still missing?
The linking issue also still exists :( I don't understand why this only happens with |
parquet_static needs Thrift because cpp/src/parquet/thrift_internal.h uses Thrift. See also: microsoft/vcpkg#22552 (comment) We can test this by the following command lines: $ cd cpp/examples/parquet/parquet_arrow $ export Arrow_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ export Parquet_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ cmake -S . -B build -DPARQUET_LINK_SHARED=OFF $ cmake --build build --verbose I also noticed that OpenTelemetry related dependencies are also missed because I usually use -DARROW_WITH_OPENTELEMETRY=ON option for Apache Arrow development. The problem is also fixed in this.
@JackBoosY I've created it: apache/arrow#13863 |
@assignUser Sorry. My patch had a typo: This may not be related to |
parquet_static needs Thrift because cpp/src/parquet/thrift_internal.h uses Thrift. See also: microsoft/vcpkg#22552 (comment) We can test this by the following command lines: $ cd cpp/examples/parquet/parquet_arrow $ export Arrow_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ export Parquet_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ cmake -S . -B build -DPARQUET_LINK_SHARED=OFF $ cmake --build build --verbose I also noticed that OpenTelemetry related dependencies are also missed because I usually use -DARROW_WITH_OPENTELEMETRY=ON option for Apache Arrow development. The problem is also fixed in this.
parquet_static needs Thrift because cpp/src/parquet/thrift_internal.h uses Thrift. See also: microsoft/vcpkg#22552 (comment) We can test this by the following command lines: $ cd cpp/examples/parquet/parquet_arrow $ export Arrow_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ export Parquet_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ cmake -S . -B build -DPARQUET_LINK_SHARED=OFF $ cmake --build build --verbose I also noticed that OpenTelemetry related dependencies are also missed because I usually use -DARROW_WITH_OPENTELEMETRY=ON option for Apache Arrow development. The problem is also fixed in this. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
…3863) parquet_static needs Thrift because cpp/src/parquet/thrift_internal.h uses Thrift. See also: microsoft/vcpkg#22552 (comment) We can test this by the following command lines: $ cd cpp/examples/parquet/parquet_arrow $ export Arrow_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ export Parquet_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ cmake -S . -B build -DPARQUET_LINK_SHARED=OFF $ cmake --build build --verbose I also noticed that OpenTelemetry related dependencies are also missed because I usually use -DARROW_WITH_OPENTELEMETRY=ON option for Apache Arrow development. The problem is also fixed in this. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
…3863) parquet_static needs Thrift because cpp/src/parquet/thrift_internal.h uses Thrift. See also: microsoft/vcpkg#22552 (comment) We can test this by the following command lines: $ cd cpp/examples/parquet/parquet_arrow $ export Arrow_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ export Parquet_DIR=${ARROW_INSTALL_PREFIX}/lib/cmake/arrow $ cmake -S . -B build -DPARQUET_LINK_SHARED=OFF $ cmake --build build --verbose I also noticed that OpenTelemetry related dependencies are also missed because I usually use -DARROW_WITH_OPENTELEMETRY=ON option for Apache Arrow development. The problem is also fixed in this. Authored-by: Sutou Kouhei <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>
Parquet libraries are not linking correctly.
Environment
To Reproduce
The following Docker image will replicate the issue. Just cd into parque_test/build and run scl enable gcc-toolset-10 bash && cmake --build. It appears to not be linking to thrift, lz4, and brotli. The gcc 10 toolset is required due to an error build boost math with the base install. I see the same behavior on a more complicated docker image that uses gcc 12. Sorry, the code blocks on here don't seem to be working
Failure logs
etc
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: