[Java] Test failure in Dataset regarding TestAllTypes

[vibhatha]

Describe the bug, including details regarding any error messages, version, and platform.

There is a test failure regarding parquet based test cases in Dataset module. This issue was identified here.

Error:  Failures: 
Error:    TestAllTypes.testAllTypesParquet:261->TestDataset.assertParquetFileEquals:124 expected:<...length=2, nullCount=[2], ArrowFieldNode [length=0, nullCount=0], ArrowFieldNode [length=2, nullCount=2], ArrowFieldNode [length=0, nullCount=0], ArrowFieldNode [length=2, nullCount=2], ArrowFieldNode [length=0, nullCount=0], ArrowFieldNode [length=2, nullCount=2], ArrowFieldNode [length=2, nullCount=0]], #buffers=71, buffersLayout=[ArrowBuffer [offset=0, size=1], ArrowBuffer [offset=8, size=1], ArrowBuffer [offset=16, size=1], ArrowBuffer [offset=24, size=2], ArrowBuffer [offset=32, size=1], ArrowBuffer [offset=40, size=4], ArrowBuffer [offset=48, size=1], ArrowBuffer [offset=56, size=8], ArrowBuffer [offset=64, size=1], ArrowBuffer [offset=72, size=16], ArrowBuffer [offset=88, size=1], ArrowBuffer [offset=96, size=2], ArrowBuffer [offset=104, size=1], ArrowBuffer [offset=112, size=4], ArrowBuffer [offset=120, size=1], ArrowBuffer [offset=128, size=8], ArrowBuffer [offset=136, size=1], ArrowBuffer [offset=144, size=16], ArrowBuffer [offset=160, size=1], ArrowBuffer [offset=168, size=8], ArrowBuffer [offset=176, size=1], ArrowBuffer [offset=184, size=16], ArrowBuffer [offset=200, size=1], ArrowBuffer [offset=208, size=12], ArrowBuffer [offset=224, size=1], ArrowBuffer [offset=232, size=1], ArrowBuffer [offset=240, size=12], ArrowBuffer [offset=256, size=1], ArrowBuffer [offset=264, size=1], ArrowBuffer [offset=272, size=12], ArrowBuffer [offset=288, size=1], ArrowBuffer [offset=296, size=1], ArrowBuffer [offset=304, size=12], ArrowBuffer [offset=320, size=1], ArrowBuffer [offset=328, size=1], ArrowBuffer [offset=336, size=2], ArrowBuffer [offset=344, size=1], ArrowBuffer [offset=352, size=8], ArrowBuffer [offset=360, size=1], ArrowBuffer [offset=368, size=8], ArrowBuffer [offset=376, size=1], ArrowBuffer [offset=384, size=16], ArrowBuffer [offset=400, size=1], ArrowBuffer [offset=408, size=16], ArrowBuffer [offset=424, size=1], ArrowBuffer [offset=432, size=16], ArrowBuffer [offset=448, size=1], ArrowBuffer [offset=456, size=16], ArrowBuffer [offset=472, size=1], ArrowBuffer [offset=480, size=16], ArrowBuffer [offset=496, size=1], ArrowBuffer [offset=504, size=16], ArrowBuffer [offset=520, size=1], ArrowBuffer [offset=528, size=32], ArrowBuffer [offset=560, size=1], ArrowBuffer [offset=568, size=32], ArrowBuffer [offset=600, size=1], ArrowBuffer [offset=608, size=12], ArrowBuffer [offset=624, size=0], ArrowBuffer [offset=624, size=0], ArrowBuffer [offset=624, size=1], ArrowBuffer [offset=632, size=12], ArrowBuffer [offset=648, size=0], ArrowBuffer [offset=648, size=0], ArrowBuffer [offset=648, size=1], ArrowBuffer [offset=656, size=12], ArrowBuffer [offset=672, size=0], ArrowBuffer [offset=672, size=0], ArrowBuffer [offset=672, size=1], ArrowBuffer [offset=680, size=1], ArrowBuffer [offset=688], size=8]], closed=f...> but was:<...length=2, nullCount=[1], ArrowFieldNode [length=2, nullCount=2], ArrowFieldNode [length=0, nullCount=0], ArrowFieldNode [length=2, nullCount=2], ArrowFieldNode [length=0, nullCount=0], ArrowFieldNode [length=2, nullCount=2], ArrowFieldNode [length=0, nullCount=0], ArrowFieldNode [length=2, nullCount=2], ArrowFieldNode [length=2, nullCount=0]], #buffers=73, buffersLayout=[ArrowBuffer [offset=0, size=1], ArrowBuffer [offset=8, size=1], ArrowBuffer [offset=16, size=1], ArrowBuffer [offset=24, size=2], ArrowBuffer [offset=32, size=1], ArrowBuffer [offset=40, size=4], ArrowBuffer [offset=48, size=1], ArrowBuffer [offset=56, size=8], ArrowBuffer [offset=64, size=1], ArrowBuffer [offset=72, size=16], ArrowBuffer [offset=88, size=1], ArrowBuffer [offset=96, size=2], ArrowBuffer [offset=104, size=1], ArrowBuffer [offset=112, size=4], ArrowBuffer [offset=120, size=1], ArrowBuffer [offset=128, size=8], ArrowBuffer [offset=136, size=1], ArrowBuffer [offset=144, size=16], ArrowBuffer [offset=160, size=1], ArrowBuffer [offset=168, size=4], ArrowBuffer [offset=176, size=1], ArrowBuffer [offset=184, size=8], ArrowBuffer [offset=192, size=1], ArrowBuffer [offset=200, size=16], ArrowBuffer [offset=216, size=1], ArrowBuffer [offset=224, size=12], ArrowBuffer [offset=240, size=1], ArrowBuffer [offset=248, size=1], ArrowBuffer [offset=256, size=12], ArrowBuffer [offset=272, size=1], ArrowBuffer [offset=280, size=1], ArrowBuffer [offset=288, size=12], ArrowBuffer [offset=304, size=1], ArrowBuffer [offset=312, size=1], ArrowBuffer [offset=320, size=12], ArrowBuffer [offset=336, size=1], ArrowBuffer [offset=344, size=1], ArrowBuffer [offset=352, size=2], ArrowBuffer [offset=360, size=1], ArrowBuffer [offset=368, size=8], ArrowBuffer [offset=376, size=1], ArrowBuffer [offset=384, size=8], ArrowBuffer [offset=392, size=1], ArrowBuffer [offset=400, size=16], ArrowBuffer [offset=416, size=1], ArrowBuffer [offset=424, size=16], ArrowBuffer [offset=440, size=1], ArrowBuffer [offset=448, size=16], ArrowBuffer [offset=464, size=1], ArrowBuffer [offset=472, size=16], ArrowBuffer [offset=488, size=1], ArrowBuffer [offset=496, size=16], ArrowBuffer [offset=512, size=1], ArrowBuffer [offset=520, size=16], ArrowBuffer [offset=536, size=1], ArrowBuffer [offset=544, size=32], ArrowBuffer [offset=576, size=1], ArrowBuffer [offset=584, size=32], ArrowBuffer [offset=616, size=1], ArrowBuffer [offset=624, size=12], ArrowBuffer [offset=640, size=0], ArrowBuffer [offset=640, size=0], ArrowBuffer [offset=640, size=1], ArrowBuffer [offset=648, size=12], ArrowBuffer [offset=664, size=0], ArrowBuffer [offset=664, size=0], ArrowBuffer [offset=664, size=1], ArrowBuffer [offset=672, size=12], ArrowBuffer [offset=688, size=0], ArrowBuffer [offset=688, size=0], ArrowBuffer [offset=688, size=1], ArrowBuffer [offset=696, size=1], ArrowBuffer [offset=704], size=8]], closed=f...>

Component(s)

Java

[kou]

Can you reproduce this on local?

[vibhatha]

I am working on it at the moment 🙂

[vibhatha]

@kou I cannot build the JNI libraries in both Ubuntu and Mac due to a different reason. I am working on that...

[vibhatha]

-- Building using CMake version: 3.28.3
-- The C compiler identification is GNU 13.2.0
-- The CXX compiler identification is GNU 13.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /bin/c++ - skipped

This is part of the trace.

[kou]

Could you also show dpkg -l | grep libstd?

[vibhatha]

$ dpkg -l | grep libstd
ii  libstdc++-11-dev:amd64                     11.4.0-1ubuntu1~22.04                                 amd64        GNU Standard C++ Library v3 (development files)
ii  libstdc++6:amd64                           12.3.0-1ubuntu1~22.04                                 amd64        GNU Standard C++ Library v3
ii  libstdc++6:i386                            12.3.0-1ubuntu1~22.04                                 i386         GNU Standard C++ Library v3

[kou]

Could you remove llvm installed by conda?

[kou]

Or could you also install clang by conda?

[kou]

It seems that you mixed LLVM from conda and LLVM by apt:

CMake Warning at cmake_modules/FindLLVMAlt.cmake:58 (find_package):
  Could not find a configuration file for package "LLVM" that is compatible
  with requested version "18.1".

  The following configuration files were considered but not accepted:

    /home/asus/miniforge3/envs/pyarrow-dev/lib/cmake/llvm/LLVMConfig.cmake, version: 17.0.6
    /usr/lib/llvm-15/cmake/LLVMConfig.cmake, version: 15.0.7
    /usr/lib/llvm-13/cmake/LLVMConfig.cmake, version: 13.0.1
    /usr/lib/llvm-15/lib/cmake/llvm/LLVMConfig.cmake, version: 15.0.7
    /usr/lib/llvm-13/lib/cmake/llvm/LLVMConfig.cmake, version: 13.0.1
    /usr/lib/llvm-15/share/llvm/cmake/LLVMConfig.cmake, version: 15.0.7
    /usr/lib/llvm-13/share/llvm/cmake/LLVMConfig.cmake, version: 13.0.1
    /lib/llvm-15/cmake/LLVMConfig.cmake, version: 15.0.7
    /lib/llvm-13/cmake/LLVMConfig.cmake, version: 13.0.1

Call Stack (most recent call first):
  src/gandiva/CMakeLists.txt:30 (find_package)


-- Using LLVMConfig.cmake in: /home/asus/miniforge3/envs/pyarrow-dev/lib/cmake/llvm
-- Found llvm-link /usr/lib/llvm-15/bin/llvm-link
-- Found clang /usr/lib/llvm-15/bin/clang-15

[vibhatha]

@kou I removed llvm from apt and also installed clang.
But I get a different error now.

CMake Error at cmake_modules/ThirdpartyToolchain.cmake:4988 (set_property):
  The link interface of target "AWS::aws-c-cal" contains:

    OpenSSL::Crypto

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  cmake_modules/ThirdpartyToolchain.cmake:176 (build_awssdk)
  cmake_modules/ThirdpartyToolchain.cmake:308 (build_dependency)
  cmake_modules/ThirdpartyToolchain.cmake:5030 (resolve_dependency)
  CMakeLists.txt:543 (include)


CMake Error at cmake_modules/BuildUtils.cmake:301 (target_link_libraries):
  Target "gandiva_objlib" links to:

    OpenSSL::Crypto

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  src/gandiva/CMakeLists.txt:140 (add_arrow_lib)


CMake Error at cmake_modules/BuildUtils.cmake:475 (target_link_libraries):
  Target "gandiva_static" links to:

    OpenSSL::Crypto

  but the target was not found.  Possible reasons include:

    * There is a typo in the target name.
    * A find_package call is missing for an IMPORTED target.
    * An ALIAS target is missing.

Call Stack (most recent call first):
  src/gandiva/CMakeLists.txt:140 (add_arrow_lib)

should I explicitly install openssl libs?

[vibhatha]

clang was installed by conda.

[kou]

Yes. Could you install OpenSSL by ... conda?

[vibhatha]

@kou I was able to fix my dev environment and thanks for the tips.

Also I can verify that the issue recorded here can be reproduced in Ubuntu 22.04.4 LTS

[vibhatha]

@kou looking into the issue, in high level what's happening is that there is a comparison between in memory generated Arrow Java data which is later written to the disk vs a predefined test file alltypes-java.parquet which has been introduced here: https://github.com/apache/arrow-testing/blob/ad82a736c170e97b7c8c035ebd8a801c17eec170/data/parquet/README.md. In the description it mentions Parquet file written by using the Java DatasetWriter class in Arrow 14.0.... I am just thinking out loud, some modification happened in Arrow 15.0 could be causing this?

[kou]

Could you try git bisect to identify which commit is related to this?

[vibhatha]

@kou the main change in the source is associated with #39681

Here we introduce the Float16, if we do it here, don't we have to update the alltypes-java.parquet in accordance with that?

[vibhatha]

If we check the testing in the main: https://github.com/apache/arrow-testing/tree/ad82a736c170e97b7c8c035ebd8a801c17eec170

And if we check the latest commit, we don't find what is introduced by this PR: apache/arrow-testing#99

[vibhatha]

Seems like we have to update the submodule?

[lidavidm]

Issue resolved by pull request 40662
#40662