Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Packaging][Release][Python] Binary verification of wheels on conda is segfaulting on ORC test_example_using_json #43232

Closed
raulcd opened this issue Jul 12, 2024 · 14 comments

Comments

@raulcd
Copy link
Member

raulcd commented Jul 12, 2024

Describe the bug, including details regarding any error messages, version, and platform.

The verify-rc-binaries-wheels-linux-conda-latest-amd64 segfaults on test_example_using_json when checking the file:

 Fatal Python error: Aborted

Current thread 0x00007f48f1c1a740 (most recent call first):
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pyarrow/orc.py", line 187 in read
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pyarrow/tests/test_orc.py", line 96 in check_example_file
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pyarrow/tests/test_orc.py", line 140 in test_example_using_json
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/python.py", line 162 in pytest_pyfunc_call
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/python.py", line 1632 in runtest
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 341 in from_call
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 364 in pytest_runtestloop
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 339 in _main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 178 in main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/_pytest/config/__init__.py", line 206 in console_main
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/site-packages/pytest/__main__.py", line 7 in <module>
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/runpy.py", line 87 in _run_code
  File "/tmp/arrow-17.0.0.cgU0S/mambaforge/envs/conda-wheel-3.8-manylinux_2_17_x86_64.manylinux2014_x86_64/lib/python3.8/runpy.py", line 194 in _run_module_as_main
/arrow/ci/scripts/python_wheel_unix_test.sh: line 97:  3967 Aborted                 (core dumped) python -m pytest -r s --pyargs pyarrow
tests/test_orc.py .Failed to verify release candidate. See /tmp/arrow-17.0.0.cgU0S for details.

This can be reproduced locally with archery docker run -e VERIFY_VERSION="17.0.0" -e VERIFY_RC="2" -e TEST_DEFAULT=0 -e TEST_WHEELS=1 conda-verify-rc. This is the only test failing as I can have a successful verification with the following local change:

diff --git a/ci/scripts/python_wheel_unix_test.sh b/ci/scripts/python_wheel_unix_test.sh
index a25e5c5..8445dce 100755
--- a/ci/scripts/python_wheel_unix_test.sh
+++ b/ci/scripts/python_wheel_unix_test.sh
@@ -93,5 +93,5 @@ if [ "${CHECK_UNITTESTS}" == "ON" ]; then
 
   # Execute unittest, test dependencies must be installed
   python -c 'import pyarrow; pyarrow.create_library_symlinks()'
-  python -m pytest -r s --pyargs pyarrow
+  python -m pytest -r s --pyargs pyarrow -k "not test_example_using_json"
 fi

Component(s)

Packaging, Python, Release

@raulcd
Copy link
Member Author

raulcd commented Jul 12, 2024

@kou @pitrou @wgtmac any idea what might cause this? This test is passing on non-conda verification tasks, see the rest of binary verifications here: #43220

@raulcd
Copy link
Member Author

raulcd commented Jul 12, 2024

This is a single test on a single job failing for the release. I am also not sure if we want to block the release for this one, @jorisvandenbossche @AlenkaF FYI

@kou
Copy link
Member

kou commented Jul 12, 2024

Can we get a backtrace by gdb?

@wgtmac
Copy link
Member

wgtmac commented Jul 12, 2024

What's the version of ORC? I'm not sure if it is related to apache/orc#1882

@h-vetinari Does it mean we have to install tzdata in the conda env after this change?

@raulcd
Copy link
Member Author

raulcd commented Jul 12, 2024

What's the version of ORC? I'm not sure if it is related to https://github.com/apache/orc/pull/1882/files

The wheel was built with:

-- ARROW_ORC_BUILD_VERSION: 2.0.1
-- ARROW_ORC_BUILD_SHA256_CHECKSUM: 1ffac0228aa83f04a1b1cf2788a3af5953e82587ae3a77c41900e99f2557132d

@wgtmac
Copy link
Member

wgtmac commented Jul 12, 2024

Yes, the commit in doubt is included in the 2.0.1 release.

@kou
Copy link
Member

kou commented Jul 12, 2024

Could you try this?

diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
index fcaaa423a4..bd136054e4 100755
--- a/dev/release/verify-release-candidate.sh
+++ b/dev/release/verify-release-candidate.sh
@@ -1153,7 +1153,7 @@ test_linux_wheels() {
     local pyver=${python/m}
     for platform in ${platform_tags}; do
       show_header "Testing Python ${pyver} wheel for platform ${platform}"
-      CONDA_ENV=wheel-${pyver}-${platform} PYTHON_VERSION=${pyver} maybe_setup_conda
+      CONDA_ENV=wheel-${pyver}-${platform} PYTHON_VERSION=${pyver} maybe_setup_conda tzdata
       if ! VENV_ENV=wheel-${pyver}-${platform} PYTHON_VERSION=${pyver} maybe_setup_virtualenv; then
         continue
       fi

@raulcd
Copy link
Member Author

raulcd commented Jul 12, 2024

Thanks @kou ! That fixes the issue

@pitrou
Copy link
Member

pitrou commented Jul 12, 2024

So ORC C++ still segfaults when it cannot find a timezone database? I thought that had been fixed @wgtmac

@wgtmac
Copy link
Member

wgtmac commented Jul 12, 2024

So ORC C++ still segfaults when it cannot find a timezone database? I thought that had been fixed @wgtmac

It is fixed in apache/orc#1893 by @ffacs and released with v2.0.1. This segfault was caused by reading TestOrcFile.testDate1900.orc whose schema is struct<time:timestamp,date:date>.

kou pushed a commit that referenced this issue Jul 12, 2024
…ement to avoid ORC failure (#43233)

### Rationale for this change

Binary verifications for wheels on conda are failing on ORC test due to missing tzdata

### What changes are included in this PR?

Adding tzdata as conda requirement when setting up the environment on the verification script

### Are these changes tested?

Those changes have been tested locally

### Are there any user-facing changes?
No
* GitHub Issue: #43232

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Sutou Kouhei <[email protected]>
@kou kou added this to the 18.0.0 milestone Jul 12, 2024
@kou
Copy link
Member

kou commented Jul 12, 2024

Issue resolved by pull request 43233
#43233

@kou kou closed this as completed Jul 12, 2024
@kou
Copy link
Member

kou commented Jul 12, 2024

So ORC C++ still segfaults when it cannot find a timezone database? I thought that had been fixed @wgtmac

It is fixed in apache/orc#1893 by @ffacs and released with v2.0.1. This segfault was caused by reading TestOrcFile.testDate1900.orc whose schema is struct<time:timestamp,date:date>.

I think that ORC C++ should not segfault for the case.
It seems that ORC C++ throws an exception when it can't find a timezone database. Do we miss a ORC_CATCH_NOT_OK() in our adapter to catch the exception?
https://github.com/apache/arrow/blob/main/cpp/src/arrow/adapters/orc/adapter.cc
(A backtrace for the case may help us.)

@pitrou
Copy link
Member

pitrou commented Jul 12, 2024

Yes, regardless of who's responsible ;-), we should definitely not segfault when trying to read a valid ORC file (or even an invalid one, actually).

@wgtmac
Copy link
Member

wgtmac commented Jul 13, 2024

Yes, we should not segfault in this case. I think we are missing a ORC_CATCH_NOT_OK at this line: https://github.com/apache/arrow/blob/main/cpp/src/arrow/adapters/orc/adapter.cc#L151

Let me fix this and check other lines as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants