Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[python] Trying to open SOMAs under error conditions occasionally segfaults (pytest only) #725

Closed
thetorpedodog opened this issue Jan 13, 2023 · 6 comments
Closed

[python] Trying to open SOMAs under error conditions occasionally segfaults (pytest only) #725

thetorpedodog opened this issue Jan 13, 2023 · 6 comments
Assignees
@nguyenv
Labels
blocked

Comments

@thetorpedodog
Copy link
Contributor

I've found that I can somewhat reliably get a segfault in tiledbsoma from a fresh checkout:

#!/bin/sh

set -ex

SCRATCH="$(mktemp -d --tmpdir= maybe-segfault-XXXXXXXXXXXXX)"

cd "$SCRATCH"
trap 'echo "TEMPORARY DIRECTORY IS $SCRATCH"' EXIT

git clone --branch new-experiment --single-branch --filter=tree:0 https://github.com/single-cell-data/TileDB-SOMA.git


cd TileDB-SOMA

python -m venv venv
. ./venv/bin/activate
pip install --upgrade pip wheel pytest typeguard

make install

cd apis/python
pytest -vs -k test_error_corners ./tests

This branch includes the changes from 5151b6f.

[build output elided]
+ pytest -vs -k test_error_corners ./tests
========================================================================================== test session starts ==========================================================================================
platform linux -- Python 3.9.2, pytest-7.2.0, pluggy-1.0.0 -- /tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/apis/python
plugins: typeguard-2.13.3
collected 936 items / 935 deselected / 1 selected                                                                                                                                                       

tests/test_experiment_query.py::test_error_corners[2833-107] Fatal Python error: Segmentation fault

Current thread 0x00007f008979b740 (most recent call first):
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/tiledb/group.py", line 396 in __repr__
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_io/saferepr.py", line 76 in repr_instance
  File "/usr/lib/python3.9/reprlib.py", line 62 in repr1
  File "/usr/lib/python3.9/reprlib.py", line 52 in repr
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_io/saferepr.py", line 64 in repr
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_io/saferepr.py", line 115 in saferepr
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 731 in repr_args
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 821 in repr_traceback_entry
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 871 in repr_traceback
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 944 in repr_excinfo
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/_code/code.py", line 669 in getrepr
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/nodes.py", line 484 in _repr_failure_py
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/python.py", line 1823 in repr_failure
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/reports.py", line 349 in from_item_and_call
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/runner.py", line 366 in pytest_runtest_makereport
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/runner.py", line 131 in runtestprotocol
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/runner.py", line 112 in pytest_runtest_protocol
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/main.py", line 324 in _main
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 167 in main
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/lib/python3.9/site-packages/_pytest/config/__init__.py", line 190 in console_main
  File "/tmp/maybe-segfault-IWIwkutY1afPJ/TileDB-SOMA/venv/bin/pytest", line 8 in <module>
Segmentation fault
+ echo TEMPORARY DIRECTORY IS /tmp/maybe-segfault-IWIwkutY1afPJ
TEMPORARY DIRECTORY IS /tmp/maybe-segfault-IWIwkutY1afPJ
@Shelnutt2
Copy link
Contributor

@thetorpedodog do you have a core?

@gspowley
Copy link
Member

The segfault comes from TileDB-Py, when this line is executed:

        soma.Experiment(uri="foobar").axis_query("foobar")
Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffc1a2d2a6 in tiledb::Group::dump(bool) const () from /home/gspowley/work/debug/soma-segfault/TileDB-SOMA/venv/lib/python3.10/site-packages/tiledb/cc.cpython-310-x86_64-linux-gnu.so
#0  0x00007fffc1a2d2a6 in tiledb::Group::dump(bool) const () from /home/gspowley/work/debug/soma-segfault/TileDB-SOMA/venv/lib/python3.10/site-packages/tiledb/cc.cpython-310-x86_64-linux-gnu.so
#1  0x00007fffc1a2f337 in pybind11::cpp_function::initialize<pybind11::cpp_function::initialize<std::string, tiledb::Group, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(std::string (tiledb::Group::*)(bool) const, pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(tiledb::Group const*, bool)#1}, std::string, tiledb::Group const*, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(pybind11::cpp_function::initialize<std::string, tiledb::Group, bool, pybind11::name, pybind11::is_method, pybind11::sibling>(std::string (tiledb::Group::*)(bool) const, pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(tiledb::Group const*, bool)#1}&&, std::string (*)(tiledb::Group const*, bool), pybind11::name const&, pybind11::is_method const&, pybind11::sibling const&)::{lambda(pybind11::detail::function_call&)#3}::_FUN(pybind11::detail::function_call&) () from /home/gspowley/work/debug/soma-segfault/TileDB-SOMA/venv/lib/python3.10/site-packages/tiledb/cc.cpython-310-x86_64-linux-gnu.so

@gspowley
Copy link
Member

Running the line above outside of pytest we get a proper stack trace from python.

Traceback (most recent call last):
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/apis/python/src/tiledbsoma/collection.py", line 231, in _load_tdb_group
    with self._tiledb_open("r") as G:
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/apis/python/src/tiledbsoma/collection.py", line 361, in _tiledb_open
    return tiledb.Group(self._uri, mode=mode, ctx=self._ctx)
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/venv/lib/python3.10/site-packages/tiledb/group.py", line 285, in __init__
    super().__init__(self._ctx, uri, query_type)
tiledb.cc.TileDBError: [TileDB::GroupDirectory] Error: Cannot open group; Group does not exist.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/test.py", line 14, in <module>
    soma.Experiment(uri="foobar").axis_query("foobar")
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/apis/python/src/tiledbsoma/experiment.py", line 89, in axis_query
    return somacore.ExperimentAxisQuery(
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/venv/lib/python3.10/site-packages/somacore/query/query.py", line 68, in __init__
    if measurement_name not in experiment.ms:
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/apis/python/src/tiledbsoma/experiment.py", line 76, in ms
    return cast(CollectionBase[Measurement], self["ms"])
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/apis/python/src/tiledbsoma/collection.py", line 132, in __getitem__
    self._load_tdb_group_cache()
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/apis/python/src/tiledbsoma/collection.py", line 249, in _load_tdb_group_cache
    tdb_group = self._load_tdb_group()
  File "/home/gspowley/work/debug/soma-segfault/TileDB-SOMA/apis/python/src/tiledbsoma/collection.py", line 236, in _load_tdb_group
    raise DoesNotExistError("Collection not created") from e
tiledbsoma.exception.DoesNotExistError: Collection not created

This looks like a similar issue we have seen, potentially related to pytest-dev/pytest#4659.

@johnkerl
Copy link
Member

Looks like this is addressed by @nguyenv 's TileDB-Inc/TileDB-Py#1545 which isn't in the currently released version of tiledb-py

@gspowley gspowley assigned nguyenv and unassigned gspowley Jan 17, 2023
@johnkerl johnkerl changed the title Trying to open SOMAs under error conditions occasionally segfaults Trying to open SOMAs under error conditions occasionally segfaults (pytest only) Jan 19, 2023
@johnkerl
Copy link
Member

I labeled this as pending-other as I believe we should revisit once TileDB core 2.14.0 / TileDB-Py 0.20.0 are released as that will have the above TileDB-Inc/TileDB-Py#1545 in it

@johnkerl johnkerl changed the title Trying to open SOMAs under error conditions occasionally segfaults (pytest only) [python] Trying to open SOMAs under error conditions occasionally segfaults (pytest only) Jan 19, 2023
@johnkerl
Copy link
Member

#752 will address this as TileDB-Py 0.20 has TileDB-Inc/TileDB-Py#1545 in it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nguyenv nguyenv

Labels
blocked
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@Shelnutt2 @johnkerl @nguyenv @thetorpedodog @gspowley