Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Non determinsistic segault seemingly introduced by #41 #42

Open
joshc-slac opened this issue Sep 11, 2024 · 3 comments
Open

Non determinsistic segault seemingly introduced by #41 #42

joshc-slac opened this issue Sep 11, 2024 · 3 comments

Comments

@joshc-slac
Copy link
Collaborator

Current Behavior

The "old" behavior of a weird interaction of caproto and pytest has resurfaced since PR #41 . In the logs I am seeing a non deterministic segfault. This may be nothing, an edge case exercised by caproto and pytest that will never get us in the real world, it may also be real and I want to track it here..
@joshc-slac
Copy link
Collaborator Author

joshc-slac commented Sep 11, 2024 •
edited by tangkong
Loading

beams/tests/test_tree_generator.py ..Fatal Python error: Segmentation fault

Thread 0x00007fcadb1ff6c0 (most recent call first):
  File /usr/lib/python3.11/threading.py, line 320 in wait
  File /usr/lib/python3.11/multiprocessing/queues.py, line 231 in _feed
  File /usr/lib/python3.11/threading.py, line 975 in run
  File /usr/lib/python3.11/threading.py, line 1038 in _bootstrap_inner
  File /usr/lib/python3.11/threading.py, line 995 in _bootstrap

Current thread 0x00007fcae098c2c0 (most recent call first):
  Garbage-collecting
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 1896 in clear_subscription
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 670 in wrapper
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 546 in wrapper
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 353 in _clear_auto_monitor_subscription
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 46 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 1152 in disconnect
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 53 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 1165 in __del__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 1021 in connect_channel
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 576 in wrapper
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 445 in connect
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 46 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 430 in wait_for_connection
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 46 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 156 in get_pv
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/__init__.py, line 150 in caget
  File /home/josh/Code/playpen/BEAMS/beams/tree_config.py, line 148 in cond_func
  File /home/josh/Code/playpen/BEAMS/beams/behavior_tree/ActionNode.py, line 40 in work_wrapper
  File /usr/lib/python3.11/multiprocessing/process.py, line 108 in run
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/multiproc.py, line 39 in run
  File /usr/lib/python3.11/multiprocessing/process.py, line 314 in _bootstrap
  File /usr/lib/python3.11/multiprocessing/popen_fork.py, line 71 in _launch
  File /usr/lib/python3.11/multiprocessing/popen_fork.py, line 19 in __init__
  File /usr/lib/python3.11/multiprocessing/context.py, line 281 in _Popen
  File /usr/lib/python3.11/multiprocessing/context.py, line 224 in _Popen
  File /usr/lib/python3.11/multiprocessing/process.py, line 121 in start
  File /home/josh/Code/playpen/BEAMS/beams/sequencer/helpers/Worker.py, line 42 in start_work
  File /home/josh/Code/playpen/BEAMS/beams/behavior_tree/ActionNode.py, line 99 in setup
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/py_trees/trees.py, line 110 in visited_setup
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/py_trees/trees.py, line 119 in setup
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/py_trees/trees.py, line 378 in setup
  File /home/josh/Code/playpen/BEAMS/beams/tests/test_tree_generator.py, line 53 in test_father_tree_execution
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/python.py, line 194 in pytest_pyfunc_call
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/python.py, line 1792 in runtest
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 169 in pytest_runtest_call
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 262 in <lambda>
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 341 in from_call
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 261 in call_runtest_hook
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 222 in call_and_report
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 133 in runtestprotocol
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 114 in pytest_runtest_protocol
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 350 in pytest_runtestloop
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 325 in _main
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 271 in wrap_session
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 318 in pytest_cmdline_main
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/config/__init__.py, line 169 in main
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/config/__init__.py, line 192 in console_main
  File /home/josh/Code/playpen/BEAMS/venv/bin/pytest, line 8 in <module>

Extension modules: yaml._yaml, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, google._upb._message, grpc._cython.cygrpc, apischema.serialization.methods, apischema.deserialization.methods (total: 18)

@joshc-slac joshc-slac mentioned this issue Sep 11, 2024
Merged
8 tasks
@joshc-slac
Copy link
Collaborator Author

fascinatingly removing this line:

BEAMS/beams/tests/conftest.py

Line 16 in b74af9a

setup_logging(logging.DEBUG)
makes this error no longer occur. But then I don't get logging in my tests

@tangkong
Copy link
Contributor

This makes sense, if the logging level is INFO (a higher severity than DEBUG), the logging record will get dropped before it gets processed. This does seem to implicate logging as the process that's getting left behind.

We probably need to figure out a good way of killing the logging thread after each test. Or at least figure out what pytest is doing with the logging messages

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tangkong @joshc-slac