Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: avoid MAPDL commands execution when gRPC connection fails. #3686

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from

Conversation

germa89
Copy link
Collaborator

@germa89 germa89 commented Jan 21, 2025

Description

Making sure we avoid executing more commands when exiting MAPDL.

Issue linked

Close #3685

Checklist

@germa89 germa89 requested a review from a team as a code owner January 21, 2025 15:04
@ansys-reviewer-bot
Copy link
Contributor

Thanks for opening a Pull Request. If you want to perform a review write a comment saying:

@ansys-reviewer-bot review

@germa89 germa89 self-assigned this Jan 21, 2025
…ing-more-mapdl-calls-when-the-gRPC-connection-failed
@github-actions github-actions bot added the bug Issue, problem or error in PyMAPDL label Jan 21, 2025
@germa89
Copy link
Collaborator Author

germa89 commented Jan 21, 2025

Profiler info:


   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     3587    0.144    0.000  853.568    0.238 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/pytest_rerunfailures.py:537(pytest_runtest_protocol)
85498/46379    0.304    0.000  852.081    0.018 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_hooks.py:498(__call__)
85498/46379    0.117    0.000  851.816    0.018 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_manager.py:111(_hookexec)
85498/46379    1.355    0.000  851.745    0.018 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py:53(_multicall)
     3587    0.049    0.000  850.568    0.237 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py:118(runtestprotocol)
    10677    0.098    0.000  850.502    0.080 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py:226(call_and_report)
    10677    0.078    0.000  848.459    0.079 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py:319(from_call)
    10677    0.024    0.000  848.347    0.079 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py:242(<lambda>)
     3503    0.020    0.000  615.039    0.176 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py:163(pytest_runtest_call)
     3503    0.012    0.000  614.962    0.176 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/python.py:1625(runtest)
     3503    0.039    0.000  614.868    0.176 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/python.py:152(pytest_pyfunc_call)
97414/94111    1.315    0.000  557.436    0.006 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/errors.py:299(wrapper)
   210471  316.201    0.002  316.201    0.002 {method 'acquire' of '_thread.lock' objects}
4832/4693    0.068    0.000  301.783    0.064 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/mapdl_core.py:1332(inner_wrapper)
     2772    0.199    0.000  291.017    0.105 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/mapdl_grpc.py:1719(input)
    38538    0.256    0.000  289.859    0.008 /__t/Python/3.11.11/x64/lib/python3.11/threading.py:295(wait)
    19115    0.383    0.000  288.791    0.015 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/grpc/_channel.py:936(_next)
    19114    0.080    0.000  288.146    0.015 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/grpc/_common.py:121(wait)
    23566    0.051    0.000  288.006    0.012 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/grpc/_common.py:111(_wait_once)
    12215    0.019    0.000  287.205    0.024 /__t/Python/3.11.11/x64/lib/python3.11/site-packages/grpc/_channel.py:542(__next__)

Insights

Optimize Hooks in pytest:

Investigate why pluggy hooks are being called so frequently. Consider reducing the number of hooks or optimizing hook implementations.

Analyze Locking Overhead:

The high time spent in _thread.lock.acquire suggests contention or excessive synchronization. Profiling thread usage in ansys.mapdl or other parts of the code could identify improvements.

Improve ansys.mapdl Functions:

Functions like wrapper and inner_wrapper should be examined for overhead. Profiling within these functions might reveal inefficient logic or excessive I/O calls.

Investigate gRPC Communication:

High cumulative time in gRPC functions suggests delays in message handling or transport. Review the communication patterns, message sizes, and server-client interactions.

Focus on High Cumulative Time:

Target the functions with the highest cumtime (pytest_runtest_protocol, wrapper, inner_wrapper) for optimization, as they are the biggest contributors to runtime.

@germa89
Copy link
Collaborator Author

germa89 commented Jan 21, 2025

Checking the profiler, it seems that a lot of time is spent on wrapping the XSEL commands:

image

which basically seems to come from save_selection:

def __exit__(self, *args):
self._parent()._log.debug("Exiting saving selection context")
selection = self.selection.pop()
mapdl = self._parent()
cmps = selection.pop("cmsel")
if cmps:
mapdl.components.select(cmps)
for select_cmd, ids in selection.items():
if ids.size > 0:
func = getattr(mapdl, select_cmd)
func(vmin=ids)

I need to improve this mess...

@germa89 germa89 marked this pull request as draft January 22, 2025 11:29
Copy link

codecov bot commented Jan 22, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.09%. Comparing base (8be236d) to head (a72ddfa).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3686      +/-   ##
==========================================
- Coverage   87.09%   87.09%   -0.01%     
==========================================
  Files         187      187              
  Lines       14697    14698       +1     
==========================================
  Hits        12801    12801              
- Misses       1896     1897       +1     

@germa89
Copy link
Collaborator Author

germa89 commented Jan 22, 2025

Reason

It seems we are executing save when exiting testing because of an exception. I still need to check the teardown time.

Error

Error code


2025-01-22T12:32:34.3559625Z tests/test_post.py::Test_contact_solve::test_set[4] �[33mRERUN�[0m�[31m                [  0%]�[0m
2025-01-22T12:33:10.6699111Z tests/test_post.py::Test_contact_solve::test_set[4] �[33mRERUN�[0m�[31m                [  0%]�[0m
2025-01-22T12:34:08.4267297Z tests/test_post.py::Test_contact_solve::test_set[4] �[31mERROR�[0m�[31m                [  0%]�[0mTraceback (most recent call last):
2025-01-22T12:34:08.4278149Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/errors.py", line 318, in wrapper
2025-01-22T12:34:08.4279089Z     out = func(*args, **kwargs)
2025-01-22T12:34:08.4279469Z           ^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4280433Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/mapdl_grpc.py", line 1065, in _send_command
2025-01-22T12:34:08.4281393Z     grpc_response = self._stub.SendCommand(request)
2025-01-22T12:34:08.4281898Z                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4282748Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/grpc/_channel.py", line 1181, in __call__
2025-01-22T12:34:08.4283668Z     return _end_unary_response_blocking(state, call, False, None)
2025-01-22T12:34:08.4284251Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4285311Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/grpc/_channel.py", line 1006, in _end_unary_response_blocking
2025-01-22T12:34:08.4286487Z     raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
2025-01-22T12:34:08.4287099Z     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4287751Z grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
2025-01-22T12:34:08.4288483Z 	status = StatusCode.UNAVAILABLE
2025-01-22T12:34:08.4289501Z 	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50052: Failed to connect to remote host: connect: Connection refused (111)"
2025-01-22T12:34:08.4291784Z 	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-01-22T12:34:06.424385523+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50052: Failed to connect to remote host: connect: Connection refused (111)"}"
2025-01-22T12:34:08.4293494Z >
2025-01-22T12:34:08.4293659Z 
2025-01-22T12:34:08.4293958Z During handling of the above exception, another exception occurred:
2025-01-22T12:34:08.4294427Z 
2025-01-22T12:34:08.4294592Z Traceback (most recent call last):
2025-01-22T12:34:08.4295187Z   File "/__t/Python/3.11.11/x64/bin/pytest", line 8, in <module>
2025-01-22T12:34:08.4295761Z     sys.exit(console_main())
2025-01-22T12:34:08.4296571Z              ^^^^^^^^^^^^^^
2025-01-22T12:34:08.4297433Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/config/__init__.py", line 201, in console_main
2025-01-22T12:34:08.4298287Z     code = main()
2025-01-22T12:34:08.4298597Z            ^^^^^^
2025-01-22T12:34:08.4299367Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/config/__init__.py", line 175, in main
2025-01-22T12:34:08.4300344Z     ret: ExitCode | int = config.hook.pytest_cmdline_main(config=config)
2025-01-22T12:34:08.4300967Z                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4301813Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
2025-01-22T12:34:08.4302823Z     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
2025-01-22T12:34:08.4303718Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4304608Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
2025-01-22T12:34:08.4305568Z     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
2025-01-22T12:34:08.4306190Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4307053Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
2025-01-22T12:34:08.4308322Z     raise exception.with_traceback(exception.__traceback__)
2025-01-22T12:34:08.4309329Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
2025-01-22T12:34:08.4310132Z     res = hook_impl.function(*args)
2025-01-22T12:34:08.4310550Z           ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4311378Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/main.py", line 330, in pytest_cmdline_main
2025-01-22T12:34:08.4312183Z     return wrap_session(config, _main)
2025-01-22T12:34:08.4312590Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4313315Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/main.py", line 318, in wrap_session
2025-01-22T12:34:08.4314051Z     config.hook.pytest_sessionfinish(
2025-01-22T12:34:08.4314804Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
2025-01-22T12:34:08.4315749Z     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
2025-01-22T12:34:08.4316406Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4317249Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
2025-01-22T12:34:08.4318195Z     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
2025-01-22T12:34:08.4318806Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4319624Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
2025-01-22T12:34:08.4320551Z     raise exception.with_traceback(exception.__traceback__)
2025-01-22T12:34:08.4321503Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
2025-01-22T12:34:08.4322398Z     teardown.throw(exception)  # type: ignore[union-attr]
2025-01-22T12:34:08.4322915Z     ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4323787Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/logging.py", line 868, in pytest_sessionfinish
2025-01-22T12:34:08.4324662Z     return (yield)
2025-01-22T12:34:08.4324972Z             ^^^^^
2025-01-22T12:34:08.4325700Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
2025-01-22T12:34:08.4326600Z     teardown.throw(exception)  # type: ignore[union-attr]
2025-01-22T12:34:08.4327115Z     ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4327986Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/terminal.py", line 893, in pytest_sessionfinish
2025-01-22T12:34:08.4329144Z     result = yield
2025-01-22T12:34:08.4329455Z              ^^^^^
2025-01-22T12:34:08.4330173Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
2025-01-22T12:34:08.4331074Z     teardown.throw(exception)  # type: ignore[union-attr]
2025-01-22T12:34:08.4331608Z     ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4332564Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/warnings.py", line 141, in pytest_sessionfinish
2025-01-22T12:34:08.4333435Z     return (yield)
2025-01-22T12:34:08.4333759Z             ^^^^^
2025-01-22T12:34:08.4334481Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
2025-01-22T12:34:08.4335271Z     res = hook_impl.function(*args)
2025-01-22T12:34:08.4335669Z           ^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4336485Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py", line 107, in pytest_sessionfinish
2025-01-22T12:34:08.4337388Z     session._setupstate.teardown_exact(None)
2025-01-22T12:34:08.4338284Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py", line 557, in teardown_exact
2025-01-22T12:34:08.4339094Z     raise exceptions[0]
2025-01-22T12:34:08.4339826Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/runner.py", line 546, in teardown_exact
2025-01-22T12:34:08.4340602Z     fin()
2025-01-22T12:34:08.4341472Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/fixtures.py", line 1032, in finish
2025-01-22T12:34:08.4342261Z     raise exceptions[0]
2025-01-22T12:34:08.4342980Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/fixtures.py", line 1021, in finish
2025-01-22T12:34:08.4343916Z     fin()
2025-01-22T12:34:08.4344704Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/_pytest/fixtures.py", line 907, in _teardown_yield_fixture
2025-01-22T12:34:08.4345593Z     next(it)
2025-01-22T12:34:08.4346107Z   File "/__w/pymapdl/pymapdl/tests/conftest.py", line 623, in mapdl
2025-01-22T12:34:08.4347016Z     mapdl.exit(save=True, force=True)
2025-01-22T12:34:08.4347893Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/errors.py", line 501, in wrapper
2025-01-22T12:34:08.4348942Z     return function(self, *args, **kwargs)
2025-01-22T12:34:08.4349379Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4350242Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/errors.py", line 501, in wrapper
2025-01-22T12:34:08.4351126Z     return function(self, *args, **kwargs)
2025-01-22T12:34:08.4351558Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4352401Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/mapdl_grpc.py", line 1152, in exit
2025-01-22T12:34:08.4354861Z     self.save()
2025-01-22T12:34:08.4355778Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/_commands/database/setup.py", line 162, in save
2025-01-22T12:34:08.4356953Z     return self.run(f"SAVE,{fname},{ext},,{slab}", **kwargs)
2025-01-22T12:34:08.4357534Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4358432Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/mapdl_core.py", line 2294, in run
2025-01-22T12:34:08.4361163Z     text = self._run(command, verbose=verbose, mute=mute)
2025-01-22T12:34:08.4361917Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4362667Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/mapdl_grpc.py", line 1014, in _run
2025-01-22T12:34:08.4364984Z     response = self._send_command(cmd, mute=mute)
2025-01-22T12:34:08.4365561Z                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-22T12:34:08.4366886Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/errors.py", line 379, in wrapper
2025-01-22T12:34:08.4368427Z     handle_generic_grpc_error(error, func, args, kwargs, reason, suggestion)
2025-01-22T12:34:08.4369568Z   File "/__t/Python/3.11.11/x64/lib/python3.11/site-packages/ansys/mapdl/core/errors.py", line 464, in handle_generic_grpc_error
2025-01-22T12:34:08.4370288Z     raise MapdlExitedError(msg)
2025-01-22T12:34:08.4370662Z ansys.mapdl.core.errors.MapdlExitedError: Error:
2025-01-22T12:34:08.4371139Z MAPDL server connection terminated unexpectedly while running:
2025-01-22T12:34:08.4371554Z   SAVE,,,,
2025-01-22T12:34:08.4371762Z called by:
2025-01-22T12:34:08.4371968Z   _send_command
2025-01-22T12:34:08.4372103Z 
2025-01-22T12:34:08.4372199Z Suggestions:
2025-01-22T12:34:08.4372626Z   MAPDL *might* have died because it executed a not-allowed command or ran out of memory.
2025-01-22T12:34:08.4373168Z   Check the MAPDL command output for more details.
2025-01-22T12:34:08.4373702Z   Open an issue on GitHub if you need assistance: https://github.com/ansys/pymapdl/issues
2025-01-22T12:34:08.4374184Z Error:
2025-01-22T12:34:08.4374777Z   failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50052: Failed to connect to remote host: connect: Connection refused (111)
2025-01-22T12:34:08.4375455Z Full error:
2025-01-22T12:34:08.4375720Z <_InactiveRpcError of RPC that terminated with:
2025-01-22T12:34:08.4376059Z 	status = StatusCode.UNAVAILABLE
2025-01-22T12:34:08.4376850Z 	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50052: Failed to connect to remote host: connect: Connection refused (111)"
2025-01-22T12:34:08.4378236Z 	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-01-22T12:34:06.424385523+00:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50052: Failed to connect to remote host: connect: Connection refused (111)"}"
2025-01-22T12:34:08.4379238Z >
2025-01-22T12:34:08.9497168Z ##[error]Process completed with exit code 1.
2025-01-22T12:34:08.9508480Z ##[debug]Finishing: Unit testing

From: https://github.com/ansys/pymapdl/actions/runs/12907800033/job/35992191911?pr=3686#step:11:197

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue, problem or error in PyMAPDL
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Teardown raises exceptions
2 participants