A Communication Issue Perhaps?

[Ahmed-Zeid]

Hey!

I have been using PyMAPDL for the past few weeks with no issue. All of a sudden, and for the past two day, I can't run a single simulation and the following error is displayed:

PyMAPDL is taking longer than expected to connect to an MAPDL session.
Checking if there are any available licenses...

Error in instance GRPC_127.0.0.1:50052

I can confirm that nothing has changed since it was last working. Any idea?

Python 3.11.7
ansys-mapdl-core 0.68.1
Ansys 2022 R2

[germa89]

Hi @Ahmed-Zeid

Are you sure you have available licenses? You can try to start MAPDL manually, using the Mechanical launcher and see if you get any licenses issues.

[jorisparet]

Hello,

I'm having a similar issue with the following configuration:

• Python 3.10.9
• ansys-mapdl-core 0.68.3
• Ansys 2021 R2

95% of the time, when I try to create a new MAPDL instance with launch_mapdl, I have the same warning message. Sometimes it just ends up running my simulation after some time, but most of the time it crashes my code and gives me the following error:

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
File ~/db-vv40d-sdyn-femu/src/auto_femu/database/database.py:113, in Database.update_materials_by_component(self, new_materials, log, clean)
    111 # Launch a new MAPDL instance
    112 jobname = secrets.token_hex(8)
--> 113 mapdl = launch_mapdl(run_location=sim_cwd, nproc=1, jobname=jobname, license_type=self.license_type, port=MAPDL_GRPC_PORT, start_timeout=APDL_START_TIMEOUT)
    114 _ = mapdl.finish()
    116 # Resume from the original database

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/ansys/mapdl/core/launcher.py:1829, in launch_mapdl(exec_file, run_location, jobname, nproc, ram, mode, override, loglevel, additional_switches, start_timeout, port, cleanup_on_exit, start_instance, ip, clear_on_connect, log_apdl, remove_temp_files, remove_temp_dir_on_exit, verbose_mapdl, license_server_check, license_type, print_com, add_env_vars, replace_env_vars, version, **kwargs)
   1826         LOG.debug("Checking license server.")
   1827         lic_check.check()
-> 1829     raise exception
   1831 # Stopping license checker
   1832 if license_server_check:

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/ansys/mapdl/core/launcher.py:1791, in launch_mapdl(exec_file, run_location, jobname, nproc, ram, mode, override, loglevel, additional_switches, start_timeout, port, cleanup_on_exit, start_instance, ip, clear_on_connect, log_apdl, remove_temp_files, remove_temp_dir_on_exit, verbose_mapdl, license_server_check, license_type, print_com, add_env_vars, replace_env_vars, version, **kwargs)
   1773 if _debug_no_launch:
   1774     # Early exit, just for testing
   1775     return pack_parameters(
   1776         port,
   1777         ip,
   (...)
   1788         version,
   1789     )
-> 1791 port, actual_run_location, process = launch_grpc(
   1792     port=port,
   1793     add_env_vars=add_env_vars,
   1794     replace_env_vars=replace_env_vars,
   1795     **start_parm,
   1796 )
   1798 if just_launch:
   1799     out = [ip, port]

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/ansys/mapdl/core/launcher.py:515, in launch_grpc(exec_file, jobname, nproc, ram, run_location, port, additional_switches, override, timeout, verbose, add_env_vars, replace_env_vars, **kwargs)
    512         LOG.debug(f"Port in use.  Incrementing port number. port={port}")
    514 else:
--> 515     if port_in_use(port):
    516         proc = get_process_at_port(port)
    517         if proc:

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/ansys/mapdl/core/launcher.py:220, in port_in_use(port, host)
    210 def port_in_use(port: Union[int, str], host: str = LOCALHOST) -> bool:
    211     """Returns True when a port is in use at the given host.
    212 
    213     Must actually "bind" the address.  Just checking if we can create
   (...)
    218       access permissions.
    219     """
--> 220     return port_in_use_using_socket(port, host) or port_in_use_using_psutil(port)

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/ansys/mapdl/core/launcher.py:273, in port_in_use_using_psutil(port)
    267 def port_in_use_using_psutil(port: Union[int, str]) -> bool:
    268     """Returns True when a port is in use at the given host using psutil.
    269 
    270     This function iterate over all the process, and their connections until
    271     it finds one using the given port.
    272     """
--> 273     if get_process_at_port(port):
    274         return True
    275     else:

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/ansys/mapdl/core/launcher.py:254, in get_process_at_port(port)
    252 for proc in psutil.process_iter():
    253     try:
--> 254         connections = proc.connections(
    255             kind="inet"
    256         )  # just to check if we can access the
    257     except psutil.AccessDenied:
    258         continue

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/psutil/_common.py:675, in deprecated_method.<locals>.outer.<locals>.inner(self, *args, **kwargs)
    672 @functools.wraps(fun)
    673 def inner(self, *args, **kwargs):
    674     warnings.warn(msg, category=DeprecationWarning, stacklevel=2)
--> 675     return getattr(self, replacement)(*args, **kwargs)

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/psutil/__init__.py:1223, in Process.net_connections(self, kind)
   1201 def net_connections(self, kind='inet'):
   1202     """Return socket connections opened by process as a list of
   1203     (fd, family, type, laddr, raddr, status) namedtuples.
   1204     The *kind* parameter filters for connections that match the
   (...)
   1221     +------------+----------------------------------------------------+
   1222     """
-> 1223     return self._proc.net_connections(kind)

File ~/db-vv40d-sdyn-femu/.venv/lib/python3.10/site-packages/psutil/_pslinux.py:1726, in wrap_exceptions.<locals>.wrapper(self, *args, **kwargs)
   1724 self._raise_if_zombie()
   1725 if not os.path.exists("%s/%s" % (self._procfs_path, self.pid)):
-> 1726     raise NoSuchProcess(self.pid, self._name)
   1727 raise

NoSuchProcess: process no longer exists (pid=437872)

When I try to directly create a Mapdl instance...

from ansys.mapdl.core import Mapdl

Mapdl(port=50052)

I get the following:

MapdlConnectionError: Unable to connect to MAPDL gRPC instance at dns:///127.0.0.1:50052.
Reached either maximum amount of connection attempts (5) or timeout (15 s).The MAPDL process has died.

When I run the GUI launcher, it starts without any issue however. I am pretty sure (not 100% though) that is has nothing to do with a lack of licenses. Any clue?

Thanks a lot.

[germa89]

Hi @jorisparet

Couple of things... the easiest first:

The following code:

from ansys.mapdl.core import Mapdl

Mapdl(port=50052)

Does not start MAPDL, it just try to connect to it at the given port. That's why it shows the MapdlConnectionError. You should try to use either launch_mapdl or start MAPDL manually.

Regarding the other issue, it seems that there is a conflict with the port. Maybe something was running at a given port, but it died before we could really workout what kind of process it was.

You could try to set port=None, that will bypass the port check so we can see if the rest of the library works. Once we have that running, we can see what is the next step but I envision I might need to add NoSuchProcess as an allowable exception (same as psutil.AccessDenied in line 257 in launcher.py file).

[jorisparet]

Hello @germa89,

Thanks a lot for your quick reply.

The following code:

from ansys.mapdl.core import Mapdl

Mapdl(port=50052)
Does not start MAPDL, it just try to connect to it at the given port. That's why it shows the MapdlConnectionError. You should try to use either launch_mapdl or start MAPDL manually.

Sorry for the mix up, I saw that in another issue (#2702) and thought it was useful for this test. I tried the code snippet you shared to start MAPDL manually, trying to run and exit new instances successively:

from ansys.mapdl.core import LOG

LOG.setLevel("DEBUG")
LOG.log_to_file("mylog.log")

from ansys.mapdl.core import launch_mapdl

for i in range(20):
    mapdl = launch_mapdl(loglevel="DEBUG", license_type='meba')
    mapdl.exit(force=True)

It ended with the same warning and the same NoSuchProcess error I mentioned in my previous post. I attached the log for more detail (mylog.log), but it stops at "PyMAPDL is taking longer than expected to connect to an MAPDL session. Checking if there are any available licenses..." and the error only appears on Python's side.

About the port number, I forgot to mention that I changed the port from 50052 to 50053 in my previous post, because I kept having a PortAlreadyInUseByAnMAPDLInstance error with 50052, even though I systematically exit every open session with mapdl.exit(). But anyway, it looks like I can have the same NoSuchProcess error with port 50052 in my test above. Note that is produces the same result with port=None, as you mentioned.

I had suspicions that it is not always using the right license type, because the person in charge of the licenses in my organization told me I was mostly using "ansys" type licenses, even though I always set license_type="meba" when creating a new instance. The log seems to indicate that I am indeed using a "meba" license, though. I don't think it makes any difference, but maybe you should know about it anyway: in launcher.py, the function signature of launch_mapdl says license_type: Optional[bool] = None even though I assume it should be a str. The docstring does say license_type : str, optional, however.

To add a little more context, I am running this in a Jupyter notebook. But I restart the kernel every time I run the code, so it should kill every instances of MAPDL anyway and free the ports.

[germa89]

Hi @jorisparet

Thank you for the detailed comment.

It seems that PyMAPDL can start at least 3 MAPDL sessions and connect to it. But for some reason, later it fails. I would check what is the output of the latest run in the working directory (in your log: /tmp/ansys_mnmnbggtjm). Check the MAPDL output file there (file0.out or similar) and see if you can get more infor about it. It might be indeed a license problem.

The inner works of the license server are obscure to me. @mcMunich @pmaroneh can a license server give us a ansys license even though we are requesting a meba license?

Thank you for pointing out the typos in the function signature.

Killing Jupyter notebook kernel does not necessary means that the MAPDL instances will be killed. It shoudl if the OS consider that the MAPDL instances are "child processes" of the Python file. But not always. I would make sure you are killing the MAPDL instances, by using mapdl.exit and checking the task manager.

[jorisparet]

Hello @germa89,

Last week, I added the exception that you mentioned in your previous message in launcher.py:

def get_process_at_port(port) -> Optional[psutil.Process]:
        ...
        except psutil.AccessDenied:
            continue
        except psutil.NoSuchProcess: # <-- NEW EXCEPTION HERE
            continue

Since then, I was able to run hundreds of simulations in a row without any problem (I only tried one at a time, though). I only had one crash with a different error, but it was an isolated event. However, I keep having the message

PyMAPDL is taking longer than expected to connect to an MAPDL session.
Checking if there are any available licenses...

at almost every simulation, but it ends up running them anyway after some time. If your colleagues have an idea about why this message keeps appearing and whether the wrong license could be used instead of meba, I am interested.

Unfortunately, I cannot check the log and output files that you mentioned, because I don't have any "failed" simulation files left that I'm aware of. Your fix in launcher.py did the trick, thanks!

[germa89]

Hi @jorisparet

That message is just a warning, for the cases that MAPDL takes more than expected to get a license. It might be showing too much though.

Regarding using wrong license instead of meba, I guess if the license server ran out of meba licenses, then it is giving you back something similar but available. I guess you should check with your license manager if that is the case.

Glad it helps!