Slow startup of nrniv -python [find libpython]

[ferdonline]

On some setups, mainly with network filesystems, using PYTHON_DYNAMIC, launching nrniv -python is extremely slow, taking more than 1 minute. This was observed in the neurosim cluster.

The problem boils down to the find (...) libpython in nrnpyenv.sh and its recursive behavior. However, if one uses -maxdepth 1(See PR #560 ) in some cases libpython is not found, like in the Travis CI Python.
By attempting to create a similar setup one could observe that some Pythons put libraries inside a folder x86_64-linux-gnu. Such directory could potentially be know from

import sysconfig
sysconfig.get_config_var('multiarchsubdir')

That is potentially since this is drawn from pure observation, without formal documentation.

Instead of such quick-patch, we could evaluate other solutions eventually more robust, namely:

• Make the bash part work for both Darwin and Linux, where (1) the library directory is attempted from settings in sysconfig, and (2) from by the libraries loaded by the python interpreter using the Linux equivalent of DYLD_PRINT_LIBRARIES: LD_DEBUG=files.

For discussion, a potentially very clean solution but disruptive , would be to replace a call to nrniv -python with python via exec.

[nrnhines]

Not a solution but an expected observation. Launching nrnpyenv.sh will be avoided if there is already a
PYTHONHOME and NRN_PYLIB environment variables set.

[nrnhines]

It has always been a puzzle to me why python just does not robustly give the path of its library via sys.

[pramodk]

Not a solution but an expected observation. Launching nrnpyenv.sh will be avoided if there is already a PYTHONHOME and NRN_PYLIB environment variables set.

@nrnhines : This sounds reasonable to me to avoid the current problem with the wheel.

[pramodk]

@nrnhines : here are the details that you asked on Slack message:

On a machine that take a long time to launch nrniv -python and then import neuron... Now that you know where nrnpyenv.sh is. How long does it take to execute it and what is the PYTHONHOME and NRN_PYENV that it produces? If you export those variables, how long does it then take to launch nrniv -python and import neuron?

I have added set -x to nrnpyenv.sh to print all commands and here is the output:

$ time bash -x /u/kumbhar/conda_env/lib/python3.7/site-packages/neuron/.data/bin/nrnpyenv.sh
+ set -x
+ export originalPATH=/u/kumbhar/conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/u/kumbhar/bin
+ originalPATH=/u/kumbhar/conda_env/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/u/kumbhar/bin
+ export originalPYTHONPATH=/usr/arch/nrn:/usr/arch/nrn:
+ originalPYTHONPATH=/usr/arch/nrn:/usr/arch/nrn:
+ export originalPYTHONHOME=
+ originalPYTHONHOME=
+ export originalLDLIBRARYPATH=
+ originalLDLIBRARYPATH=
+ [[ '' == \-\-\N\E\U\R\O\N\_\H\O\M\E\=* ]]
+ test '' '!=' ''
+ WHICH=which
+ PYTHON=
+ test '' '!=' ''
+ which python3
+ PYTHON=python3
++ which python3
+ echo '# PYTHON=/u/kumbhar/conda_env/bin/python3'
# PYTHON=/u/kumbhar/conda_env/bin/python3
+ z=
+ type -P uname
++ uname
+ z=Linux
+ test Linux = Darwin
+ python3
# find /usr/site/nrniv/local/python/anaconda3/lib/python3.7 -name libpython3.7\*.so
# find /usr/site/nrniv/local/python/anaconda3/lib -name libpython3.7\*.so
# ['/usr/site/nrniv/local/python/anaconda3/lib/libpython3.7m.so']
# items in sys.path = 7
# beginning with sys.prefix = 1
# beginning with site-3 = 3
# in neither location ['.', '/usr/arch/nrn', '/u/kumbhar']
# sys.prefix = /u/kumbhar/conda_env
# site-3 = /usr/site/nrniv/local/python/anaconda3
# if launch python, then need:
export PYTHONPATH="/usr/arch/nrn:/usr/arch/nrn:.:/usr/site/nrniv/local/python/anaconda3/lib/python3.7:/usr/site/nrniv/local/python/anaconda3/lib/python3.7/lib-dynload"
# if launch nrniv, then likely need:
export PYTHONHOME="/u/kumbhar/conda_env"
export NRN_PYLIB="/usr/site/nrniv/local/python/anaconda3/lib/libpython3.7m.so"
real	1m5.652s
user	0m0.791s
sys	0m3.034s

So the culprits here are:

# find /usr/site/nrniv/local/python/anaconda3/lib/python3.7 -name libpython3.7\*.so
# find /usr/site/nrniv/local/python/anaconda3/lib -name libpython3.7\*.so

those took long time to finish.

If I do those manually on command line:

$ time find /usr/site/nrniv/local/python/anaconda3/lib/python3.7 -name libpython3.7\*.so
real	0m39.929s
user	0m0.353s
sys	0m1.568s

If I use sysconfig to find out library path:

$ time python -c "import sysconfig; print(sysconfig.get_config_var('LIBDIR')); print(sysconfig.get_config_var('LIBRARY'))"
/usr/site/nrniv/local/python/anaconda3/lib
libpython3.7m.a

real	0m0.080s
user	0m0.035s
sys	0m0.013s

[nrnhines]

I tried two experiments. Both, failures. First tried the C-API function Py_GetPythonHome(), but, checking it after python3.7.6 was fully initialized, it returned None (it appears to only return the value of the PYTHONHOME environment variable). So the most robust determination of PYTHONHOME I know so far is the value of sys.base_prefix which was introduced in python 3.3.
But one can become slightly more robust by noting that a 2.7 virtualenv created environment has
a sys.base_prefix and a sys.real_prefix

pyenv shell 2.7.17
pyenv virtualenv t27
pyenv shell t27
python
>>> import sys
>>> sys.base_prefix
'/home/hines/.pyenv/versions/2.7.17'
>>> sys.real_prefix
'/home/hines/.pyenv/versions/2.7.17'
>>> sys.prefix
'/home/hines/.pyenv/versions/t27'
>>> 

So the simplest determination of PYTHONHOME appears to be sys.base_prefix if it exists and otherwise fall back to sys.prefix and hope for the best.

The second failure was to try to use dladdr to determine the python library path. This only works if python is not a static executable (which it often is, even though there is also a libpython...so). For example, just for experimental purposes, I repurposed a nrn.Section method so that

{
Dl_info info;
int err = dladdr((const void*)&Py_BuildValue, &info);
char* z = strdup(info.dli_fname);
PyObject* s = Py_BuildValue("s", z);
return s;
}

and it prints

hines@hines-T7500:~/neuron/nrnpyenv-slow/build$ python
Python 3.7.6 (default, Feb 17 2020, 15:09:28) 
[GCC 7.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neuron import h
>>> s = h.Section()
>>> s.x3d()
'/home/hines/.pyenv/versions/3.7.6/bin/python'
>>> 

This is also why I never got much benefit from ldd or otool (also the latter does not exist on mac unless command line tools are installed) Note the size of python and libpython are

-rwxr-xr-x 2 hines hines 14788680 Feb 17 15:16 /home/hines/.pyenv/versions/3.7.6/bin/python3.7
-r-xr-xr-x  1 hines hines 14259496 Feb 17 15:16 /home/hines/.pyenv/versions/3.7.6/lib/libpython3.7m.so.1.0

[pramodk]

So the most robust determination of PYTHONHOME I know so far is the value of sys.base_prefix which was introduced in python 3.3.
But one can become slightly more robust by noting that a 2.7 virtualenv created environment has a sys.base_prefix and a sys.real_prefix
So the simplest determination of PYTHONHOME appears to be sys.base_prefix if it exists and otherwise fall back to sys.prefix and hope for the best.

@nrnhines : I would go for this simple solution. I think in practice it should cover almost all cases (?)

[nrnhines]

I'm very puzzled that even though

ldd /home/hines/.pyenv/versions/3.7.6/bin/python

does not mention libpython. When one launches it and uses lsof to determine the open files, it is listed!

/home/hines/.pyenv/versions/3.7.6/bin/python
>>> import os, subprocess
>>> pid = os.getpid()
>>> cmd = "lsof -p %d"%pid
>>> f = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT).stdout
>>> for line in f:
...   print(line)
...
b'python  38121 hines  txt    REG    8,1 14788680 19056115 /home/hines/.pyenv/versions/3.7.6/bin/python3.7\n'
...
b'python  38121 hines  mem    REG    8,1 14259496 19006500 /home/hines/.pyenv/versions/3.7.6/lib/libpython3.7m.so.1.0\n'
...

[pramodk]

This was investigated. As far as I remember, slow startup of wheel was fixed.