Python shared library & Actions runners

[ijlyttle]

First off, I am trying mightily to keep this from being a "fix-my-runner" issue. Rather, I'm aiming for "how can I give reticulate the information it needs to succeed?"

My company has an instance of GitHub Enterprise; we use a self-hosted ubuntu runner. I am able to get it to work with renv and Quarto actions, but I run into trouble when I try to use reticulate.

Everything works fine on public GitHub:

• using CRAN reticulate
• using dev reticulate

I am using essentially the same workflow file on both GitHub and GitHub Enterprise.

In both places, I run:

  which python
  python --version
  python -c "import sysconfig; print(sysconfig.get_config_var('Py_ENABLE_SHARED'))"

and

  utils::sessionInfo()
  reticulate::py_discover_config()
  reticulate::py_config()

A difference seems to be that on GHE, libpython is not found despite LD_LIBRARY_PATH being set.

As well, FWIW, this seems a little bit like #836.

Thanks for any light you can shed on this!

✅ GitHub

  which python
  python --version
  python -c "import sysconfig; print(sysconfig.get_config_var('Py_ENABLE_SHARED'))"
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.13/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
/opt/hostedtoolcache/Python/3.9.13/x64/bin/python
Python 3.9.13
1

  utils::sessionInfo()
  utils::sessionInfo()
  reticulate::py_discover_config()
  reticulate::py_config()
  shell: /usr/local/bin/Rscript {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.13/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
    R_LIBS_USER: /home/runner/work/_temp/Library
    TZ: UTC
    _R_CHECK_SYSTEM_CLOCK_: FALSE
    NOT_CRAN: true
    RSPM: https://packagemanager.rstudio.com/all/__linux__/focal/latest
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.2.1
python:         /opt/hostedtoolcache/Python/3.9.13/x64/bin/python3
libpython:      /opt/hostedtoolcache/Python/3.9.13/x64/lib/libpython3.9.so
pythonhome:     /opt/hostedtoolcache/Python/3.9.13/x64:/opt/hostedtoolcache/Python/3.9.13/x64
version:        3.9.13 (main, Jul 22 2022, 09:23:34)  [GCC 9.4.0]
numpy:          /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/numpy
numpy_version:  1.23.2

python versions found: 
 /opt/hostedtoolcache/Python/3.9.13/x64/bin/python3
 /usr/share/miniconda/bin/python
python:         /opt/hostedtoolcache/Python/3.9.13/x64/bin/python3
libpython:      /opt/hostedtoolcache/Python/3.9.13/x64/lib/libpython3.9.so
pythonhome:     /opt/hostedtoolcache/Python/3.9.13/x64:/opt/hostedtoolcache/Python/3.9.13/x64
version:        3.9.13 (main, Jul 22 2022, 09:23:34)  [GCC 9.4.0]
numpy:          /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/numpy
numpy_version:  1.23.2

python versions found: 
 /opt/hostedtoolcache/Python/3.9.13/x64/bin/python3
 /usr/share/miniconda/bin/python

❌ GitHub Enterprise

  which python
  python --version
  python -c "import sysconfig; print(sysconfig.get_config_var('Py_ENABLE_SHARED'))"
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64
    LD_LIBRARY_PATH: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib
/home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/bin/python
Python 3.9.13
1

  utils::sessionInfo()
  reticulate::py_discover_config()
  reticulate::py_config()
  shell: /usr/local/bin/Rscript {0}
  env:
    pythonLocation: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64
    LD_LIBRARY_PATH: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib
    R_LIBS_USER: /home/azureuser/actions-runner/_work/_temp/Library
    TZ: UTC
    _R_CHECK_SYSTEM_CLOCK_: FALSE
    NOT_CRAN: true
    RSPM: https://packagemanager.rstudio.com/all/__linux__/focal/latest
R version 4.2.1 (2022-06-23)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
 [7]LC_PAPER=C.UTF-8        LC_NAME=C              LC_ADDRESS=C          
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.2.1
python:         /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/bin/python3
libpython:      [NOT FOUND]
pythonhome:     /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64:/home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64
version:        3.9.13 (main, Jul 22 2022, 09:23:34)  [GCC 9.4.0]
numpy:          /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib/python3.9/site-packages/numpy
numpy_version:  1.23.2

python versions found: 
 /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/bin/python3
 /usr/bin/python3
 /usr/bin/python
Error: Error: Python shared library not found, Python bindings not loaded.
Use reticulate::install_miniconda() if you'd like to install a Miniconda Python environment.
Execution halted
Error: Process completed with exit code 1.

[kevinushey]

How was Python built / installed? Normally, one needs to pass --enable-shared when configuring Python so that the companion libpython.so library is also created during build. My best guess is that this didn't happen?

EDIT: Sorry, I see you're logging that via Py_ENABLE_SHARED. Are you able to provide any other debug output (e.g. is libpython.so installed or available at some location on that runner?)

[kevinushey]

The relevant code for finding libpython is here:

reticulate/R/config.R

Lines 715 to 747 in f77d611

	} else {
	# default to NULL
	libpython <- NULL
	# check multiple library directories
	# (necessary for virtualenvs that don't copy over the shared library)
	libsrcs <- c("LIBPL", "LIBDIR", "Prefix", "ExecPrefix", "BaseExecPrefix")
	for (libsrc in libsrcs) {
	# skip null entries in config
	src <- config[[libsrc]]
	if (is.null(src))
	next
	# get appropriate libpython extension for platform
	ext <- switch(
	Sys.info()[["sysname"]],
	Darwin = ".dylib",
	Windows = ".dll",
	".so"
	)
	# try to resolve libpython in this location
	pattern <- sprintf("^libpython%sd?m?%s", version, ext)
	candidates <- list.files(src, pattern = pattern, full.names = TRUE)
	if (length(candidates)) {
	libpython <- candidates
	break
	}
	}
	}

Are you able to check whether those configuration variables are set as expected?

The output of the following might also be helpful:

reticulate:::python_config_impl("/path/to/python")

as that will dump some of the configuration variables that reticulate uses when introspecting the Python installation.

[ijlyttle]

Sorry - I should have been clearer labeling that first diagnostic 😳

Here's the listing for each LD_LIBRARY_PATH directory (they look the same to me).

I'll spend some time poking around the config.R code.

Thanks!

✅ GitHub

ls -l $LD_LIBRARY_PATH
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /opt/hostedtoolcache/Python/3.9.13/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.9.13/x64/lib
total 21180
lrwxrwxrwx   1 runner runneradmin       19 Aug 14 22:36 libpython3.9.so -> libpython3.9.so.1.0
-rwxrwxrwx   1 runner runneradmin 21653840 Aug 14 22:36 libpython3.9.so.1.0
-rwxrwxrwx   1 runner runneradmin    15648 Aug 14 22:36 libpython3.so
drwxrwxrwx+  2 runner runneradmin     4096 Aug 14 22:36 pkgconfig
drwxrwxrwx+ 36 runner runneradmin    12288 Aug 14 22:36 python3.9

❌ GHE

ls -l $LD_LIBRARY_PATH
  shell: /usr/bin/bash -e {0}
  env:
    pythonLocation: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64
    LD_LIBRARY_PATH: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib
total 21180
lrwxrwxrwx  1 azureuser azureuser       19 Aug 17 21:40 libpython3.9.so -> libpython3.9.so.1.0
-rwxr-xr-x  1 azureuser azureuser 21653840 Aug 17 21:40 libpython3.9.so.1.0
-rwxr-xr-x  1 azureuser azureuser    15648 Aug 17 21:40 libpython3.so
drwxr-xr-x  2 azureuser azureuser     4096 Aug 17 21:40 pkgconfig
drwxr-xr-x 36 azureuser azureuser    12288 Aug 17 21:40 python3.9

[ijlyttle]

As for config: on GHE, there seems to be a mismatch between c("LIBPL", "LIBDIR") and c("Prefix", "ExecPrefix", "BaseExecPrefix").

This line, and the break suggest that reticulate takes the first one it finds, which will be "LIBPL":

reticulate/R/config.R

Line 722 in f77d611

libsrcs <- c("LIBPL", "LIBDIR", "Prefix", "ExecPrefix", "BaseExecPrefix")

Let me try a fork, where it will prefer "Prefix", and see if that solves the problem.

I'll need to talk to "my people", but I think c("LIBPL", "LIBDIR") refer to a standard-load Python for the runner, and I write over it using the actions/setup-python action.

✅ GitHub

reticulate:::python_config_impl(
  paste0(Sys.getenv("pythonLocation"), "/bin/python3")
)

 [1] "Architecture: 64bit"                                                                                                                                                                                                                                         
 [2] "Version: 3.9.13 (main, Jul 22 2022, 09:23:34)  [GCC 9.4.0]"                                                                                                                                                                                                  
 [3] "VersionNumber: 3.9"                                                                                                                                                                                                                                          
 [4] "Prefix: /opt/hostedtoolcache/Python/3.9.13/x64"                                                                                                                                                                                                              
 [5] "ExecPrefix: /opt/hostedtoolcache/Python/3.9.13/x64"                                                                                                                                                                                                          
 [6] "BaseExecPrefix: /opt/hostedtoolcache/Python/3.9.13/x64"                                                                                                                                                                                                      
 [7] "PythonPath: /opt/hostedtoolcache/Python/3.9.13/x64/lib/python39.zip:/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9:/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/lib-dynload:/opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages"
 [8] "LIBPL: /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/config-3.9-x86_64-linux-gnu"                                                                                                                                                                     
 [9] "LIBDIR: /opt/hostedtoolcache/Python/3.9.13/x64/lib"                                                                                                                                                                                                          
[10] "NumpyPath: /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/site-packages/numpy"                                                                                                                                                                         
[11] "NumpyVersion: 1.23.2"        

❌ GHE

reticulate:::python_config_impl(
  paste0(Sys.getenv("pythonLocation"), "/bin/python3")
)

[1] "Architecture: 64bit"                                                                                                                                                                                                                                                                                                                                 
 [2] "Version: 3.9.13 (main, Jul 22 2022, 09:23:34)  [GCC 9.4.0]"                                                                                                                                                                                                                                                                                          
 [3] "VersionNumber: 3.9"                                                                                                                                                                                                                                                                                                                                  
 [4] "Prefix: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64"                                                                                                                                                                                                                                                                                
 [5] "ExecPrefix: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64"                                                                                                                                                                                                                                                                            
 [6] "BaseExecPrefix: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64"                                                                                                                                                                                                                                                                        
 [7] "PythonPath: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib/python39.zip:/home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib/python3.9:/home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib/python3.9/lib-dynload:/home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib/python3.9/site-packages"
 [8] "LIBPL: /opt/hostedtoolcache/Python/3.9.13/x64/lib/python3.9/config-3.9-x86_64-linux-gnu"                                                                                                                                                                                                                                                             
 [9] "LIBDIR: /opt/hostedtoolcache/Python/3.9.13/x64/lib"                                                                                                                                                                                                                                                                                                  
[10] "NumpyPath: /home/azureuser/actions-runner/_work/_tool/Python/3.9.13/x64/lib/python3.9/site-packages/numpy"                                                                                                                                                                                                                                           
[11] "NumpyVersion: 1.23.2"

[ijlyttle]

Hi @kevinushey -

I poked around some more, and I think I have a theory of the case.

The root of the problem seems to be that when I use actions/setup-python@3 on a self-hosted runner on GHE,
is installs Python in a different place from the "existing" installation:

• existing: /opt/hostedtoolcache
• new: /home/azureuser/actions-runner/_work/_tool

Poking around the setup-python repo, I see an issue that looks suspiciously familiar:

• Cannot create directory ‘/opt/hostedtoolcache’ after recent release actions/setup-python#459

I also see a series of recent merged PRs to solve the issue:

• Fix Tool Path handling for self-hosted runners actions/setup-python#466 in v4.2.0
• Assign default value of AGENT_TOOLSDIRECTORY if not set actions/setup-python#394 in v4.1.0
• Force TOOLCACHE_ROOT to be equal AGENT_TOOLSDIRECTORY actions/setup-python#338 in v4.0.0

I realize that I am working with v3 (as this is what my GHE supports now); the larger point is that I don't think this is a reticulate issue - I will have a chat with "my folks" to see what can be done.

With your permission, I'll close this now with my thanks, but I will add to the issue with details on how we get this sorted out, in case someone else might find it interesting.

Thanks!

[kevinushey]

Thanks for reporting back with the extra investigation! Please still let us know if there's something we can do on the reticulate side to smooth this over.