ld: cannot find -lpython3.10

[joaodlf]

Description:
Getting ld: cannot find -lpython3.10when installing some dependencies.

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:

3.10.0

We are trying to move our action to a self hosted environment, this is how the file looks:

name: Deploy

on:
  push:
    branches:
      - master

jobs:
  build:
    runs-on: self-hosted
    container: ubuntu:20.04
    services:
      postgres:
        ...

    steps:
      - uses: actions/checkout@v1

      ...

      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: 3.10.0

      - name: Install poetry
        run: |
          python -m pip install poetry==1.1.11

      - name: Configure poetry
        run: |
          python -m poetry config virtualenvs.in-project true

      - name: Upgrade pip
        run: |
          python -m poetry run pip install --upgrade pip

      - name: Cache the virtualenv
        uses: actions/cache@v2
        with:
          path: ./.venv
          key: ${{ runner.os }}-venv1-${{ hashFiles('**/poetry.lock') }}

      - name: Install dependencies
        run: |
          python -m poetry install

     ...

We've managed to get to the Install dependencies step, most of the dependencies are installing correctly, but when we get to uwsgi, we are getting the following:

2021-11-17T14:24:25.7315863Z       gcc -pthread -o /__w/project/project/.venv/bin/uwsgi -L/opt/hostedtoolcache/Python/3.10.0/x64/lib -Wl,-rpath,/opt/hostedtoolcache/Python/3.10.0/x64/lib core/utils.o core/protocol.o core/socket.o core/logging.o core/master.o core/master_utils.o core/emperor.o core/notify.o core/mule.o core/subscription.o core/stats.o core/sendfile.o core/async.o core/master_checks.o core/fifo.o core/offload.o core/io.o core/static.o core/websockets.o core/spooler.o core/snmp.o core/exceptions.o core/config.o core/setup_utils.o core/clock.o core/init.o core/buffer.o core/reader.o core/writer.o core/alarm.o core/cron.o core/hooks.o core/plugins.o core/lock.o core/cache.o core/daemons.o core/errors.o core/hash.o core/master_events.o core/chunked.o core/queue.o core/event.o core/signal.o core/strings.o core/progress.o core/timebomb.o core/ini.o core/fsmon.o core/mount.o core/metrics.o core/plugins_builder.o core/sharedarea.o core/rpc.o core/gateway.o core/loop.o core/cookie.o core/querystring.o core/rb_timers.o core/transformations.o core/uwsgi.o proto/base.o proto/uwsgi.o proto/http.o proto/fastcgi.o proto/scgi.o proto/puwsgi.o lib/linux_ns.o core/yaml.o core/dot_h.o core/config_py.o plugins/python/python_plugin.o plugins/python/pyutils.o plugins/python/pyloader.o plugins/python/wsgi_handlers.o plugins/python/wsgi_headers.o plugins/python/wsgi_subhandler.o plugins/python/web3_subhandler.o plugins/python/pump_subhandler.o plugins/python/gil.o plugins/python/uwsgi_pymodule.o plugins/python/profiler.o plugins/python/symimporter.o plugins/python/tracebacker.o plugins/python/raw.o plugins/gevent/gevent.o plugins/gevent/hooks.o plugins/ping/ping_plugin.o plugins/cache/cache.o plugins/nagios/nagios.o plugins/rrdtool/rrdtool.o plugins/carbon/carbon.o plugins/rpc/rpc_plugin.o plugins/corerouter/cr_common.o plugins/corerouter/cr_map.o plugins/corerouter/corerouter.o plugins/fastrouter/fastrouter.o plugins/http/http.o plugins/http/keepalive.o plugins/http/https.o plugins/http/spdy3.o plugins/ugreen/ugreen.o plugins/signal/signal_plugin.o plugins/syslog/syslog_plugin.o plugins/rsyslog/rsyslog_plugin.o plugins/logsocket/logsocket_plugin.o plugins/router_uwsgi/router_uwsgi.o plugins/router_redirect/router_redirect.o plugins/router_basicauth/router_basicauth.o plugins/zergpool/zergpool.o plugins/redislog/redislog_plugin.o plugins/mongodblog/mongodblog_plugin.o plugins/router_rewrite/router_rewrite.o plugins/router_http/router_http.o plugins/logfile/logfile.o plugins/router_cache/router_cache.o plugins/rawrouter/rawrouter.o plugins/router_static/router_static.o plugins/sslrouter/sslrouter.o plugins/spooler/spooler_plugin.o plugins/cheaper_busyness/cheaper_busyness.o plugins/symcall/symcall_plugin.o plugins/transformation_tofile/tofile.o plugins/transformation_gzip/gzip.o plugins/transformation_chunked/chunked.o plugins/transformation_offload/offload.o plugins/router_memcached/router_memcached.o plugins/router_redis/router_redis.o plugins/router_hash/router_hash.o plugins/router_expires/expires.o plugins/router_metrics/plugin.o plugins/transformation_template/tt.o plugins/stats_pusher_socket/plugin.o -lpthread -lm -rdynamic -ldl -lcrypt -lpthread -ldl -lutil -lm -lm -lpython3.10 -lcrypt
2021-11-17T14:24:25.7330403Z       /usr/bin/ld: cannot find -lpython3.10
2021-11-17T14:24:25.7331040Z       collect2: error: ld returned 1 exit status
2021-11-17T14:24:25.7331619Z       *** error linking uWSGI ***
2021-11-17T14:24:25.7332292Z       ----------------------------------------```

The step has the following log output (before it is run):

python -m poetry install
  shell: sh -e {0}
  env:
    pythonLocation: /__w/_tool/Python/3.10.0/x64
    LD_LIBRARY_PATH: /__w/_tool/Python/3.10.0/x64/lib

Any help would be appreciated! Have not been able to find much about this issue...

edit:
just tried downgrading python version to 3.8, but same situation with -lpython3.8.

[joaodlf]

Further investigation, I can see the following files in our system:

/opt/hostedtoolcache/Python/3.10.0/x64/lib/libpython3.10.so
/opt/hostedtoolcache/Python/3.10.0/x64/lib/libpython3.10.so.1.0

We are even running the runner as root (RUNNER_ALLOW_RUNASROOT=1 ./run.sh) to avoid any possible permissions issue.

Not sure if this is important (my knowledge over docker and containers is very limited), but the host machine is a Centos 8 box, which I know isn't supported, but I was hoping to circumvent that issue by setting the container to ubuntu:20.04.

[nikita-bykov]

Hello @joaodlf, thanks for reporting! We will take a look.

[dsame]

Hello @joaodlf,

According to README it is necessary to set environment variable AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache in order the python to be installed in /opt/hostedtoolcache where your build expects to find the library

      - name: Setup Python
        uses: actions/setup-python@v2
        env:
          AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
        with:
          python-version: 3.10.0

According to your note

I can see the following files in our system:

/opt/hostedtoolcache/Python/3.10.0/x64/lib/libpython3.10.so
/opt/hostedtoolcache/Python/3.10.0/x64/lib/libpython3.10.so.1.0

I assume you meant the host system while your actual build is executed within the docker container

jobs:
  build:
    runs-on: self-hosted
    container: ubuntu:20.04

This is why the setup-python installs the python binaries into /__w/_tool/Python/3.10.0/x64/lib/ within the docker filesystem

Please confirm AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache solves your problem.
NOTE: you may face an error during the setup-pyton task the binary not found - in this case please wait the PR #338 is merged

[marko-zivic-93]

Hello @joaodlf
Due to the lack of response, I am going to close this issue :)
If you have any additional problems or concerns, please feel free to reopen it or to create a new one.

[gchamon]

@dsame I had this issue with a similar setup as OP and I can confirm, at least on my end, setting AGENT_TOOLSDIRECTORY to /opt/hostedtoolcache solved the issue for me. In order for me not to modify every single workflow file deployed in my organization, I decided to set the variable at the runner level, in the file /opt/actions-runner/.env.

[tyrken]

With our self-hosted runners (pods on EKS helped by https://github.com/actions/actions-runner-controller ) changing AGENT_TOOLSDIRECTORY didn't work for us - still getting gcc compilation errors about missing lpython3.9 library.

Instead we see this action already sets LD_LIBRARY_PATH to a custom value, e.g. /home/runner/_work/_tool/Python/3.9.19/x64/lib for an old app we're testing. While that works for running applications, that env-var isn't used for compilation. Setting LIBRARY_PATH to the same value as LD_LIBRARY_PATH does work.

It would be nice if this action was updated to set both natively...