Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

self-hosted tools cache setup not working automatically on Linux & Mac with v2 #89

Closed
shoukoo opened this issue May 4, 2020 · 12 comments
Assignees
Labels
bug Something isn't working v2 relates to the v2 version of setup-python

Comments

@shoukoo
Copy link

shoukoo commented May 4, 2020

Hi, I am getting Version 2.x with arch x64 not found error message from my self-hosted runner on ubuntu 18.04. However, I can see that version 2.7.18 is in the versions-manifest.json file.

Run actions/setup-python@v2
Version 2.x was not found in the local cache
Version 2.x is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/2.7.18-20200429.22/python-2.7.18-ubuntu-1804-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./7ab082fa-bc6a-4a50-9c0e-bb389ac0bcfd -f /home/runner/actions-runner/_work/_temp/7ab082fa-bc6a-4a50-9c0e-bb389ac0bcfd
Execute installation script
##[error]Version 2.x with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/master/versions-manifest.json

manifest.json

 {
    "version": "2.7.17",
    "stable": true,
    "release_url": "https://github.com/actions/python-versions/releases/tag/2.7.17-20200429.31",
    "files": [
      {
        "filename": "python-2.7.17-macos-1014-x64.tar.gz",
        "arch": "x64",
        "platform": "darwin",
        "platform_version": "10.14",
        "download_url": "https://github.com/actions/python-versions/releases/download/2.7.17-20200429.31/python-2.7.17-macos-1014-x64.tar.gz"
      },
      {
        "filename": "python-2.7.17-macos-1014-x64.tar.gz",
        "arch": "x64",
        "platform": "darwin",
        "platform_version": "10.15",
        "download_url": "https://github.com/actions/python-versions/releases/download/2.7.17-20200429.31/python-2.7.17-macos-1014-x64.tar.gz"
      },
      {
        "filename": "python-2.7.17-ubuntu-1604-x64.tar.gz",
        "arch": "x64",
        "platform": "linux",
        "platform_version": "16.04",
        "download_url": "https://github.com/actions/python-versions/releases/download/2.7.17-20200429.31/python-2.7.17-ubuntu-1604-x64.tar.gz"
      },
      {
        "filename": "python-2.7.17-ubuntu-1804-x64.tar.gz",
        "arch": "x64",
        "platform": "linux",
        "platform_version": "18.04",
        "download_url": "https://github.com/actions/python-versions/releases/download/2.7.17-20200429.31/python-2.7.17-ubuntu-1804-x64.tar.gz"
      },
      {
        "filename": "python-2.7.17-windows-2016-x64.zip",
        "arch": "x64",
        "platform": "win32",
        "download_url": "https://github.com/actions/python-versions/releases/download/2.7.17-20200429.31/python-2.7.17-windows-2016-x64.zip"
      },
      {
        "filename": "python-2.7.17-windows-2016-x86.zip",
        "arch": "x86",
        "platform": "win32",
        "download_url": "https://github.com/actions/python-versions/releases/download/2.7.17-20200429.31/python-2.7.17-windows-2016-x86.zip"
      }
    ]
  }
@shoukoo
Copy link
Author

shoukoo commented May 4, 2020

I am also getting the error message when installing python 3.x

Version 3.x was not found in the local cache
Version 3.x is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.8.2-20200429.26/python-3.8.2-ubuntu-1804-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./cc4c6f29-d541-47e7-ba23-ddf4a88a26a5 -f /home/runner/actions-runner/_work/_temp/cc4c6f29-d541-47e7-ba23-ddf4a88a26a5
Execute installation script
##[error]Version 3.x with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/master/versions-manifest.json

@konradpabjan
Copy link
Collaborator

konradpabjan commented May 4, 2020

This looks like the local tools cache setup isn't working for the first time when running on linux?

The download appears to be fine, however, the setup script must be running into some issues. Investigating... (most likely an env variable isn't set or something)


Edit: If you set an env variable called AGENT_TOOLSDIRECTORY with something such as export AGENT_TOOLSDIRECTORY=~ it will almost successfully set everything up. Still figuring out why it doesn't work..

If the value is set to ~, it would create a directory ~/Python and everything would be copied over and setup in there.

@konradpabjan konradpabjan added the bug Something isn't working label May 4, 2020
@konradpabjan konradpabjan changed the title Version 2.x with arch x64 not found self-hosted tools cache setup not working on Linux with v2 May 4, 2020
@konradpabjan konradpabjan self-assigned this May 4, 2020
@konradpabjan konradpabjan added the v2 relates to the v2 version of setup-python label May 4, 2020
@bharath-kotha
Copy link

I'm also getting a similar error but it says that bash has exited with exit code 1 while installing.

Run actions/setup-python@v2
  with:
    python-version: 3.6
    architecture: x64
    token: ***
Version 3.6 was not found in the local cache
Version 3.6 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.6.10-20200429.8/python-3.6.10-ubuntu-1804-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./6eef5cd6-1d7a-44d8-b3ad-e88d6b006160 -f /home/ubuntu/actions-runner/_work/_temp/6eef5cd6-1d7a-44d8-b3ad-e88d6b006160
Execute installation script
##[error]The process '/bin/bash' failed with exit code 1

@galleon
Copy link

galleon commented May 6, 2020

Same issue on macos as well.
Is that connected to the fact of running Runner with elevated rights ?

@freeatnet
Copy link

@konradpabjan Happy Satellite day!

I'm seeing a similar issue with a self-hosted runner I set up yesterday on Ubuntu 18.04. Here is what happens when I try to set up Python 3.8 with the latest actions/python-versions release:

Version 3.8 was not found in the local cache
Version 3.8 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.8.2-20200505.19/python-3.8.2-ubuntu-1804-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./d34df31f-a1dc-4f12-97fd-72c05b03d365 -f /home/runner/actions-runner/_work/_temp/d34df31f-a1dc-4f12-97fd-72c05b03d365
Execute installation script
##[error]The process '/bin/bash' failed with exit code 127

Are the steps to debugging this documented somewhere? I don't see any output from setup.sh, can I fork and modify python-versions to debug it?

@konradpabjan
Copy link
Collaborator

konradpabjan commented May 6, 2020

So getting this to work automatically on Mac and Linux is going to require some extra steps. I'm working on updating the README with a easy(ish) guide on how to do this after I figure out the last step to this puzzle.

There are a couple of things going on here.

  • The AGENT_TOOLSDIRECTORY env variable needs to be set to /opt/hostedtoolcache (on Linux at least, different on Mac). We compile Python in this directory with --enable-shared so it is non-portable and it has to always be located in the same place (see here). If the path is different you'll get an error like error while loading shared libraries: libpython3.7m.so.1.0 when reaching the end of setup.sh
  • Because we're in the /opt/hostedtoolcache directory, you need to be running as sudo to modify this. However, if you're tried starting the runner with sudo, you'll get an error that says Must not run interactively with sudo. To get around this, you have to configure passwordless sudo which is what our hosted runners do
  • Even with both set, there needs to either be some modifications done to the setup.sh file that is downloaded so that all that necessary commands like mkdir run with sudo or so that setup-python does all the necessary magic behind the scenes (what i'm trying to get working)

@konradpabjan
Copy link
Collaborator

@freeatnet Thanks for the wishes! There were some pretty cool stuff announced at satellite that we can't wait for everyone to get their hands on😀

You can go ahead and fork this! We have some guidelines on how you can debug and test locally here

The output of setup.sh is silenced, that is configured by exec here

You can also run the setup.sh file locally. So if you have a self-hosted runner configured and you run a workflow with setup-python@v2, there is a _work directory. If you navigate down you should see something like _work/my-repo/my-repo/some-crazy-hash-that-is-unique-per-run/. Inside here you'll find the setup.sh file that gets downloaded along with the .tgz file for a specific version of Python.

@freeatnet
Copy link

freeatnet commented May 7, 2020

@konradpabjan You're spot on with the shared library issue, that's actually where I stopped last time. However, I see that the runner sets RUNNER_TOOL_CACHE=/home/runner/actions-runner/_work/_tool, which is now picked up by the setup script. Is that the right path or should RUNNER_TOOL_CACHE be /opt/hostedtoolcache?

Edit: Well, this is a terrible, terrible, no-good hack, but this works:

      env:
        LD_LIBRARY_PATH: /home/runner/actions-runner/_work/_tool/Python/3.8.2/x64/lib

@BrightRan
Copy link

@konradpabjan konradpabjan changed the title self-hosted tools cache setup not working on Linux with v2 self-hosted tools cache setup not working automatically on Linux & Mac with v2 May 8, 2020
@konradpabjan
Copy link
Collaborator

konradpabjan commented May 8, 2020

Figured it out, PR to update the README with some new setup info. #90

It's not going to be fully automatic and some manual setups are required, but it works!

@konradpabjan
Copy link
Collaborator

The README has been updated with instructions on how to configure self-hosted runners! Should work now for Mac and Linux. Windows has gotten some more instructions too.

@Mogost
Copy link

Mogost commented May 13, 2020

I followed all the recommendations from the README file, but still get the error. I don’t know where I could make a mistake. Everywhere as far as I can see the correct permissions and recommended variables. In the logs, I do not see anything critical.

 Set up Python13s
##[error]The process '/bin/bash' failed with exit code 1
Run actions/setup-python@v2
  with:
    python-version: 3.6
    token: ***
  env:
    SKIP_SELENIUM_TESTS: 1
    DJANGO_SETTINGS_MODULE: varys.settings.test
Version 3.6 was not found in the local cache
Version 3.6 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.6.10-20200505.23/python-3.6.10-ubuntu-1604-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./050358cb-c7cf-46ce-bcfc-2daf00a5f799 -f /home/github/actions-runner/_work/_temp/050358cb-c7cf-46ce-bcfc-2daf00a5f799
Execute installation script
##[error]The process '/bin/bash' failed with exit code 1```

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working v2 relates to the v2 version of setup-python
Projects
None yet
Development

No branches or pull requests

7 participants