Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

self-hosted tools cache setup not working automatically on Linux & Mac with v2 #92

Closed
shoukoo opened this issue May 15, 2020 · 20 comments
Closed
Assignees
Labels
bug Something isn't working

Comments

@shoukoo
Copy link

shoukoo commented May 15, 2020

The issue is related to this PR #89
I am still getting an error when using setup-python v2 on self-hosted runners on ubuntu18.04

Run actions/setup-python@v2
Version 3.8 was not found in the local cache
Version 3.8 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.8.3-20200514.1/python-3.8.3-ubuntu-1804-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./72938570-f04c-4e44-b107-26e5832ecd2d -f /home/runner/_work/_temp/72938570-f04c-4e44-b107-26e5832ecd2d
Execute installation script
##[error]Version 3.8 with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/master/versions-manifest.json

I have set AGENT_TOOLSDIRECTORY in the config file

- name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v2
        env:
          AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
        with:
          python-version: "${{ matrix.python-version }}"

I can see it created some folders and files under /opt/hostedtoolcache

ubuntu@ip-xx-xx-xx-xx:/opt/hostedtoolcache/Python$ ls
3.7.7  3.8.3
@konradpabjan
Copy link
Collaborator

Thanks for opening this issue!

What happens if you navigate into one of the directories and attempt to run Python from there? If you go to /opt/hostedtoolcache/Python/3.7.7/x64/bin for example and try to run ./python. It should open up a Python shell with Python 3.7.7 (default, May..

Based on the logs, if appears the setup and download goes through successfully, however it later can't find what it installed... 🤔

I'm assuming you're using 3.7 and 3.8 in your YAML. Do you get the same error if you change it to include the patch version as well? (something like 3.7.7 and 3.8.3)

@shoukoo
Copy link
Author

shoukoo commented May 21, 2020

@konradpabjan, thanks for responding!

This is the result I got when trying 3.7.7 directly

Run zendesk/setup-python@v2
  with:
    python-version: 3.7.7
    token: ***
  env:
    PIPENV_PYTHON: /home/runner/.pyenv/shims/python
    AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
Version 3.7.7 was not found in the local cache
Version 3.7.7 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.7.7-20200505.20/python-3.7.7-ubuntu-1804-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./01975160-cf7b-425d-bfb5-fd017b03d44b -f /home/runner/_work/_temp/01975160-cf7b-425d-bfb5-fd017b03d44b
Execute installation script
##[error]Version 3.7.7 with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/master/versions-manifest.json

I am able open the python shell when calling python directly from the bin folder.

runner@ip-172-xxx-xxx-xxx:~$ /opt/hostedtoolcache/Python/3.7.7/x64/bin/python
Python 3.7.7 (default, May  5 2020, 13:33:17)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

Also runner user has the right permission to modify content in the folder

drwxr-xr-x  3 runner runner   20 May 21 00:14 hostedtoolcache

@konradpabjan
Copy link
Collaborator

konradpabjan commented May 21, 2020

So the install and setup appear to be working fine, it must be failing at this step

Digging a little further into the tc.find method, it must be throwing an error sometime around here 🤔

There is this _getCacheDirectory() method that looks at the RUNNER_TOOL_CACHE env variable. Could you try overriding the env variable to be the same thing as the value of AGENT_TOOLSDIRECTORY? I wonder if this env variable is the problem.

Also, could you turn on step debugging? I don't think it's currently on and it might provide some extra information: how to turn on step-debugging

@konradpabjan
Copy link
Collaborator

standby... i have a repo on another machine

@konradpabjan konradpabjan self-assigned this May 22, 2020
@tabossert
Copy link

tabossert commented May 23, 2020

Same issue for me 3.x works fine and installs 3.8, but if I specify 3.7.x or 3.7.7 it fails even though it did download and executing python from the installed directory invokes the python shell

/opt/hostedtoolcache/Python/3.7.7/x64/bin$ ./python --version
Python 3.7.7

but still getting
Version 3.7.7 was not found in the local cache Version 3.7.7 is available for downloading Download from "https://github.com/actions/python-versions/releases/download/3.7.7-20200505.20/python-3.7.7-ubuntu-1804-x64.tar.gz" Extract downloaded archive /bin/tar xz --warning=no-unknown-keyword -C ./5699b3cd-9f58-4e1e-89bc-c6f625e45879 -f /home/ubuntu/actions-runner/_work/_temp/5699b3cd-9f58-4e1e-89bc-c6f625e45879 Execute installation script ##[error]Version 3.7.7 with arch x64 not found The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/master/versions-manifest.json

@shoukoo
Copy link
Author

shoukoo commented May 25, 2020

I also can confirm that 2.7 and 3.6 works for me because they are already pre-installed on the server

Run zendesk/setup-python@v2
  with:
    python-version: 3.6
    token: ***
Version 3.6 was not found in the local cache
Version 3.6 is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.6.10-20200505.23/python-3.6.10-ubuntu-1804-x64.tar.gz"
Extract downloaded archive
/bin/tar xz --warning=no-unknown-keyword -C ./c9c172be-cf2b-4fd4-9c41-e1e153690009 -f /home/runner/_work/_temp/c9c172be-cf2b-4fd4-9c41-e1e153690009
Execute installation script
Successfully setup CPython (3.6.10)

@pgalvarez-elsevier
Copy link

Same issue here as of today ✋

        uses: actions/setup-python@v2
        with:
          python-version: '3.8.x'
Version 3.8.x is available for downloading
Download from "https://github.com/actions/python-versions/releases/download/3.8.5-20200721.1/python-3.8.5-linux-20.04-x64.tar.gz"
Extract downloaded archive
/usr/bin/tar xz --warning=no-unknown-keyword -C /githubrunner/_work/_temp/0cb745b0-0112-4ac5-8ffc-c59a1cbc0c73 -f /githubrunner/_work/_temp/065b59bc-c4ab-48ea-87ef-0451ea345e9b
Execute installation script
Check if Python hostedtoolcache folder exist...
Deleting Python 3.8.5
Create Python 3.8.5 folder
Copy Python binaries to hostedtoolcache folder
Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
Upgrading PIP...
Looking in links: /tmp/tmphapnm5zt
Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages (47.1.0)
Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.8.5/x64/lib/python3.8/site-packages (20.1.1)
Collecting pip
Using cached pip-20.2.3-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Successfully installed pip-20.2.3
Create complete file
##[error]Version 3.8.x with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
Python 3.8.5 (default, Jul 21 2020, 12:23:14) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>```

Thanks!

@estraph
Copy link

estraph commented Sep 29, 2020

Same problem with self-hosted agent running on Amazon Linux 2.

@konradpabjan konradpabjan removed their assignment Oct 14, 2020
@jimmybrancaccio
Copy link

jimmybrancaccio commented Oct 23, 2020

@estraph Amazon Linux won't work.

That said, I am still having this issue.

2020-10-23T14:58:14.0106485Z ##[group]Run actions/setup-python@v2
2020-10-23T14:58:14.0106894Z with:
2020-10-23T14:58:14.0107163Z   python-version: 3.9.0
2020-10-23T14:58:14.0107768Z   token: ***
2020-10-23T14:58:14.0108061Z env:
2020-10-23T14:58:14.0108329Z   AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
2020-10-23T14:58:14.0108597Z ##[endgroup]
2020-10-23T14:58:14.1488736Z Version 3.9.0 was not found in the local cache
2020-10-23T14:58:14.7159015Z Version 3.9.0 is available for downloading
2020-10-23T14:58:14.7165217Z Download from "https://github.com/actions/python-versions/releases/download/3.9.0-83838/python-3.9.0-linux-18.04-x64.tar.gz"
2020-10-23T14:58:18.8533536Z Extract downloaded archive
2020-10-23T14:58:18.8803967Z [command]/bin/tar xz --warning=no-unknown-keyword -C /home/github/_work/_temp/1c4ea770-c9d0-4502-8675-f33c0e823638 -f /home/github/_work/_temp/3af1c900-a83d-4c12-99d8-6f11778befe2
2020-10-23T14:58:24.6292401Z Execute installation script
2020-10-23T14:58:24.6564478Z Check if Python hostedtoolcache folder exist...
2020-10-23T14:58:24.6580347Z Deleting Python 3.9.0
2020-10-23T14:58:25.4109740Z Create Python 3.9.0 folder
2020-10-23T14:58:25.4143364Z Copy Python binaries to hostedtoolcache folder
2020-10-23T14:58:27.4744259Z Create additional symlinks (Required for the UsePythonVersion Azure Pipelines task and the setup-python GitHub Action)
2020-10-23T14:58:27.4778865Z Upgrading PIP...
2020-10-23T14:58:30.7909963Z Looking in links: /tmp/tmp4bmatpf1
2020-10-23T14:58:30.7938320Z Requirement already satisfied: setuptools in /opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages (49.2.1)
2020-10-23T14:58:30.8293801Z Requirement already satisfied: pip in /opt/hostedtoolcache/Python/3.9.0/x64/lib/python3.9/site-packages (20.2.3)
2020-10-23T14:58:34.2365954Z Collecting pip
2020-10-23T14:58:34.2511641Z Using cached pip-20.2.4-py2.py3-none-any.whl (1.5 MB)
2020-10-23T14:58:34.3945815Z Installing collected packages: pip
2020-10-23T14:58:36.5817771Z Successfully installed pip-20.2.4
2020-10-23T14:58:36.6907566Z Create complete file
2020-10-23T14:58:36.6988191Z ##[error]Version 3.9.0 with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
github@github-runner-jimmybrancaccio-teefury-scraper-85b67b9f67-bjwkj:/opt/hostedtoolcache/Python/3.9.0/x64/bin$ ./python
Python 3.9.0 (default, Oct  6 2020, 11:05:56) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
github@github-runner-jimmybrancaccio-teefury-scraper-85b67b9f67-bjwkj:/opt/hostedtoolcache/Python/3.9.0/x64/bin$ ./python --version
Python 3.9.0

@williamdes
Copy link

Same issue using the tool act

[Run tests/Test on python 3.x and ubuntu-latest]   ❗  ::error::Version 3.x with arch x64 not found%0AThe list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
[Run tests/Test on python 3.x and ubuntu-latest]   ❌  Failure - Use python ${{ matrix.python-version }}
Error: exit with `FAILURE`: 1

@ghost
Copy link

ghost commented Mar 9, 2021

Same here just like williamdes

[main/Checking before and after merging into master] Exec command '[node /actions/actions-setup-python@v2/dist/index.js]'
[main/Checking before and after merging into master]   💬  ::debug::Semantic version spec of 3.9 is 3.9
[main/Checking before and after merging into master]   💬  ::debug::isExplicit: 
[main/Checking before and after merging into master]   💬  ::debug::explicit? false
[main/Checking before and after merging into master]   💬  ::debug::evaluating 0 versions
[main/Checking before and after merging into master]   💬  ::debug::match not found
| Version 3.9 was not found in the local cache
[main/Checking before and after merging into master]   💬  ::debug::check 3.10.0-alpha.5 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.10.0-alpha.4 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.10.0-alpha.3 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.10.0-alpha.2 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.10.0-alpha.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.2 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && darwin===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && win32===linux
[main/Checking before and after merging into master]   💬  ::debug::x86===x64 && win32===linux
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.2-rc.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && darwin===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && win32===linux
[main/Checking before and after merging into master]   💬  ::debug::x86===x64 && win32===linux
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.1-rc.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.0 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && darwin===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && linux===linux
[main/Checking before and after merging into master]   💬  ::debug::x64===x64 && win32===linux
[main/Checking before and after merging into master]   💬  ::debug::x86===x64 && win32===linux
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.0-rc.2 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.0-rc.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.0-beta.5 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.9.0-beta.4 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.8 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.7 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.6 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.5 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.4 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.3 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.2 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.8.0 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.7.10 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.7.9 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.7.8 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.7.7 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.7.6 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.7.5 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.7.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.6.13 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.6.12 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.6.11 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.6.10 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.6.9 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.6.8 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.6.7 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.5.10 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.5.9 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.5.4 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.4.10 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.4.4 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.3.7 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.3.5 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.2.5 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.1.4 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 3.0.1 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 2.7.18 satisfies 3.9
[main/Checking before and after merging into master]   💬  ::debug::check 2.7.17 satisfies 3.9
[main/Checking before and after merging into master]   ❗  ::error::Version 3.9 with arch x64 not found%0AThe list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json
[main/Checking before and after merging into master]   ❌  Failure - Set up Python ${{ matrix.python-version }}

@timdhughes
Copy link

timdhughes commented May 7, 2021

Same here calling setup-python on a self-hosted agent to allow for a custom 'pip install' command. The lines below are in the codeql-analysis.yml file, based on the comments on this page and the setup-python reference page: https://github.com/actions/setup-python#using-setup-python-with-a-self-hosted-runner.

Current runner version: '2.273.5'
EC2 runner instance: t2.large, 2 cpu 8GB with Amazon Linux 2 + added packages

    name: Analyze
    runs-on: [ self-hosted ]

    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Setup agent tools directory
      run: |
        sudo mkdir -p -m 777 /opt/hostedtoolcache
        export AGENT_TOOLSDIRECTORY=/opt/hostedtoolcache
    - name: Set up Python
      uses: actions/setup-python@v2
      env:
        AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
      with:
        python-version: '3.8.2'

    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip

Below is the error message from the logs:

Run actions/setup-python@v2
  with:
    python-version: 3.8.2
    token: ***
  env:
    AGENT_TOOLSDIRECTORY: /opt/hostedtoolcache
Version 3.8.2 was not found in the local cache
Error: Version 3.8.2 with arch x64 not found
The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json

@brcrista
Copy link
Contributor

@timdhughes I believe you're hitting that error because you're on Amazon Linux 2. From the doc you linked:

Python distributions are only available for the same environments that GitHub Actions hosted environments are available for. If you are using an unsupported version of Ubuntu such as 19.04 or another Linux distribution such as Fedora, setup-python will not work.

The reason for this limitation is that the binaries are pre-built targeting Ubuntu.

@timdhughes
Copy link

@timdhughes I believe you're hitting that error because you're on Amazon Linux 2. From the doc you linked:

Python distributions are only available for the same environments that GitHub Actions hosted environments are available for. If you are using an unsupported version of Ubuntu such as 19.04 or another Linux distribution such as Fedora, setup-python will not work.

The reason for this limitation is that the binaries are pre-built targeting Ubuntu.

@brcrista Amazon Linux 2 (Red Hat Enterprise Linux 7) looks to be supported for self-hosted runners from this page:
https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#supported-architectures-and-operating-systems-for-self-hosted-runners

@brcrista
Copy link
Contributor

Yes, self-hosted runners are supported on RHEL but the precompiled setup-python binaries are not. For a self-hosted runner on RHEL, you'll have to install or compile your own version of Python.

@Gladdstone
Copy link

@brcrista is there a possibility of adding RHEL support or is that not something that's on the horizon?

@brcrista
Copy link
Contributor

brcrista commented Sep 1, 2021

No, we aren't prioritizing this right now.

@nikita-bykov nikita-bykov added the bug Something isn't working label Dec 6, 2021
fangohr referenced this issue in oscovida/oscovida.github.io Dec 23, 2021
- attempt to use system python i
- the bug seems to be reported but not fixed:
  - https://github.com/actions/setup-python/issues/92��
@poolpitako
Copy link

This also happens in the vanilla debian image in google cloud

@e-korolevskii
Copy link
Contributor

Hi @poolpitako,
Could you provide more details? Perhaps you can create a new issue that is more relevant. Thank you!

@e-korolevskii e-korolevskii self-assigned this Jul 7, 2022
@poolpitako
Copy link

Hi @poolpitako, Could you provide more details? Perhaps you can create a new issue that is more relevant. Thank you!

I am very sorry but I don't remember what was my issue and I moved on.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests