Four days ago, Python installed to /opt/hostedtoolcache, but today it installed to .../_work/_tool.

[dcharles-godaddy]

Description:
Four days ago, Python installed to /opt/hostedtoolcache, but today it installed to .../_work/_tool. This manifested as a failure to find Python.h while building the native dependencies of python-ldap. Somehow, the installation procedure of python-ldap knew to put -I/opt/hostedtoolcache/Python/3.8.10/x64/include/python3.8 on the command line, but there was nothing in the /opt/hostedtoolcache folder. I'm currently working around this by setting the AGENT_TOOLSDIRECTORY environment variable to /opt/hostedtoolcache in my workflow.

Action version:
b55428b

Platform:

• Ubuntu
• macOS
• Windows

Runner type:

• Hosted
• Self-hosted

Tools version:
3.8.10

Repro steps:

      - name: Install Python 3.8.10
        uses: actions/setup-python@b55428b1882923874294fa556849718a1d7f2ca5
        with:
          python-version: '3.8.10'

Expected behavior:
I would expect this to install Python to /opt/hostedtoolcache like it previously did.

Actual behavior:
It now installs Python to .../_work/_tool within the job's workspace.

[vsafonkin]

Hi @dcharles-godaddy, thank you for your report, we will take a look at this.

[taybin]

I ran into this as well. I don't have a workaround yet though.

[szhekpisov]

Was able to apply a workaround by adding an environmental variable, as TS mentioned:

name: Back-End quality gate

env:
  AGENT_TOOLSDIRECTORY: "/opt/hostedtoolcache"

[dcharles-godaddy]

Yes, setting AGENT_TOOLSDIRECTORY is precisely the workaround we applied after 2h of debugging. 👍

[ottumm]

It looks like https://github.com/actions/setup-python/pull/466/files#diff-8164d45a0a98040cbf5161c5b6e696920163460bea5f24100a7a01720c4a9a91 might be responsible for this behavioral change.

[ottumm]

It looks like https://github.com/actions/setup-python/pull/466/files#diff-8164d45a0a98040cbf5161c5b6e696920163460bea5f24100a7a01720c4a9a91 might be responsible for this behavioral change.

Actually, I take this back. That change seems tangentially-related, but in my company's environment we're seeing this problem occur only on our self-hosted runner and not on GitHub hosted runners (we are invoking setup-python exactly the same on both). So, I think this is actually resulting from some change to https://github.com/actions/runner itself.

@dcharles-godaddy are you seeing this on a self-hosted runner or a GitHub hosted runner?

[jatoben]

I encountered this as well, although the self-hosted runner I'm using has not used /opt/hostedtoolcache since it was installed months ago. /opt is completely empty and the runner user does not have write access to it; actions/setup-python appears to have always put Python in the _work/_tool subdirectory.

Since I did not want to change the runner's configuration, I worked around the problem by setting the CFLAGS env var in my actions workflow, e.g.:

- name: build dependencies
  env:
    CFLAGS: "-I/path/to/runner/_work/_tool/Python/3.10.1/x64/include/python3.10"
  run: ...

This let my build find Python.h successfully.

However, nothing has changed recently with the runner, the workflow, or the Python version I'm using, so I'd love to understand why it's now getting the include path wrong.

[dcharles-godaddy]

@ottumm , I'm seeing it on a self-hosted runner:

Current runner version: '2.295.0'

It's actually the same version of the runner producing both the success and the failure.

[eware-godaddy]

We're also seeing the same issue. Unfortunately we don't have permissions to create the /opt/hostedtoolcache/ folder using the AGENT_TOOLSDIRECTORY workaround. Will try the include hack and see if that works.

[eware-godaddy]

The include hack did work for us to move forward.

[eware-godaddy]

@vsafonkin - we hit this issue yesterday as long as quite a few other python teams in our company. The commonality seems to be running on ubuntu on a self-hosted GHA runner. The CFLAGS workaround mentioned above is helping teams to move forward on this. Is there any more information we can provide to diagnose / resolve as this is affecting a lot of teams and GHA workflows. I haven't been able to track this down to any changes in setup-python or our infrastructure, or correlation with a runner version (unless asset caching is masking a change in the runner version).

[vsafonkin]

Hi @eware-godaddy, you can use any custom directory to install python on your self-hosted runner using AGENT_TOOLSDIRECTORY environment variable.

https://github.com/actions/setup-python/blob/main/docs/advanced-usage.md#linux

[vsafonkin]

I'm going to close this issue. Please, feel free to reopen it if you have any concerns.