Trouble installing numpy and scipy from requirements

[PeterSR]

Trying to install numpy and scipy to use in Python unit test.

Using the following action .github/workflows/python-test.yml:

name: Test
on:
  push:
    branches: '*'
jobs:
  test:
    runs-on: ubuntu-20.04
    steps:
      - name: Checkout
        uses: actions/checkout@v2
        with:
          fetch-depth: 1
      - name: Install package
        run: |
          sudo apt-get -y install gcc gfortran python-dev libopenblas-dev liblapack-dev cython
      - name: Setup Python
        uses: actions/setup-python@v2
        with:
          python-version: '3.8'
      - name: Test
        uses: onichandame/[email protected]

requirements.txt:

numpy~=1.20.1
scipy~=1.6.1

It fails with the following output:
output.txt

The Ubuntu packages installed are based on https://docs.scipy.org/doc/scipy/reference/building/linux.html#debian-ubuntu, but I have also tried a bunch of other packages.

I suspect that packages installed earlier are not visible for this action when it is running.

Any ideas?

[onichandame]

This action is a script running directly on the host machine, not in a container. So the packages installed previously should be seen in this action.

I could not reproduce the same error in either ubuntu:20.04 docker image or native ubuntu:20 box. I guess the only way to debug would be directly running it in the github machines. Would you mind sharing a repository that can reproduce the error so that I can fork and test it?

[PeterSR]

Hi! Thank you for getting back to me so quickly!

I have created the following repo: https://github.com/PeterSR/debug-github-actions-python-unit-test

[onichandame]

from the logs, I found that a docker container was started to run this action, which is not consistent with the docs. I still have no idea why this is, but it is most likely to be the cause of this issue: the container cannot make use of the packages installed on the host machine.

I cannot think of an easy fix to this problem for now, so the simplest workaround is to dump this action and write down every command you need in your action file.

[PeterSR]

Hi!

Thank you for the digging work. It's a shame, but it sounds like that is the best option.

[PeterSR]

Ended up going with this approach: https://github.com/PeterSR/debug-github-actions-python-unit-test/blob/ba8befc50139f6ae78627b1ccd543622b187afca/.github/workflows/python-test.yml

[onichandame]

@PeterSR glad to know it's worked out :D
Just a thought of mine, if pip install is needed in a CI step, it would be beneficial to keep the installed packages in a local virtual environment. Due to the nasty way pip deals with the dependencies, a globally installed package may conflicts with another project which requires the same package but a different version.
In a simple environment with a simple project, this won't be a problem though.

[PeterSR]

That's a good point - But the question is: Does it matter inside GitHub Actions? I am pretty new to it, but it seems very much like Docker, where each build is a separate container, isolated from other projects.

[onichandame]

It depends. refer to the docs (this part can be trusted at least 😆 )
According to the docs, a runner (VM or DinD) is spinned up for every job in an action. The steps in a job are run in containers or directly on the runner, depending on the type of the step. So if multiple projects are involved in the same job, pip could cause a problem.

[PeterSR]

Thank you so much for the link and insights! Very helpful. Will have a look at it.

Btw feel free to close this issue if you find it outside the scope of the project to do anything about it.

[onichandame]

closed until a solution is found