Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI][Dev][Archery] ARM self-hosted runners fail to set up archery due to missing Python.h when installing ruamel.yaml #37999

Closed
raulcd opened this issue Oct 3, 2023 · 2 comments · Fixed by #38009

Comments

@raulcd
Copy link
Member

raulcd commented Oct 3, 2023

Describe the bug, including details regarding any error messages, version, and platform.

There seems to be something going on our self-hosted runners, possibly related to the new Python version?
When installing archery is failing to pip install ruamel.yaml due to missing Python.h. The errors I am seeing are:

Run python3 -m pip install -e dev/archery[docker]
  python3 -m pip install -e dev/archery[docker]
  shell: /usr/bin/bash -e {0}
  env:
    ARROW_ENABLE_TIMING_TESTS: OFF
    DOCKER_VOLUME_PREFIX: .docker/
    ARCH: arm64v8
    ARROW_SIMD_LEVEL: 
    CLANG_TOOLS: 10
    LLVM: 10
    UBUNTU: 20.04
Obtaining file:///runner/_work/arrow/arrow/dev/archery
Collecting click>=7
  Downloading click-8.1.7-py3-none-any.whl (97 kB)
Collecting python-dotenv
  Downloading python_dotenv-1.0.0-py3-none-any.whl (19 kB)
Collecting ruamel.yaml
  Downloading ruamel.yaml-0.17.33-py3-none-any.whl (112 kB)
Collecting ruamel.yaml.clib>=0.2.7; platform_python_implementation == "CPython" and python_version < "3.12"
  Downloading ruamel.yaml.clib-0.2.8.tar.gz (213 kB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Building wheels for collected packages: ruamel.yaml.clib
  Building wheel for ruamel.yaml.clib (PEP 517): started
  Building wheel for ruamel.yaml.clib (PEP 517): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /tmp/tmpoo46iv2v build_wheel /tmp/tmpderco71j
       cwd: /tmp/pip-install-whusjjd5/ruamel.yaml.clib
  Complete output (20 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-aarch64-cpython-38
  creating build/lib.linux-aarch64-cpython-38/ruamel
  creating build/lib.linux-aarch64-cpython-38/ruamel/yaml
  creating build/lib.linux-aarch64-cpython-38/ruamel/yaml/clib
  copying ./__init__.py -> build/lib.linux-aarch64-cpython-38/ruamel/yaml/clib
Failed to build ruamel.yaml.clib
  copying ./setup.py -> build/lib.linux-aarch64-cpython-38/ruamel/yaml/clib
  copying ./LICENSE -> build/lib.linux-aarch64-cpython-38/ruamel/yaml/clib
  running build_ext
  building '_ruamel_yaml' extension
  creating build/temp.linux-aarch64-cpython-38
  aarch64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/usr/include/python3.8 -c _ruamel_yaml.c -o build/temp.linux-aarch64-cpython-38/_ruamel_yaml.o
  _ruamel_yaml.c:16:10: fatal error: Python.h: No such file or directory
     16 | #include "Python.h"
        |          ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/aarch64-linux-gnu-gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for ruamel.yaml.clib
ERROR: Could not build wheels for ruamel.yaml.clib which use PEP 517 and cannot be installed directly
Error: Process completed with exit code 1.

This is happening on PRs and main:
https://github.com/apache/arrow/actions/runs/6397483310/job/17365504255?pr=37901
https://github.com/apache/arrow/actions/runs/6397483312/job/17365504903?pr=37901
https://github.com/apache/arrow/actions/runs/6397427693/job/17365323528

Component(s)

Archery, Continuous Integration, Developer Tools

@raulcd
Copy link
Member Author

raulcd commented Oct 3, 2023

@assignUser @amaldonadomat can you investigate this issue with our self-hosted runners? I'll be traveling for the rest of the week and won't be able to follow up

@assignUser
Copy link
Member

assignUser commented Oct 3, 2023

I think it's missing libpython3-dev?

raulcd added a commit to raulcd/arrow that referenced this issue Oct 4, 2023
assignUser pushed a commit that referenced this issue Oct 4, 2023
…s to Python.h (#38009)

### Rationale for this change
Currently CI on ARM is failing due to ruamel.yaml requiring Python.h

### What changes are included in this PR?

Install python3-dev

### Are these changes tested?

Yes, CI
### Are there any user-facing changes?

No
* Closes: #37999

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
@assignUser assignUser added this to the 14.0.0 milestone Oct 4, 2023
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 23, 2023
… access to Python.h (apache#38009)

### Rationale for this change
Currently CI on ARM is failing due to ruamel.yaml requiring Python.h

### What changes are included in this PR?

Install python3-dev

### Are these changes tested?

Yes, CI
### Are there any user-facing changes?

No
* Closes: apache#37999

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
… access to Python.h (apache#38009)

### Rationale for this change
Currently CI on ARM is failing due to ruamel.yaml requiring Python.h

### What changes are included in this PR?

Install python3-dev

### Are these changes tested?

Yes, CI
### Are there any user-facing changes?

No
* Closes: apache#37999

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
… access to Python.h (apache#38009)

### Rationale for this change
Currently CI on ARM is failing due to ruamel.yaml requiring Python.h

### What changes are included in this PR?

Install python3-dev

### Are these changes tested?

Yes, CI
### Are there any user-facing changes?

No
* Closes: apache#37999

Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Jacob Wujciak-Jens <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment