Nightly builds broken on Linux

[mhdawson]

Discussing this with @richardlau it looks like a recent change to he error is coming out of tools/gyp/pylib/gyp/common.py has caused the release machines to fail building nightlies:

https://nodejs.org/download/nightly/v23.0.0-nightly202405131d7d094a98/

Most likely nodejs/node#52835

@richardlau mentioned that the test machines have a newer version of Python due to some other issue, so that PR might have broken compatibility with Python 3.6 which is on the release machines, while the test machines have Python 3.7

@targos I think you pulled 52835 into the release proposal for the next 22 so FYI you might have problems building the release.

[richardlau]

@richardlau mentioned that the test machines have a newer version of Python due to some other issue, so that PR might have broken compatibility with Python 3.6 which is on the release machines, while the test machines have Python 3.7

The test machines run tap2junit (the release machines do not) and that IIRC dropped support for Python 3.6 and required later versions of Python 3.

[richardlau]

All rhel8 release machines are failing:
e.g. https://ci-release.nodejs.org/job/iojs+release/10200/nodes=rhel8-x64-release/console

07:00:23 python3 ./configure \
07:00:23 	--prefix=/ \
07:00:23 	--dest-cpu=x64 \
07:00:23 	--tag=nightly202405131d7d094a98 \
07:00:23 	--release-urlbase=https://nodejs.org/download/nightly/ \
07:00:23 	--download-path=/home/iojs/node-icu/ --verbose --download=all --with-intl=full-icu
07:00:23 Node.js configure: Found Python 3.6.8...
07:00:23 Traceback (most recent call last):
07:00:23   File "./configure", line 29, in <module>
07:00:23     import configure
07:00:23   File "/home/iojs/build/ws/configure.py", line 30, in <module>
07:00:23     from gyp.common import GetFlavor
07:00:23   File "tools/gyp/pylib/gyp/__init__.py", line 9, in <module>
07:00:23     import gyp.input
07:00:23   File "tools/gyp/pylib/gyp/input.py", line 8, in <module>
07:00:23     import gyp.common
07:00:23   File "tools/gyp/pylib/gyp/common.py", line 435
07:00:23     if CC := os.environ.get("CC_target") or os.environ.get("CC"):
07:00:23            ^
07:00:23 SyntaxError: invalid syntax
07:00:23 make: *** [Makefile:1272: node-v23.0.0-nightly202405131d7d094a98-linux-x64.tar] Error 1

Since the test machines have newer Python, maybe rerunning Ansible against the release machines would update to/add a later Python 3 version? I could try that tomorrow when I'm back on my work computer (if nobody else with access to the release machines beats me to it 🙂).

[targos]

Running Ansible against release-osuosl-rhel8-ppc64_le-1 changed nothing:

PLAY RECAP *******************************************************************************************************************************************
release-osuosl-rhel8-ppc64_le-1 : ok=47   changed=0    unreachable=0    failed=0    skipped=117  rescued=0    ignored=0

[targos]

I guess it makes sense. There's nothing in the Ansible config to install python39 on these hosts.

[targos]

@richardlau From you in #3163:

I've deployed this onto all of the RHEL 8 ppc64le, s390x and x64 test machines. (We don't currently run the V8 CI on the RHEL8 x64 machines because the builds on those take a lot longer than on the Nearform-hosted Intel machine(s).)

[targos]

#3719

[targos]

Test runs:
https://ci-release.nodejs.org/job/iojs+release/10202/
https://ci-release.nodejs.org/job/iojs+release/10204/

[richardlau]

Test runs: https://ci-release.nodejs.org/job/iojs+release/10202/ https://ci-release.nodejs.org/job/iojs+release/10204/

Looks like 10202 picked up Python 3.6 but 10204 picked up Python 3.9.

[targos]

Yes. Only https://ci-release.nodejs.org/job/iojs+release/10202/nodes=rhel8-arm64-release/ is relevant on 10202

[richardlau]

Thanks for looking into this.