Composite Actions using Node 20 don't work with CentOS 7

[ajschmidt8]

Problem

As mentioned in the GitHub blog post below, Actions is currently transitioning from Node 16 to Node 20.

• https://github.blog/changelog/2023-09-22-github-actions-transitioning-from-node-16-to-node-20/

However, Node 20 seems to be incompatible with CentOS 7, which is listed as a compatible OS for self-hosted runners below:

• https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#linux

This incompatibility seems to stem from the fact that NodeJS version 20 supports glibc >=2.28 (as seen in the link below), but CentOS 7 only has glibc==2.17.

• https://github.com/nodejs/node/blob/v20.0.0/BUILDING.md#platform-list

The consequence of this is that composite actions that have been updated to use Node 20 no longer work with CentOS 7 host machines or containers.

Here is an example of the error message that results from trying to run a composite action using Node 20 on a CentOS 7 container:

• https://github.com/rapidsai/dask-cuda/actions/runs/6402970861/job/17380789311?pr=1255#step:19:8

As a result of this incompatibility, we've had to intentionally downgrade our composite actions to a version that uses Node 16 so that our workflows continue to work.

To Reproduce

Run the following workflow to exhibit the error message

name: centos7

on:
  push:

jobs:
  run:
    runs-on: ubuntu-latest
    container: centos:7
    steps:
      - uses: actions/checkout@v4
      - run: echo "hello, world"

Expected behavior

Since the documentation below states that self-hosted runners support CentOS 7, I would expect the above workflow to run to completion without any errors

• https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners#linux

Runner Version and Platform

Ubuntu 22.04, runner version 2.309.0

[firewave]

This also affects

ubuntu:16:04 (https://github.com/danmar/cppcheck/actions/runs/7677304162/job/20926809147?pr=5917)

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)

ubuntu:18.04 (https://github.com/danmar/cppcheck/actions/runs/7677304162/job/20926809224?pr=5917)

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

[TomasNovo]

hey @ajschmidt8 ,
I'm currently facing this problem. I've tried several ways to use Node 16 but seems that the action always calls Node 20

/lib64/libm.so.6: version 'GLIBC_2.27' not found (required by /opt/actions-runner/externals/node20/bin/node)

how did you effectively managed to downgrade your composite actions to a version that uses Node 16 so that the workflows continue to work?

[troyxmccall]

@TomasNovo you'll need to audit all of your workflows and look up Node v16 compatible versions

ie:

FranzDiebold/github-env-vars-action@v2

would need to be rolled back to a specific version of

FranzDiebold/[email protected]

per https://github.com/FranzDiebold/github-env-vars-action/releases/tag/v2.7.0

[luciansmith]

I feel like this isn't getting the attention it needs: anyone using github actions to create manylinux2014 builds will no longer be able to build on that platform at all, if the proposed Node16->Node20 change becomes required. Is there any proposed mitigation of this? Am I incorrect in my assessment?

[alex]

Your assessment is correct. As a result, this change will have substantial negative impact on the Python community.

#3128 sketches out a direction Github could take to resolve this, but there has been no response thus far.

[dixyes]

For those who are struggling here,

I have a dirty way to make node 20 work with centos:7

1. mount a dir with jobs.<name>.container.volumes, one ro on /__e/node20, another rw on /node20217
2. download and extract unofficial node-v20.9.0-linux-x64-glibc-217.tar.xz into /node20217
3. all workflow works

NOTE: since we cannot cache that tar before node is set-up, this is abusing nodejs distribution, so you should make your own docker image that contains that tar (rather than download it every build)

https://github.com/dixyes/ghactionsplay/blob/main/.github/workflows/glibc217node20.yml

https://github.com/dixyes/ghactionsplay/actions/runs/9075609934/job/24936557854

[Y--]

We were able to keep using Node 16 with:

   env:
      ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
      ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

(from this article )

But that is likely to break very soon indeed. We need a longer term solution.

[sethrj]

@Y-- the ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16 seems not to be needed: at least, not for the case where our failure was due to running on a CentOS7 image.

[danielefranceschi]

We handled this notorious issue using the appropriate node16-compatible actions versions in centos7 containers (for instance, actions/checkout@v3 instead of @v4).

With a curious coincidence with the Centos7 EOL, everything WAS fine until some days ago, but now node20 is used anyway even if action.yml specifies runs.using=node16.

FTR: self hosted runners 2.317.0.

Solved putting the suggested environment variable in the job:

generate-RH7:
  runs-on: [self-hosted, linux, docker]
  env:
    ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: 'true'
  container:
    image: ghcr.io/xxxxx/yyyy:5-centos7

[solardiz]

FWIW, in lkrg-org/lkrg#344 we've just dealt with both CentOS 7 EOL and then this issue with:

+++ b/.github/workflows/builds.yml
@@ -4,12 +4,17 @@ on: [push, pull_request]
 jobs:
     build:
         runs-on: ubuntu-22.04
+        env:
+            ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
+            ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
         strategy:
             fail-fast: false
             matrix:
                 include:
                     - image: centos:7
-                      run: yum install -y kernel-devel kernel gcc make elfutils-libelf-devel
+                      run: |
+                          sed -i 's,^mirrorlist=,#,; s,^#baseurl=http://mirror\.centos\.org/centos/$releasever,baseurl=https://vault.centos.org/7.9.2009,' /etc/yum.repos.d/CentOS-Base.repo
+                          yum install -y kernel-devel kernel gcc make elfutils-libelf-devel
                     - image: almalinux:8
                       run: yum install -y kernel-devel kernel gcc make elfutils-libelf-devel
                     - image: rockylinux:8

This is with actions/checkout@v3. I hope (and ask that) this remains supported for a few years more, since there are now Enterprise Linux 7 prolonged support distros so we are not dropping our EL7 support, which we continue to test on CentOS 7.

[klepiz]

For those who are struggling here,

I have a dirty way to make node 20 work with centos:7

1. mount a dir with jobs.<name>.container.volumes, one ro on /__e/node20, another rw on /node20217
2. download and extract unofficial node-v20.9.0-linux-x64-glibc-217.tar.xz into /node20217
3. all workflow works

NOTE: since we cannot cache that tar before node is set-up, this is abusing nodejs distribution, so you should make your own docker image that contains that tar (rather than download it every build)

https://github.com/dixyes/ghactionsplay/blob/main/.github/workflows/glibc217node20.yml

https://github.com/dixyes/ghactionsplay/actions/runs/9075609934/job/24936557854

Hello! this workaround worked for me for GH runner version 2.317 and 2.318 on a C7, but for 2.319 this is not working anymore and I am getting the issue

externals/node20/bin/node --version
externals/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by externals/node20/bin/node)
externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by externals/node20/bin/node) 

Is there another way to make the latest GH runner version work on C7 + node20?

[olsen232]

+1 to fixing this issue
I'm using the Python manylinux2014 image but that still has the same issue. AFAICT there is no newer manylinux image than that, ie, Python best practice is still to build wheels on a linux image from 2014 that has an old GLIBC, but the builtin GH actions - due to using dynamically linked up-to-date nodejs - don't work with an old GLIBC. This is a pain and will surely only get worse as support is dropped for older node versions - this has already broken our build more than once as various actions we depend on switch to newer nodes, and each time I have to figure out how to pin whatever has gone wrong, which will only work as long as support is maintained for that old version.

I think using a statically linked node that just works for the built-in actions would be the way to go - as demonstrated here: #3128
Or - allow people to provide their own node that works for the container. I made sure that node was installed and working before using actions/checkout - but actions/checkout doesn't care, it installs a broken version of node anyway and uses that.

Lots of people have hit this already:
#337
#801
#2115
#2906
#3373