Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

checkout@v3 using node20 instead of node16 #1809

Open
jcooke-impinj opened this issue Jul 3, 2024 · 43 comments
Open

checkout@v3 using node20 instead of node16 #1809

jcooke-impinj opened this issue Jul 3, 2024 · 43 comments

Comments

@jcooke-impinj
Copy link

Running on rhel7 with self-hosted runners and starting today we're running into issues where @V3 is using node20 instead of node16.
Happening on runner versions 2.316.1 and 2.317.0.
Oddly some of our Repos don't run into this issue when executing checkout@v3 on the same runners.

Run actions/checkout@v3
/ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libm.so.6: version GLIBC_2.27' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: version GLIBCXX_3.4.20' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node)
/ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: version CXXABI_1.3.9' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node)
/ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version GLIBC_2.28' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node) /ci_storage_agent_1/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version GLIBC_2.25' not found (required by /ci_storage_agent_1/actions-runner/externals/node20/bin/node)

@jcooke-impinj
Copy link
Author

Our team found that setting these before starting the runners get's things going again. Still unsure what's causing this.

export ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16
export ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION=node16

@adpaco-aws
Copy link

This doesn't seem to be specific to actions/checkout@v3 since we tried to upgrade to actions/checkout@v4 (here) and got the same failure (here).

@biswalc
Copy link

biswalc commented Jul 4, 2024

I am having the same issue while running below versions, on ubuntu 20.04 VM:

actions/[email protected]
actions/checkout@v3
actions/checkout@v4
actions/[email protected]

Error:

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

Issue started today, i.e. 3 Jul 10am PDT.

@junghyun-apr
Copy link

Hello! Our development team is also experiencing the same issue. We are using AWS instances as self-hosted.
We temporarily resolved the issue by simply changing the node16 directory to node20 in the same path. 🥲

AWS AMI :

amzn2-ami-kernel-5.10-hvm-2.0.20240223.0-x86_64-gp2

Git Action Error :

/home/ec2-user/actions-runner/externals/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /home/ec2-user/actions-runner/externals/node20/bin/node)
/home/ec2-user/actions-runner/externals/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /home/ec2-user/actions-runner/externals/node20/bin/node)

@GuyAv46
Copy link

GuyAv46 commented Jul 4, 2024

Following this blog post: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
Adding this to the envs that require node16 fixed the issue

ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true

@Flamefire
Copy link

Flamefire commented Jul 4, 2024

Our team found that setting these before starting the runners get's things going again. Still unsure what's causing this.

export ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16
export ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION=node16

I tried setting this inside a workflow running on GH hosted runners and it didn't work.
So it looks like this only works when the runner itself was started with those set, i.e. you cannot set them for a running runner which includes the GH hosted ones.

This doesn't seem to be specific to actions/checkout@v3 since we tried to upgrade to actions/checkout@v4 (here) and got the same failure (here).

v4 always used Node 20

I am having the same issue while running below versions, on ubuntu 20.04 VM:

actions/[email protected]
actions/checkout@v3
actions/checkout@v4
actions/[email protected]

See above

@david-batranu
Copy link

Fixed by renaming the node20 folder and symlinking node16 to node20:

cd $RUNNER_HOME/actions-runner/externals
mv node20 node20.bak
ln -s node16 node20

@eshogo
Copy link

eshogo commented Jul 4, 2024

I tried modifying /home/runner/actions-runner/.env and env.sh and it works fine.

% cat .env
ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION=node16
ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION=node16

and adding their environment variables into 'varCheckList' array in env.sh.
(not needed?)

varCheckList=(
    'LANG'
    'JAVA_HOME'
    'ANT_HOME'
    'M2_HOME'
    'ANDROID_HOME'
    'ANDROID_SDK_ROOT'
    'GRADLE_HOME'
    'NVM_BIN'
    'NVM_PATH'
    'LD_LIBRARY_PATH'
    'PERL5LIB'
    'ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION'
    'ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION'
    )

Then restart runner service

% systemctl restart github-actions-runner.service

@david-batranu
Copy link

@n1ngu
Copy link

n1ngu commented Jul 4, 2024

Following this blog post: https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/ Adding this to the envs that require node16 fixed the issue

ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true

Quoting the post paragraph about that workaround: "This will only work until we upgrade the runner removing Node16 later in the spring"

We are walking a tightrope.

Just wanted to raise some awareness about that. Plan to actually upgrade your github actions in the short term or move away from them. E.g. I think I'll quit my job. 😄

@biswalc
Copy link

biswalc commented Jul 4, 2024

I am having the same issue while running below versions, on ubuntu 20.04 VM:

actions/[email protected]
actions/checkout@v3
actions/checkout@v4
actions/[email protected]

Error:

/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

Issue started today, i.e. 3 Jul 10am PDT.

My glibc version:

ldd --version
ldd (Ubuntu GLIBC 2.31-0ubuntu9.16) 2.31
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

So why do I get this error? I am using [email protected]. I do not understand what I have to do to upgrade.

@ayizhi
Copy link

ayizhi commented Jul 5, 2024

Here's a critical issue: Firstly, you shouldn't upgrade such fundamental environments arbitrarily. Even if you do upgrade, you must ensure the environment is correct. Upgrading to Node 20 and causing a glib error is absolutely unacceptable, especially in a product from a company like Microsoft.

@FelixSchwarz
Copy link

FelixSchwarz commented Jul 5, 2024

I wish there was an alternative version of the checkout action, maybe written in rust so we could just deploy a static binary. Then we would just compile that against a really old version of glibc. This would also simplify running GitHub Actions with nektos/act which does not map node into the container.

Update: I found checkout-action in GitHub's market place and added support for private repos. Not sure if that is a permanent solution but at least our CI jobs are passing again...

@hasufell
Copy link

hasufell commented Jul 5, 2024

This means you now can't run any docker image that has a glibc older than 2.27. That includes:

  • debian:9
  • fedora:27
  • ubuntu:18.04

What a huge f*ckup. I'm baffled by the amount of incompetence. What's wrong with providing a statically linked node binary? Do you need help with redistribution github?

hasufell added a commit to stable-haskell/haskell-language-server that referenced this issue Jul 5, 2024
hasufell added a commit to haskell/ghcup-metadata that referenced this issue Jul 5, 2024
@maksadbek
Copy link

maksadbek commented Nov 10, 2024

I'm really confused trying to understand how these workflows work:

# This workflow will build a golang project
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-go

name: Go

on:
  push:
    branches: [ "main" ]
  pull_request:
    branches: [ "main" ]

jobs:

  build:
    runs-on: ubuntu-20.04
    container: ubuntu:16.04
    steps:
    - uses: actions/checkout@v2

    - name: Set up Go
      uses: actions/setup-go@v4
      with:
        go-version: '1.20'

    - name: Build
      run: go build -v ./...

    - name: Test
      run: go test -v ./...

it's failing with the glibc not found error:

Run actions/checkout@v2
/usr/bin/docker exec  d5bc5bba7bd6d0c36d05f01be4cc266534953c3a2647d4acc3e553c2fa07eacc sh -c "cat /etc/*release | grep ^ID"
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)

but the action.yml of V2 versions states it must use node12: https://github.com/actions/checkout/blob/v2/action.yml#L75

I can't believe that Github Actions has changed ALL versions of actions/checkout to use node20

fendor added a commit to haskell/haskell-language-server that referenced this issue Nov 11, 2024
Some release jobs have to be executed on a custom runner due to GitHub forcing a
newer nodejs version: actions/checkout#1809

This whole issue is a huge rabbit hole where you can get lost.
In short:

* The actions/checkout@4 versions require the nodejs 20 version
* Prebuilt nodejs 20 are linking against a newer GLIBC version
* Even though jobs are executed in a docker image, some commands are from
    the runner machine, (e.g. nodejs is from the ubuntu-latest runner).
* Downgrading to actions/checkout@3 worked for some time.

However, due to https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
it seems like we *have* to do something.
Thus, we are running jobs for older distro images on a customer runner, provided by maerwald.
In particular, when writing this message, the distro images that have be run on the custom runner are:

* debian:9
* ubuntu:18.04
* mint:19.3
* fedora:27
* centos:7
fendor added a commit to fendor/haskell-language-server that referenced this issue Nov 11, 2024
Some release jobs have to be executed on a custom runner due to GitHub forcing a
newer nodejs version: actions/checkout#1809

This whole issue is a huge rabbit hole where you can get lost.
In short:

* The actions/checkout@4 versions require the nodejs 20 version
* Prebuilt nodejs 20 are linking against a newer GLIBC version
* Even though jobs are executed in a docker image, some commands are from
    the runner machine, (e.g. nodejs is from the ubuntu-latest runner).
* Downgrading to actions/checkout@3 worked for some time.

However, due to https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
it seems like we *have* to do something.
Thus, we are running jobs for older distro images on a customer runner, provided by maerwald.
In particular, when writing this message, the distro images that have be run on the custom runner are:

* debian:9
* ubuntu:18.04
* mint:19.3
* fedora:27
* centos:7
derevnjuk pushed a commit to NeuraLegion/bright-cli that referenced this issue Nov 11, 2024
e2e pipeline fails on Ubuntu 16.04 and 18.04 due to the recent default
switch to node20 in actions/checkout. Node20 requires GLIBC 2.28 or
higher, which these environments lack. To resolve this, we’re setting
the `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true` environment variable
to allow fallback to a compatible Node version, restoring pipeline
stability.

Useful links:
* [GitHub changelog: node20
default](https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/)
* [GitHub issue #1809](actions/checkout#1809)
cburroughs added a commit to pantsbuild/pants that referenced this issue Nov 16, 2024
While trying to land #21528 to was observed that the wheel building jobs
were failing during git checkout with wonderful errors like
`/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found
(required by /__e/node20/bin/node)`. This appears to be the same
symptoms as actions/checkout#1809 which
pointed the the deprecation notice at
<https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>.
If this all sounds familiar that is because we *mostly* cleaned this up
in #21133, but kept a fewer uses of the older actions for
`manylinux2014` compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still
available in the runner, you can choose to set
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your
runner machine. This will *only work until we upgrade the runner
removing Node16 later in the spring*. (emphasis added)"

From the wheel job failures during #21528 and my attempts to fiddle with
all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`,
`ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`,
`ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time
promised for the Spring has finally come, but I'm not that familiar with
GitHub Actions and could be missing something.

As a consequence the version of both `actions/upload-artifacts` and
`actions/checkout` is bumped to `v4`. To make this testable now and in
the future, wheels are now build on ci config changes.

Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big
deal? The "2.28" refers to glibc
[2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html)
in 2018. That is older than Debian stable and if you need to use
software that old you are presumably paying someone for an enterprise
distribution.

NOTE: I think we should bump the manylinux version in `main` regardless,
but if my read if the situation is correct we may need to backport this
any active release line. Without this or a more complex change we would
also be unable to release after December 5th per #21616.

ref #21195 #21616
benjyw pushed a commit to benjyw/pants that referenced this issue Nov 18, 2024
While trying to land pantsbuild#21528 to was observed that the wheel building jobs
were failing during git checkout with wonderful errors like
`/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found
(required by /__e/node20/bin/node)`. This appears to be the same
symptoms as actions/checkout#1809 which
pointed the the deprecation notice at
<https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>.
If this all sounds familiar that is because we *mostly* cleaned this up
in pantsbuild#21133, but kept a fewer uses of the older actions for
`manylinux2014` compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still
available in the runner, you can choose to set
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your
runner machine. This will *only work until we upgrade the runner
removing Node16 later in the spring*. (emphasis added)"

From the wheel job failures during pantsbuild#21528 and my attempts to fiddle with
all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`,
`ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`,
`ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time
promised for the Spring has finally come, but I'm not that familiar with
GitHub Actions and could be missing something.

As a consequence the version of both `actions/upload-artifacts` and
`actions/checkout` is bumped to `v4`. To make this testable now and in
the future, wheels are now build on ci config changes.

Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big
deal? The "2.28" refers to glibc
[2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html)
in 2018. That is older than Debian stable and if you need to use
software that old you are presumably paying someone for an enterprise
distribution.

NOTE: I think we should bump the manylinux version in `main` regardless,
but if my read if the situation is correct we may need to backport this
any active release line. Without this or a more complex change we would
also be unable to release after December 5th per pantsbuild#21616.

ref pantsbuild#21195 pantsbuild#21616
benjyw added a commit to pantsbuild/pants that referenced this issue Nov 19, 2024
While trying to land #21528 to was observed that the wheel building jobs
were failing during git checkout with wonderful errors like
``/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found
(required by /__e/node20/bin/node)``. This appears to be the same
symptoms as actions/checkout#1809 which
pointed the the deprecation notice at

<https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>.
If this all sounds familiar that is because we *mostly* cleaned this up
in #21133, but kept a fewer uses of the older actions for
`manylinux2014` compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still
available in the runner, you can choose to set
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your
runner machine. This will *only work until we upgrade the runner
removing Node16 later in the spring*. (emphasis added)"

From the wheel job failures during #21528 and my attempts to fiddle with
all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`,
`ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`,
`ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time
promised for the Spring has finally come, but I'm not that familiar with
GitHub Actions and could be missing something.

As a consequence the version of both `actions/upload-artifacts` and
`actions/checkout` is bumped to `v4`. To make this testable now and in
the future, wheels are now build on ci config changes.

Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big
deal? The "2.28" refers to glibc

[2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html)
in 2018. That is older than Debian stable and if you need to use
software that old you are presumably paying someone for an enterprise
distribution.

NOTE: I think we should bump the manylinux version in `main` regardless,
but if my read if the situation is correct we may need to backport this
any active release line. Without this or a more complex change we would
also be unable to release after December 5th per #21616.

ref #21195 #21616

Co-authored-by: cburroughs <[email protected]>
@yourcelf
Copy link

It appears Github has finally fulfilled their promise of removing the older node versions, and now all workflows will run with node20. The ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true workaround no longer works.

For our use case, we still needed to be able to run some CI jobs with node14, but the node 14 docker images' glibc is too old now. The thing that worked for us was to instead install node14 with nvm onto a debian 12 image for these jobs. The debian image has a new-enough glibc for the version of node20 that github loads, and nvm is happy to install ancient nodes onto that.

jobs:
  foo:
    container: debian:12
    steps:
      - name: Run on old node
        run: |
          apt update && apt -y install wget
          touch "$HOME/.bashrc"
          wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash
          export NVM_DIR="$HOME/.nvm"
          [ -s "$NVM_DIR/nvm.sh" ] && . "$NVM_DIR/nvm.sh"
          nvm install 14.21.3
          # node 14 is now available.
          ...

@marissareyes80
Copy link

sitic added a commit to sitic/monochrome that referenced this issue Nov 27, 2024
fendor added a commit to haskell/haskell-language-server that referenced this issue Dec 4, 2024
Some release jobs have to be executed on a custom runner due to GitHub forcing a
newer nodejs version: actions/checkout#1809

This whole issue is a huge rabbit hole where you can get lost.
In short:

* The actions/checkout@4 versions require the nodejs 20 version
* Prebuilt nodejs 20 are linking against a newer GLIBC version
* Even though jobs are executed in a docker image, some commands are from
    the runner machine, (e.g. nodejs is from the ubuntu-latest runner).
* Downgrading to actions/checkout@3 worked for some time.

However, due to https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/
it seems like we *have* to do something.
Thus, we are running jobs for older distro images on a customer runner, provided by maerwald.
In particular, when writing this message, the distro images that have be run on the custom runner are:

* debian:9
* ubuntu:18.04
* mint:19.3
* fedora:27
* centos:7
k-okada added a commit to k-okada/jsk_control that referenced this issue Dec 6, 2024
@minhanghuang
Copy link

Same problem with hadolint, any solution?
image
image

Run actions/checkout@v3
/usr/bin/docker exec  983eeb54e6a01eb7215e2ef5a272aeb78e1aa1fb0adf67f[16](https://github.com/minhanghuang/CyberRT/actions/runs/12247318945/job/34164927434#step:3:17)b5837a57127829a sh -c "cat /etc/*release | grep ^ID"
/__e/node20/bin/node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)

thesamesam added a commit to thesamesam/meson that referenced this issue Dec 10, 2024
Do the same as pytorch/test-infra#5959 and download
an old nodejs to keep Ubuntu Bionic working.

Bug: actions/checkout#1809
Bug: actions/runner#3373
Castaglia added a commit to Castaglia/proftpd-mod_proxy that referenced this issue Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests