Use devtoolset-8 on Centos/RHEL for Node.js 14+ #2242

richardlau · 2020-03-22T23:41:50Z

Requires access to the release machines. Install devtoolset-8 on the Centos/RHEL machines and use it to build releases for Node.js 14+.

Refs: #2168

Install devtoolset-8 on centos7:

release-osuosl-centos7-ppc64_le-1
test-osuosl-centos7-ppc64_le-1
test-osuosl-centos7-ppc64_le-2
release-digitalocean-centos7-x64-1 :
release-packetnet-centos7-arm64-1:
test-packetnet-centos7-arm64-1
test-packetnet-centos7-arm64-2
test-rackspace-centos7-x64-1
test-softlayer-centos7-x64-1:

Install on rhel7

Use devtoolset-8 for 14.x: PR #2262

select-compiler.sh:

rhel7-s390x
centos7-ppc64_le
centos7-64
centos7-arm64

VersionSelectorScript.goovy

n/a rhel7-s390x
n/a centos7-ppc64_le: ubuntu1404 already excluded for gte 14.x
centos7-64
centos7-arm64

Jenkins config changes:

test builds:

node-test-commit-linux: (x64)
- 12.x: https://ci.nodejs.org/job/node-test-commit-linux/34154/
- master: https://ci.nodejs.org/job/node-test-commit-linux/34153/
node-test-commit-arm: (arm64)
- 12.x: https://ci.nodejs.org/job/node-test-commit-arm/30641/: triggered only gcc6
- master: https://ci.nodejs.org/job/node-test-commit-arm/30645/
iojs+release: (disttype = "test")
- 12.x: https://ci-release.nodejs.org/job/iojs+release/5850/
- master: https://ci-release.nodejs.org/job/iojs+release/5882, looks good
rhel7-s390x:
- master: https://ci.nodejs.org/job/node-test-commit-linuxone/20203/ (build in progress, but gcc8 selected/configured)
centos7-ppcle:
- master: https://ci.nodejs.org/job/node-test-commit-plinux/32155/nodes=centos7-ppcle

rvagg · 2020-03-23T02:16:37Z

do we have ansible changes ready for this to run?

sam-github · 2020-03-23T19:03:30Z

devtoolset-6, the rpms weren't publically available, but I don't think we've looked yet into the situation for devtoolset-8

sam-github · 2020-04-02T22:40:30Z

Some questions about this:

Only release machines? So select-compiler.sh will do something different depending on test vs release?
actual machines to be installed are all of following, or only some of them?
- release-softlayer-centos6-x64-1
- release-osuosl-centos7-ppc64_le-1
- release-packetnet-centos7-arm64-1
- release-ibm-rhel7-s390x-1

sam-github · 2020-04-02T22:47:07Z

I need #2220 to even look into non-IBM machines (for IBM hosted platforms, I have special access from #1721)

sam-github · 2020-04-02T22:52:19Z

Can't get it from (some of) the default repos yet, will have to look further afield:

test-softlayer-centos7-x64-1: devtoolset-8 is in yum repos 🎉
release-osuosl-centos7-ppc64_le-1: devtoolset-8 not available, and the devtoolset-6 repos have disappeared!

https://cbs.centos.org/repos/sclo7-devtoolset-6-rh-release/ppc64le/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

release-ibm-rhel7-s390x-1: devtoolset-8 not available

richardlau · 2020-04-02T23:03:13Z

* release-osuosl-centos7-ppc64_le-1: devtoolset-8 not available, and the devtoolset-6 repos have disappeared!
https://cbs.centos.org/repos/sclo7-devtoolset-6-rh-release/ppc64le/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

Maybe https://cbs.centos.org/koji/buildinfo?buildID=26229?

sam-github · 2020-04-03T00:36:53Z

Previous link pointed to an entire repo, which included all the rpms, I think the single rpm alone is not sufficient -- but I'll try tomorrow and see how it goes.

rvagg · 2020-04-03T01:18:24Z

Only release machines?

I think we could take the simple approach do this globally, for everything on CentOS. We'll presumably have non-CentOS machines running other versions of GCC. It means we'll have a gap in testing (old GCC on CentOS) but I don't think that's a bad compromise considering the complexity it'll induce if we try and be more exhaustive.

sam-github · 2020-04-03T20:04:01Z

Installed on release-osuosl-centos7-ppc64_le-1

[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ ls /opt/rh/
devtoolset-6  devtoolset-8
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ ls /opt/rh/devtoolset-8
enable  root
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ . /opt/rh/devtoolset-8/enable 
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ which gcc
/opt/rh/devtoolset-8/root/usr/bin/gcc
[centos@release-osuosl-centos7-ppc64--le-1 tmp]$ gcc --version
gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)
Copyright (C) 2018 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

sam-github · 2020-04-03T21:39:39Z

gpg keys were bad for test-softlayer-centos7-x64-1, fixed with advice from https://jc-lan.org/2019/12/14/error-when-updating-centos-7-gpg-keys-are-not-correct-for-this-package/

Now it fails with "At least 32MB more space needed on the /boot filesystem."

sam-github · 2020-04-03T22:22:32Z

test-rackspace-centos7-x64-1 doesn't accept git as a package name, once I pushed a fix for that onto #2262 , it installed devtoolset-8

sam-github · 2020-04-03T22:46:57Z

Looks like changes that install devtoolset-6-libatomic-devel don't work on test-packetnet-centos7-arm64-2. Its not clear why, its not a clear package-not-found, or, perhaps it is? Its trying to get it from http://centosn7.centos.org/altarch/7.7.1908/sclo/aarch64/rh/devtoolset-6, which doesn't exist.

But, that doesn't seem to be an issue, we seem to have been doing fine without it, so I moved it out of the centos7 common in #2262

FTR:


[root@test-packetnet-centos7-arm64-2 yum.repos.d]# yum install devtoolset-6-libatomic-devel    
Loaded plugins: fastestmirror                                                                  
Loading mirror speeds from cached hostfile                                                     
 * base: packet04.centos.org                                                                   
 * centos-sclo-rh: centosn7.centos.org                                                         
 * epel: ewr.edge.kernel.org
 * extras: centosm7.centos.org
 * updates: centosg7.centos.org
Resolving Dependencies                         
--> Running transaction check
---> Package devtoolset-6-libatomic-devel.aarch64 0:6.3.1-3.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved                          

===============================================================================================
 Package                           Arch         Version              Repository           Size
===============================================================================================
Installing:                                    
 devtoolset-6-libatomic-devel      aarch64      6.3.1-3.1.el7        centos-sclo-rh       15 k

Transaction Summary                            
===============================================================================================
Install  1 Package                             

Total download size: 15 k
Installed size: 110 k                          
Is this ok [y/d/N]: y                          
Downloading packages:                          
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
devtoolset-6-libatomic-devel-6 FAILED                                           
http://centosn7.centos.org/altarch/7.7.1908/sclo/aarch64/rh/devtoolset-6/devtoolset-6-libatomic-devel-6.3.1-3.1.el7.aarch64.rpm: [Errno 14] HTTP Error 404 - Not Found
Trying other mirror. 
... a dozen other failing mirrors

EDIT: after the fix, installed devtoolset-8 OK.

sam-github · 2020-04-03T22:51:38Z

@rvagg I'm having trouble accessing an arm release machine, could you check the release key is authorized for it? I see:

core/secrets (master % u=) % ssh release-packetnet-centos7-arm64-1 ls /opt/rh
[email protected]: Permission denied (publickey).
core/secrets (master % u=) % grep -A 5 release-packetnet-centos7-arm64-1 ~/.ssh/config 
Host release-packetnet-centos7-arm64-1 
  HostName 147.75.104.218
  IdentityFile ~/.ssh/nodejs_build_release
  User root

EDIT: same on release-digitalocean-centos7-x64-1

sam-github · 2020-04-03T23:00:45Z

release-packetnet-centos7-arm64-1 has two issues:

Its jenkins secret is not in the inventory
I can't ssh in

task path: /home/sam/w/core/build/ansible/playbooks/jenkins/worker/create.yml:28
fatal: [release-packetnet-centos7-arm64-1]: FAILED! => {
    "msg": "The conditional check 'not secret' failed. The error was: error while evaluating conditional (not secret): 'secret' is undefined"
}

PLAY RECAP ************************************************************************************************************************************************************************************
release-packetnet-centos7-arm64-1 : ok=0    changed=0    unreachable=0    failed=1   

build/ansible (centos7-devtoolset-8 $ u=) % ssh release-packetnet-centos7-arm64-1
[email protected]: Permission denied (publickey).

rvagg · 2020-04-06T05:29:44Z

@sam-github can you try with an explicit -i ~/.ssh/nodejs_build_release on that ssh? I'm getting in fine and the release pubkey is in .ssh/authorized_keys. You'll have to find the jenkins node secret from jenkins itself or on the machine itself to put it in secrets repo. When inventory.yml was originally set up with secrets, many of them weren't added in (because it was tedious) so we're all responsible for adding them in as we find them missing.

sam-github · 2020-04-06T18:17:41Z

Thanks for looking. -i didn't work, but it was a problem with my local release key, I guess. I did

dotgpg cat ../secrets/build/release/nodejs_build_release.pub > ~/.ssh/nodejs_build_release.pub
dotgpg cat ../secrets/build/release/nodejs_build_release > ~/.ssh/nodejs_build_release

And now I have access. Too bad I did both at once, because I don't know what was wrong. I've no idea why the write-ssh-config playbook didn't write the most current, or why I could ssh into so many other systems, but not those ones. 🤷

I'm updating secrets as I go, recovering them from machines.

sam-github · 2020-04-06T19:33:26Z

centos6 is excluded for release & test from 12.x and later, removed from checklists

sam-github · 2020-04-07T22:51:33Z

first attempt at an x86 gcc8 build: https://ci.nodejs.org/job/node-test-commit-linux-sam-github/1/nodes=centos7-64-gcc8

sam-github · 2020-04-07T23:07:35Z

first s390x build: https://ci.nodejs.org/job/node-test-commit-linuxone-sam-github/2/nodes=rhel7-s390x (well, second, first didn't have the select-compiler update)

sam-github · 2020-04-07T23:14:13Z

first ppc build: https://ci.nodejs.org/job/node-test-commit-plinux-sam-github/1/nodes=centos7-ppcle/

rvagg · 2020-04-08T01:46:38Z

I think maybe we need a gcc -v; g++ -v before make in these builds. I can't tell for sure what version it's invoking after the devtoolset setup and ccache environment variable prepend. It looks correct but I'd have to grab a binary to be sure.

richardlau · 2020-04-08T09:35:31Z

I think maybe we need a gcc -v; g++ -v before make in these builds. I can't tell for sure what version it's invoking after the devtoolset setup and ccache environment variable prepend. It looks correct but I'd have to grab a binary to be sure.

I could change the configure script to log the versions when run with --verbose (which I think we do on the CI). The script already parses the compiler versions to check and warn if the detected compiler is not new enough.

Edit: nodejs/node#32715

richardlau · 2020-04-08T11:29:14Z

With nodejs/node#32715:

first ppc build: https://ci.nodejs.org/job/node-test-commit-plinux-sam-github/1/nodes=centos7-ppcle/

https://ci.nodejs.org/job/node-test-commit-plinux-sam-github/2/nodes=centos7-ppcle/

12:20:54 Detected C++ compiler (CXX=ccache ppc64le-redhat-linux-g++) version: 8.3.1
12:20:54 Detected C compiler (CC=ccache ppc64le-redhat-linux-gcc) version: 8.3.1

first s390x build: https://ci.nodejs.org/job/node-test-commit-linuxone-sam-github/2/nodes=rhel7-s390x (well, second, first didn't have the select-compiler update)

https://ci.nodejs.org/job/node-test-commit-linuxone-sam-github/3/nodes=rhel7-s390x/

12:25:04 Detected C++ compiler (CXX=ccache s390x-redhat-linux-g++) version: 8.3.1
12:25:04 Detected C compiler (CC=ccache s390x-redhat-linux-gcc) version: 8.3.1

first attempt at an x86 gcc8 build: https://ci.nodejs.org/job/node-test-commit-linux-sam-github/1/nodes=centos7-64-gcc8

https://ci.nodejs.org/job/node-test-commit-linux-sam-github/2/nodes=centos7-64-gcc8/

12:26:11 Detected C++ compiler (CXX=ccache g++) version: 8.2.1
12:26:11 Detected C compiler (CC=ccache gcc) version: 8.2.1

rvagg · 2020-04-08T11:49:21Z

sweet, thanks @richardlau, lgtm then.

sam-github · 2020-04-08T20:35:33Z

v12.x: node-test-commit-linux, all platforms: https://ci.nodejs.org/job/node-test-commit-linux-sam-github/3/

sam-github · 2020-04-08T20:56:40Z

@rvagg https://ci.nodejs.org/job/node-test-commit-arm/30615/

Is it intended that 14.x build against ubuntu1604-arm64?

Its not excluded for any arm builds ATM, but it's EOL, I think it should have been excluded, or at least, should be excluded for anyType gte(14.x), and perhaps even 12.x. Note that its excluded from release right now by virtue of no machine having that label, so doesn't need .groovy exclusion.

I am pushing a commit to exclude ubuntu14-04 onto #2262 , tell me if I got it wrong.

cc: @nodejs/platform-arm @nodejs/build

rvagg · 2020-04-09T05:23:58Z

No, we should probably remove 1604-arm64.

Refs: #2242

Refs: #2242 PR-URL: #2262

sam-github · 2020-04-09T17:06:15Z

#2262 landed, making ci/ci-release config changes.

sam-github · 2020-04-09T17:23:17Z

Configured, kicked off test builds of 3 relevant jobs for master+12.x

sam-github · 2020-04-09T18:19:40Z

kicked node-test-commit-arm off on master again, it was missing the bits of select-compiler.sh that are pasted into it, see #2279

sam-github · 2020-04-09T19:24:33Z

arm and x86 for iojs+release required some extra bits in their config, rerunning.

sam-github · 2020-04-09T21:12:04Z

https://ci-release.nodejs.org/job/iojs+release/5882/nodes=centos7-arm64-gcc8/consoleFull looks OK:

12:50:12 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

https://ci-release.nodejs.org/job/iojs+release/5882/nodes=centos7-ppcle-release-64/consoleFull looks OK:

12:50:02 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

https://ci-release.nodejs.org/job/iojs+release/5882/nodes=centos7-64-gcc8/consoleFull (x64) looks OK, but is still building:

13:47:42 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

https://ci-release.nodejs.org/job/iojs+release/5882/nodes=rhel7-s390x-release/consoleFull looks OK:

12:50:01 gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3)

sam-github · 2020-04-09T21:16:36Z

I think this is done, but it could use a sanity check.

richardlau mentioned this issue Mar 22, 2020

Platform requirements for Node.js 14 #2168

Closed

8 tasks

richardlau mentioned this issue Mar 24, 2020

Node.js Build WorkGroup Meeting 2020-03-24 #2235

Closed

sam-github mentioned this issue Apr 2, 2020

request for access to all release machine, and ci(-release).nodejs.org #2220

Closed

sam-github mentioned this issue Apr 3, 2020

ansible: install devtoolset-8 on centos7 #2262

Merged

This comment has been minimized.

Sign in to view

richardlau mentioned this issue Apr 8, 2020

Node.js Build WorkGroup Meeting 2020-04-14 #2272

Closed

sam-github added a commit that referenced this issue Apr 9, 2020

ansible: install devtoolset-8 on centos7

e0f5ca2

Refs: #2242

sam-github added a commit that referenced this issue Apr 9, 2020

ansible: install devtoolset-8 on centos7

bec0ae2

Refs: #2242 PR-URL: #2262

richardlau closed this as completed Apr 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use devtoolset-8 on Centos/RHEL for Node.js 14+ #2242

Use devtoolset-8 on Centos/RHEL for Node.js 14+ #2242

richardlau commented Mar 22, 2020 •

edited by sam-github

Loading

rvagg commented Mar 23, 2020

sam-github commented Mar 23, 2020

sam-github commented Apr 2, 2020

sam-github commented Apr 2, 2020

sam-github commented Apr 2, 2020 •

edited

Loading

richardlau commented Apr 2, 2020

sam-github commented Apr 3, 2020

rvagg commented Apr 3, 2020

sam-github commented Apr 3, 2020

sam-github commented Apr 3, 2020

sam-github commented Apr 3, 2020

sam-github commented Apr 3, 2020 •

edited

Loading

sam-github commented Apr 3, 2020 •

edited

Loading

sam-github commented Apr 3, 2020

rvagg commented Apr 6, 2020

sam-github commented Apr 6, 2020

sam-github commented Apr 6, 2020

This comment has been minimized.

sam-github commented Apr 7, 2020

sam-github commented Apr 7, 2020

sam-github commented Apr 7, 2020

rvagg commented Apr 8, 2020

richardlau commented Apr 8, 2020 •

edited

Loading

richardlau commented Apr 8, 2020

rvagg commented Apr 8, 2020

sam-github commented Apr 8, 2020

sam-github commented Apr 8, 2020

rvagg commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

Use devtoolset-8 on Centos/RHEL for Node.js 14+ #2242

Use devtoolset-8 on Centos/RHEL for Node.js 14+ #2242

Comments

richardlau commented Mar 22, 2020 • edited by sam-github Loading

Install devtoolset-8 on centos7:

Install on rhel7

Use devtoolset-8 for 14.x: PR #2262

rvagg commented Mar 23, 2020

sam-github commented Mar 23, 2020

sam-github commented Apr 2, 2020

sam-github commented Apr 2, 2020

sam-github commented Apr 2, 2020 • edited Loading

richardlau commented Apr 2, 2020

sam-github commented Apr 3, 2020

rvagg commented Apr 3, 2020

sam-github commented Apr 3, 2020

sam-github commented Apr 3, 2020

sam-github commented Apr 3, 2020

sam-github commented Apr 3, 2020 • edited Loading

sam-github commented Apr 3, 2020 • edited Loading

sam-github commented Apr 3, 2020

rvagg commented Apr 6, 2020

sam-github commented Apr 6, 2020

sam-github commented Apr 6, 2020

This comment has been minimized.

sam-github commented Apr 7, 2020

sam-github commented Apr 7, 2020

sam-github commented Apr 7, 2020

rvagg commented Apr 8, 2020

richardlau commented Apr 8, 2020 • edited Loading

richardlau commented Apr 8, 2020

rvagg commented Apr 8, 2020

sam-github commented Apr 8, 2020

sam-github commented Apr 8, 2020

rvagg commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

sam-github commented Apr 9, 2020

richardlau commented Mar 22, 2020 •

edited by sam-github

Loading

sam-github commented Apr 2, 2020 •

edited

Loading

sam-github commented Apr 3, 2020 •

edited

Loading

sam-github commented Apr 3, 2020 •

edited

Loading

richardlau commented Apr 8, 2020 •

edited

Loading