CI-Change: tweaks to `make-v8` test script #1536

refack · 2018-10-16T21:39:41Z

https://ci.nodejs.org/job/node-test-commit-v8-linux/jobConfigHistory/showDiffFiles?timestamp1=2018-09-11_18-19-03&timestamp2=2018-10-16_17-24-50
I've replaced the call to patch tools/make-v8, to simply overwrite it with the one in $HOME/build-tools/make-v8.patch:

-git apply $HOME/build-tools/make-v8.patch
+cp -f $HOME/build-tools/make-v8.patch tools/make-v8.sh

make-v8.patch is now a full and valid bash script:

#!/bin/bash -xe

BUILD_ARCH_TYPE=$1
V8_BUILD_OPTIONS=$2

cd deps/v8
tools/node/fetch_deps.py .

# set paths manually for now to use locally installed gn
export BUILD_TOOLS=/home/iojs/build-tools
export LD_LIBRARY_PATH=$BUILD_TOOLS:$LD_LIBRARY_PATH
export PATH=$BUILD_TOOLS:$PATH
CXX_PATH=`which $CXX |grep g++`
rm -f "$BUILD_TOOLS/g++"
rm -f "$BUILD_TOOLS/gcc"
ln -s $CXX_PATH "$BUILD_TOOLS/g++"
ln -s $CXX_PATH "$BUILD_TOOLS/gcc"
g++ --version
export PKG_CONFIG_PATH=$BUILD_TOOLS/pkg-config
gn gen -v out.gn/$BUILD_ARCH_TYPE --args='is_component_build=false is_debug=false use_goma=false goma_dir="None" use_custom_libcxx=false v8_target_cpu="s390x" target_cpu="s390x"'
ninja -v -C out.gn/$BUILD_ARCH_TYPE d8 cctest inspector-test

(added -xe to the shebang, and -v to the calls to gn and ninja)

Test job: https://ci.nodejs.org/job/node-test-commit-v8-linux/1767/nodes=rhel72-s390x,v8test=v8test/console
/CC @nodejs/v8-update @nodejs/platform-s390

The text was updated successfully, but these errors were encountered:

mhdawson · 2018-10-19T20:29:52Z

My first question is what is the motivation for this change? I made it a patch so that :

ideally we'd get any changes if there are changes to the underlying file and
if that was not possible we'd know because the application of the patch would fail?

refack · 2018-10-19T21:31:02Z

My first question is what is the motivation for this change?

The main motivation was to add verbosity (since the build was failing). I tried to add it to tools/make-v8.sh but like you said the patch failed. That led me to dig into that whole mechanism... I found out that tools/make-v8.sh is only used in node-test-commit-v8-linux, so I thought it is not optimal that 2 out of 3 of its uses patch it significantly (with patches that come from the CI worker's drive, which is fragile).

I meant to follow up with a PR to add ifs in tools/make-v8.sh, eliminating the need to manipulate that file at all in CI...

mhdawson · 2018-10-23T18:18:14Z

Is the patch failing now ?

It was intended to be a short-medium term fix while we got changes into the google repos so I agree finding something better makes sense. Just not sure the copy over is better.

Unfortunately, the person who was working on it with me has left the team so we'll have to ask if when/the change might be making it into the v8 repos. @gdams adding you into the thread as well so you are up to speed.

I think next step is to figure out if/when that change might show up in the google repos so that we could remove any special behaviour and then based on how far out it is decide what to do.

mhdawson · 2018-10-23T18:20:41Z

@refack I guess the patch was failing, do you have the output of what the patch failure told us?

refack · 2018-10-23T19:11:04Z

04:04:08 Pythoning //build/config/linux/pkg-config.py took 212ms
04:04:08 Defining config //build/config/linux:glib(//build/toolchain/linux:s390x)
04:04:08 Pythoning python -- /data/iojs/build/workspace/node-test-commit-v8-linux/nodes/rhel72-s390x/v8test/v8test/deps/v8/build/config/linux/pkg-config.py atspi-2
04:04:08 Pythoning //build/config/linux/pkg-config.py took 38ms
04:04:08 ERROR at //build/config/linux/pkg_config.gni:103:17: Script returned non-zero exit code.
04:04:08     pkgresult = exec_script(pkg_config_script, args, "value")
04:04:08                 ^----------
04:04:08 Current dir: /data/iojs/build/workspace/node-test-commit-v8-linux/nodes/rhel72-s390x/v8test/v8test/deps/v8/out.gn/s390x.release/
04:04:08 Command: python -- /data/iojs/build/workspace/node-test-commit-v8-linux/nodes/rhel72-s390x/v8test/v8test/deps/v8/build/config/linux/pkg-config.py atspi-2
04:04:08 Returned 1.
04:04:08 stderr:
04:04:08 
04:04:08 Package atspi-2 was not found in the pkg-config search path.
04:04:08 Perhaps you should add the directory containing `atspi-2.pc'
04:04:08 to the PKG_CONFIG_PATH environment variable
04:04:08 No package 'atspi-2' found
04:04:08 Could not run pkg-config.
04:04:08 
04:04:08 See //build/config/linux/BUILD.gn:104:3: whence it was called.
04:04:08   pkg_config("atspi2") {
04:04:08   ^---------------------
04:04:08 See //build/config/compiler/BUILD.gn:224:18: which caused the file to be included.
04:04:08     configs += [ "//build/config/linux:compiler" ]
04:04:08                  ^------------------------------
04:04:08 make: *** [v8] Error 1

https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=rhel72-s390x,v8test=v8test/1779/console

mhdawson · 2018-10-23T19:17:32Z

That does not look like a failure to apply the patch to the original tools/make-v8.sh file?

refack · 2018-10-23T19:26:21Z

Ohh, that fail:

17:02:36 + git apply /home/iojs/build-tools/make-v8.patch
17:02:36 /home/iojs/build-tools/make-v8.patch:23: trailing whitespace.
17:02:36 gn gen out.gn/$BUILD_ARCH_TYPE --args='is_component_build=false is_debug=false use_goma=false goma_dir="None" use_custom_libcxx=false v8_target_cpu="s390x" target_cpu="s390x"' 
17:02:36 error: patch failed: tools/make-v8.sh:5
17:02:36 error: tools/make-v8.sh: patch does not apply
17:02:36 Build step 'Execute shell' marked build as failure

After I patched tools/make-v8.sh with nodejs/node@0e62432

mhdawson · 2018-10-23T20:00:05Z

Thanks, that info lets me better understand the original issue and our current state.

And if I understand correctly, you were then going to PR in tools/make-v8.sh in the node.js repo to conditionally add the extra stuff for PPC and S390.

If that is what you were thinking, it probably makes sense. The patch has been in place a lot longer then I'd hoped and if you are going to build v8 using make-v8, even outside the CI you'd need those extra changes so maybe just putting them in the master version of make-v8 makes sense.

refack · 2018-10-23T20:53:55Z

@mhdawson submitted nodejs/node#23839

refack · 2018-11-10T22:03:35Z

nodejs/node#23839 landed, so now I had to do the same on the ppcle machine:
https://ci.nodejs.org/job/node-test-commit-v8-linux/jobConfigHistory/showDiffFiles?timestamp1=2018-11-09_09-49-37&timestamp2=2018-11-10_16-59-07

old make-v8.sh from master

#!/bin/bash

BUILD_ARCH_TYPE=$1
V8_BUILD_OPTIONS=$2

cd deps/v8
tools/node/fetch_deps.py .
PATH=~/_depot_tools:$PATH tools/dev/v8gen.py $BUILD_ARCH_TYPE --no-goma $V8_BUILD_OPTIONS
PATH=~/_depot_tools:$PATH ninja -C out.gn/$BUILD_ARCH_TYPE/ d8 cctest inspector-test

ppcle patch:

diff --git a/tools/make-v8.sh b/tools/make-v8.sh
index 4365412..0092d6c 100755
--- a/tools/make-v8.sh
+++ b/tools/make-v8.sh
@@ -5,5 +5,17 @@ V8_BUILD_OPTIONS=$2

 cd deps/v8
 tools/node/fetch_deps.py .
-PATH=~/_depot_tools:$PATH tools/dev/v8gen.py $BUILD_ARCH_TYPE --no-goma $V8_BUILD_OPTIONS
-PATH=~/_depot_tools:$PATH ninja -C out.gn/$BUILD_ARCH_TYPE/ d8 cctest inspector-test
+
+# set paths manually for now to use locally installed gn
+export BUILD_TOOLS=/home/iojs/build-tools
+export LD_LIBRARY_PATH=$BUILD_TOOLS:$LD_LIBRARY_PATH
+export PATH=$BUILD_TOOLS:$PATH
+CXX_PATH=`which $CXX |grep g++`
+rm -f "$BUILD_TOOLS/g++"
+rm -f "$BUILD_TOOLS/gcc"
+ln -s /usr/bin/$CXX "$BUILD_TOOLS/g++"
+ln -s /usr/bin/$CC "$BUILD_TOOLS/gcc"
+g++ --version
+export PKG_CONFIG_PATH=$BUILD_TOOLS/pkg-config-files
+gn gen out.gn/$BUILD_ARCH_TYPE --args='is_component_build=false is_debug=false use_goma=false goma_dir="None" use_custom_libcxx=false v8_target_cpu="ppc64" target_cpu="ppc64"'
+ninja -C out.gn/$BUILD_ARCH_TYPE d8 cctest inspector-test

new:

#!/bin/bash

BUILD_ARCH_TYPE=$1
V8_BUILD_OPTIONS=$2

cd deps/v8
tools/node/fetch_deps.py .

# set paths manually for now to use locally installed gn
export BUILD_TOOLS=/home/iojs/build-tools
export LD_LIBRARY_PATH=$BUILD_TOOLS:$LD_LIBRARY_PATH
export PATH=$BUILD_TOOLS:$PATH
CXX_PATH=`which $CXX |grep g++`
rm -f "$BUILD_TOOLS/g++"
rm -f "$BUILD_TOOLS/gcc"
ln -s /usr/bin/$CXX "$BUILD_TOOLS/g++"
ln -s /usr/bin/$CC "$BUILD_TOOLS/gcc"
g++ --version
export PKG_CONFIG_PATH=$BUILD_TOOLS/pkg-config-files
gn gen out.gn/$BUILD_ARCH_TYPE --args='is_component_build=false is_debug=false use_goma=false goma_dir="None" use_custom_libcxx=false v8_target_cpu="ppc64" target_cpu="ppc64"'
ninja -C out.gn/$BUILD_ARCH_TYPE d8 cctest inspector-test

refack · 2018-11-10T22:22:44Z

Follow Up in core - nodejs/node#24293

rvagg · 2018-11-11T03:03:48Z

👍

BethGriggs · 2018-11-13T19:01:05Z

I am hitting the following when running v8.x test-commit-v8 jobs - I am not sure if it is related to this change?

17:58:20 + tools/node/fetch_deps.py .
17:58:20 tools/make-v8.sh: line 7: tools/node/fetch_deps.py: No such file or directory

https://ci.nodejs.org/job/node-test-commit-v8-linux/nodes=rhel72-s390x,v8test=v8test/1846/console

//cc @refack @gdams

refack · 2018-11-13T19:16:42Z

I am hitting the following when running v8.x test-commit-v8 jobs - I am not sure if it is related to this change?

Probably has something to do with this... I'm looking into this.

refack · 2018-11-13T19:27:59Z

Ok, got the linuxOne to compile - https://ci.nodejs.org/job/node-test-commit-v8-linux/1849/nodes=rhel72-s390x,v8test=v8test/console

The ppcle issue seems unrelated:

14:25:33 Error: Command 'cipd ensure -log-level error -root /home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/deps -ensure-file /tmp/tmpJwz4ny.ensure' returned non-zero exit status 1
14:25:33 
14:25:33 ________ running 'cipd ensure -log-level error -root /home/iojs/build/workspace/node-test-commit-v8-linux/nodes/ppcle-ubuntu1404/v8test/v8test/deps -ensure-file /tmp/tmpJwz4ny.ensure' in '.'
14:25:33 Errors:
14:25:33   failed to resolve infra/tools/luci/isolate/linux-ppc64le@git_revision:bc125484b8513898f17bc2501ac5e95330f44a3b (line 4): no such tag
14:25:33 Running: gclient root
14:25:33 Running: gclient config --spec 'solutions = [
14:25:33   {
14:25:33     "url": "https://chromium.googlesource.com/v8/v8.git",
14:25:33     "managed": False,
14:25:33     "name": "v8",
14:25:33     "deps_file": "DEPS",
14:25:33     "custom_deps": {},
14:25:33   },
14:25:33 ]
14:25:33 '
14:25:33 Running: gclient sync --with_branch_heads

refack · 2018-11-13T19:30:31Z

The ppcle issue seems unrelated:

Last time this happened we had to pin the depot_tools version - #886 (comment)

If anyone can figure out which one works, I'll set it up.

@nodejs/platform-ppc

mhdawson · 2018-11-13T22:32:16Z

@gdams can you work with John to figure out what we should do.

PR-URL: #24293 Fixes: nodejs/build#1536 Reviewed-By: Ben Noordhuis <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Franziska Hinkelmann <[email protected]>

refack added the ci-change PSA of configuration changes label Oct 16, 2018

refack self-assigned this Oct 16, 2018

refack added platform:linuxONE ci-public labels Oct 16, 2018

refack mentioned this issue Oct 23, 2018

build,tools: update make-v8.sh for s390x nodejs/node#23839

Closed

3 tasks

refack changed the title ~~CI-Change: tweaks to make-v8 test script on s390~~ CI-Change: tweaks to make-v8 test script Nov 10, 2018

refack mentioned this issue Nov 10, 2018

build,tools: update make-v8.sh for ppc64le nodejs/node#24293

Merged

3 tasks

refack added the platform:ppc label Nov 10, 2018

refack closed this as completed in nodejs/node@3d4b281 Nov 16, 2018

sam-github mentioned this issue Jun 27, 2019

update node-test-commit-v8-linux to not overwrite make-v8.sh with a local copy #1857

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI-Change: tweaks to `make-v8` test script #1536

CI-Change: tweaks to `make-v8` test script #1536

refack commented Oct 16, 2018 •

edited

Loading

mhdawson commented Oct 19, 2018

refack commented Oct 19, 2018

mhdawson commented Oct 23, 2018

mhdawson commented Oct 23, 2018

refack commented Oct 23, 2018

mhdawson commented Oct 23, 2018

refack commented Oct 23, 2018

mhdawson commented Oct 23, 2018 •

edited

Loading

refack commented Oct 23, 2018

refack commented Nov 10, 2018

refack commented Nov 10, 2018

rvagg commented Nov 11, 2018

BethGriggs commented Nov 13, 2018

refack commented Nov 13, 2018

refack commented Nov 13, 2018

refack commented Nov 13, 2018 •

edited

Loading

mhdawson commented Nov 13, 2018

CI-Change: tweaks to make-v8 test script #1536

CI-Change: tweaks to make-v8 test script #1536

Comments

refack commented Oct 16, 2018 • edited Loading

mhdawson commented Oct 19, 2018

refack commented Oct 19, 2018

mhdawson commented Oct 23, 2018

mhdawson commented Oct 23, 2018

refack commented Oct 23, 2018

mhdawson commented Oct 23, 2018

refack commented Oct 23, 2018

mhdawson commented Oct 23, 2018 • edited Loading

refack commented Oct 23, 2018

refack commented Nov 10, 2018

refack commented Nov 10, 2018

rvagg commented Nov 11, 2018

BethGriggs commented Nov 13, 2018

refack commented Nov 13, 2018

refack commented Nov 13, 2018

refack commented Nov 13, 2018 • edited Loading

mhdawson commented Nov 13, 2018

CI-Change: tweaks to `make-v8` test script #1536

CI-Change: tweaks to `make-v8` test script #1536

refack commented Oct 16, 2018 •

edited

Loading

mhdawson commented Oct 23, 2018 •

edited

Loading

refack commented Nov 13, 2018 •

edited

Loading